Information processing system, computer program, and method for controlling information processing system

Abstract

To provide an information processing system for detecting the inclination of a light receiving surface of an image pickup device based on an image.SOLUTION: An information processing system has: image acquisition means that acquires an imaging signal from imaging means that picks up an image of a subject from above; image processing means that creates a distance image based on the imaging signal; inclination detection means that detects the inclination of a light receiving surface of the imaging means from the horizontal plane based on the distance image; and control means for notifying of information on the inclination detected by the inclination detection means.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing system, a computer program, and an information processing Controlling the system It concerns methods, etc. [Background technology]

[0002] In conventional agriculture, the growth status of agricultural crops has traditionally been determined based on the experience, knowledge, and intuition of farmers. However, in recent years, systems that use IoT (Internet of Things) technology to obtain information from environmental sensors and image data to determine the growth status of crops have been developed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-96724 Summary of the Invention [Problem to be solved by the invention]

[0004] The above-mentioned Patent Document 1 describes an information processing device that accumulates crop growth index data for multiple dates and times generated based on images of the crops and environmental information data related to the field environment, and generates display data for displaying the growth process. However, Patent Document 1 does not fully describe the configuration for acquiring images of agricultural crops with high accuracy, the installation method thereof, etc. Therefore, there are cases where the growth status of agricultural crops cannot be accurately grasped.

[0005] SUMMARY OF THE INVENTION In order to solve the above-mentioned problems of the prior art, an object of the present invention is to provide an information processing system or the like for detecting the tilt of the light receiving surface of an imaging element based on an image. [Means for solving the problem]

[0006] In order to achieve the above object, an information processing system according to one aspect of the present invention comprises: an image acquisition means for acquiring an image signal from an image acquisition means for capturing an image of a subject from above; an image processing means for generating a distance image based on the imaging signal; tilt detection means for detecting a tilt of the light receiving surface of the imaging means from the horizontal based on the distance image; a control means for notifying information about the tilt detected by the tilt detection means; death, The information about the tilt includes information about adjustment of the tilt of the light receiving surface and information about an abnormality in the tilt. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, it is possible to realize an information processing system or the like for detecting the tilt of the light receiving surface of an imaging element based on an image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating the overall configuration of an information processing system 200 using an information processing device 100 according to an embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating an example of installation of an information processing system 200 according to an embodiment. [Figure 3] 10A and 10B are diagrams illustrating an example of the positioning relationship between the camera 112, the arm 202, and the pillar 201 according to the embodiment. [Figure 4] 10A is a diagram showing the XY cross section of the first connecting portion according to the embodiment as viewed from the +Z direction, and FIG. 10B is a schematic perspective view of the first connecting portion 204. FIG. [Figure 5] 1 is a schematic diagram showing a connecting portion between a camera 112 and an arm 202 according to an embodiment, viewed from above. [Figure 6] 10 is a flowchart illustrating a process in a maintenance mode of the information processing apparatus. [Figure 7](A) is a diagram showing an example of a maintenance homepage screen, (B) is a diagram showing the state when the "Camera Photo" tab is clicked and an RGB image 705 and a Depth image 704 are displayed, and (C) is a diagram showing an example of the Depth image being enlarged. [Figure 8] 10A is a diagram showing an example in which the camera 112 is tilted normally, and FIG. 10B is a diagram showing an example in which the planarity is abnormal. [Figure 9] 10 is a diagram for explaining the operation of the information processing device 100 or the server 113 during normal operation. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, preferred embodiments of the present invention will be described by way of example with reference to the accompanying drawings. In each drawing, the same members or elements are designated by the same reference numerals, and duplicated descriptions will be omitted or simplified. In addition, in the embodiment, an example of an information processing system that uses an information processing device to measure the growth status of agricultural crops is described, but the information processing system of this embodiment can also be applied to, for example, factory or general monitoring monitor systems, and its uses are not limited.

[0010] FIG. 1 is a diagram showing the overall configuration of an information processing system 200 using an information processing device 100 according to an embodiment of the present invention. The overall configuration of an information processing system 200 using an information processing device according to an embodiment of the present invention will be described below with reference to FIG.

[0011] The information processing device 100 in this embodiment includes a battery 101, a battery control unit 102, a solar panel 103, a sub-CPU 104, an LTE module 105, a main board 106, a USB hub 107, an I / O unit 108, etc. The battery control unit 102 controls the storage of power generated by the solar panel 103 in the battery 101, and controls the supply of the power stored in the battery 101 to the inside of the information processing device.

[0012] Reference numeral 106 denotes a main board, which incorporates a CPU as a computer and a memory that stores a computer program for controlling the operation of the CPU. Reference numeral 104 denotes a sub-CPU, which communicates with the main board 106 and controls the operation of the battery control unit 102. Reference numeral 105 denotes an LTE module, which controls the main board 106 to communicate with a predetermined external server 113, such as a cloud server, using, for example, LTE (Long Term Evolution).

[0013] Reference numeral 107 denotes a USB hub, which acquires color information of the surrounding environment from color sensors 109, 110, and 111. Each of the color sensors 109, 110, and 111 has built-in R (red), G (green), and B (blue) light-receiving sensors. Each light-receiving sensor receives ambient light, and the color temperature of the light source can be acquired by calculating the output ratio of the R, G, and B light-receiving sensors. Note that a white diffusion plate, for example, may be placed on the R, G, and B light-receiving sensors. This allows the R, G, and B light-receiving sensors of the color sensors 109, 110, and 111 to receive averaged ambient light, and the color temperature of the light source can be acquired by calculating the output ratio of the R, G, and B light-receiving sensors.

[0014] Color sensors 109, 110, and 111 for detecting the ambient color temperature each supply color temperature information, information on the R, G, and B output ratios, and R, G, and B signals to the main board 106 via the USB hub 107. In this embodiment, the information output from the color sensors 109, 110, and 111 is averaged on the main board 106 to obtain final color temperature information.

[0015] Alternatively, if one of the measurements from the three color sensors 109, 110, and 111 is significantly different from the other two, the average value of the two similar measurements may be used. If the outputs of the three color sensors are significantly different from each other, it may be determined to be abnormal and a warning may be sent to the registered mobile terminal 114, etc.

[0016] An external PC 115 is configured to communicate with the main board 106 via, for example, a LAN. Reference numeral 108 denotes an I / O unit, which is used to communicate with a registered mobile terminal 114 such as a smartphone via a WIFI network or the like.

[0017] For example, the reference numeral 112 denotes a camera using an image sensor with a Dualpixel CMOS Auto Focus (DAF) function. In a Dualpixel CMOS sensor, each pixel is composed of two photodiodes, each of which receives light from a different pupil area to detect two image signals, enabling phase-difference AF.

[0018] In this embodiment, an example will be described in which the camera 112 uses one image sensor with a DAF structure, but the camera 112 may also be a stereo camera. Alternatively, the camera 112 may be a combination of a stereo camera and a camera with RGB output. Furthermore, structured light or ToF (Time of Flight) technology may also be used.

[0019] The DAF function is a function in which each pixel is configured with a set of a first light receiving element that receives light from the right exit pupil of the optical system and a second light receiving element that receives light from the left exit pupil, and is capable of measuring the distance to the subject by calculating the correlation distance between the first image obtained by the first light receiving element group and the second image obtained by the second light receiving element group. A stereo camera is a device that measures the parallax between two images and calculates depth distance using the principle of triangulation.

[0020] Furthermore, when using structured light technology, a projector function is provided, a pattern is projected by the projector, and the projection state is captured by a camera to detect the degree of distortion of the pattern. Distance can be calculated by calculating depth information from the degree of distortion. Furthermore, when using ToF technology, a light source such as an LD or LED is further equipped, and when the light is irradiated from this light source and the reflected light is received, the time from irradiation to the return of the reflected light is measured. The depth distance can be calculated by multiplying the speed of light by time.

[0021] The image sensor is, for example, a CMOS image sensor, and is configured with a plurality of pixels arranged two-dimensionally. In front of each pixel, for example, R, G, and B color filters are arranged in a Bayer pattern, for example, so that a color image signal is output from the image sensor. The pixels of the image sensor are arranged in multiple rows in the horizontal scanning direction, and the multiple rows are arranged side by side in the vertical scanning direction.

[0022] The image captured by the camera 112 is supplied to the main board 106 via, for example, a USB. The camera 112 is positioned so as to capture an image of a field such as a rice paddy from directly above and generate a distance image to measure the height distribution of agricultural crops (plants). The imaging element functions as imaging means that captures an image of a plant as a subject from above and generates an imaging signal.

[0023] The main board 106 functions as an image acquisition means for acquiring an image signal obtained from the imaging means. It also functions as an image processing means for generating a distance image based on the image signal. The main board 106 transmits the acquired image signal to the server 113, and the server 113 measures, for example, leaf color, number of stems, height, etc. of the crops (plants) based on the image signal, and determines the growth status of the crops based on the results.

[0024] In this way, the information processing device 100 of this embodiment constitutes an information processing system 200 that constantly or periodically captures images of agricultural crops (plants) such as rice in a field, transmits them to an external predetermined server, etc., and observes and records the growth status of the agricultural crops. The components in the information processing device 100 shown in FIG. 1 do not have to be housed in the same housing, and may be configured as separate devices connected to each other via signal paths.

[0025] FIG. 2 is a diagram illustrating an example of installation of an information processing system 200 according to an embodiment. In Figure 2, 201 is a pillar portion, 202 is an arm portion, 203 is a first stake portion, 204 is a first connecting portion, 205 is a rotation suppression portion, 206 is a USB cable, 209 is a group of adjustment screws, and 210 is a second connecting portion for connecting the arm portion 202 and the pillar portion 201.

[0026] Reference numerals 201 to 205, 210, etc. function as support members for supporting camera 112 with a built-in imaging element. Reference numeral 207 denotes a crop (plant), such as rice, 208 denotes the ground, and 211 denotes a second stake. Adjustment screw group 209 includes a plurality of screws for adjustment. That is, as will be described later, it includes, for example, eight screws 209a1, 209a2, 209b1, 209b2, 209c1, 209c2, 209d1, and 209d2, and functions as tilt adjustment members for adjusting the tilt of the light receiving surface of the imaging element from the horizontal.

[0027] In the information processing system 200 of this embodiment, the information processing device 100 and the solar panel 103 are supported by a second stake 211 and are installed a predetermined distance above the ground. The color sensors 109 to 111 and the camera 112 are held at the tip of an arm 202 that extends horizontally from a pillar 201, and outputs from the color sensors 109 to 111 and the camera 112 are supplied to the information processing device 100 via a USB cable 206.

[0028] 2, camera 112 faces vertically downward, and color sensors 109-111 are installed facing vertically upward at positions corresponding to the rear side of the camera installation position, but this is not limiting. For example, color sensors 109-111 may be installed facing vertically upward at the top of information processing device 100. In this way, they are installed at a relatively lower position compared to when they are installed above camera 112, which makes it easier to perform maintenance, such as wiping off dirt adhering to the color sensors.

[0029] Color sensors 109-111 are arranged facing upward and are configured to detect the color temperature of sunlight, which is the light source. Color sensors 109-111 may face the same direction, or may face slightly different directions. For example, color sensors 109-111 may be arranged to face different directions of the sun, which is the light source, as the direction of the sun changes.

[0030] In this way, by placing the color sensor for measuring color temperature facing upward, in the direction of the sun, it is possible to accurately measure the color temperature of the light source.Furthermore, by providing multiple color sensors and measuring color temperature by, for example, averaging their outputs, it has the effect of being less susceptible to the effects of factors such as the angle of the sun.

[0031] Furthermore, if one of the three color sensor outputs is significantly different from the other two, the outputs of the two similar color sensors can be averaged and used, making it possible to detect color temperature stably even if, for example, the surface of the color sensor is dirty. Therefore, the color image signal obtained from the camera 112 can be correctly white-balanced according to the color temperature.

[0032] As a result, the server 113 can accurately measure, for example, the leaf color of agricultural crops (plants) such as rice based on the RGB image, allowing the growth status of the crops to be measured correctly. Note that the measurement of leaf color in the server 113 may be realized by inference using a trained model that uses the image as input data and the leaf color numerical value as training data. In this case, the output data of the trained model is the leaf color numerical value. The final growth status information can be determined by referring to the growth status information that has been previously associated with the leaf color numerical value.

[0033] The camera 112 is positioned relative to the arm 202 and installed so as to capture images in the vertical direction (directly below). The arm 202 is connected to the post 201 in a positioned state by a second connecting part 210. This is to facilitate adjustment of the horizontal tilt of the light receiving surface of the camera 112. If the light receiving surface of the camera 112 is adjusted horizontally, the height of a crop such as rice 207 can be accurately measured when the camera acquires a range image of the field from directly above.

[0034] Furthermore, in this embodiment, when installing the camera 112, the horizontal scanning direction of the camera, for example, the Y direction, and the X direction (vertical scanning direction) perpendicular thereto, can be adjusted using the adjusting screws 209 or the like. Therefore, as described above, the height and number of stalks of agricultural crops (plants) such as rice in a field can be accurately measured using the server 113 or another terminal. In this embodiment, the server 113 or the like functions as a measuring means for measuring the height, number of stalks, and leaf color of agricultural crops (plants).

[0035] 3 is a diagram showing an example of the positioning relationship between camera 112, arm 202, and pillar 201 according to the embodiment. As shown in Fig. 3, arm 202 is provided with positioning mark 202a, and camera 112 is fixed to arm 202 after being aligned with positioning mark 112b provided on the housing of camera 112.

[0036] The positioning mark 112b corresponds to the horizontal scanning direction of the light receiving surface of the camera 112, and in this embodiment, the horizontal scanning direction of the light receiving surface 112a of the camera 112 is defined as the Y direction. That is, the positioning mark 112b and the positioning mark 202a are used to align the Y direction. On the other hand, the arm portion 202 is integrally formed with the second connecting portion 210, and a positioning mark 210a is also provided on the side surface of the second connecting portion 210. Like the positioning mark 202a, the positioning mark 210a is a positioning mark for alignment in the Y direction.

[0037] Furthermore, a positioning mark 201a is also provided on the side surface of the pillar 201, and the positioning mark 210a of the second connecting part 210 is aligned with the positioning mark 201a of the pillar 201 before connecting the second connecting part 210 to the pillar 201. In this way, the camera 112 is attached to the pillar 201 with the Y direction, which is the horizontal scanning direction of the camera 112, aligned with the positioning mark 201a of the pillar 201.

[0038] 2, the pillar portion 201 is connected to the first pile portion 203 by the first connecting portion 204, and the first pile portion 203 is screwed or driven into the ground 208 to a sufficient depth (for example, several tens of centimeters to several meters) to prevent it from wobbling. In addition, the rotation suppression portion 205 is fixed to the first pile portion 203 and has a protruding portion 205a that protrudes from the first pile portion, and the protruding portion 205a is screwed or driven into the ground to a sufficient depth to suppress rotation of the first arm portion.

[0039] The first connecting part 204 is provided with a group of adjustment screws 209, and by tightening or loosening a part of the group of adjustment screws 209, the pillar part 201 can be rotated around an axis in the X direction or an axis in the Y direction relative to the first pile part 203 in Fig. 2. The connecting length can also be adjusted in the Z direction (up and down), and after making these adjustments, the first pile part 203 and the pillar part 201 can be connected and fixed.

[0040] 4A is a diagram showing the XY cross section of the first connecting portion according to the embodiment as viewed from the +Z direction, and FIG. 4B is a schematic perspective view of the first connecting portion 204. FIG. The adjusting screw group 209 includes, for example, at least eight screws 209a1, 209a2, 209b1, 209b2, 209c1, 209c2, 209d1, and 209d2.

[0041] Screws 209a1 and 209a2, 209b1 and 209b2, 209c1 and 209c2, and 209d1 and 209d2 each form a pair, and the two screws that make up each pair are arranged side by side in the Z direction in the drawing. Note that screw 209a1 is arranged above screw 209a2, screw 209b1 above 209b2, screw 209c1 above 209c2, and screw 209d1 above 209d2 on the Z axis (toward the +Z direction).

[0042] 4(A), the set of screws 209a1 and 209a2 and the set of screws 209c1 and 209c2 are arranged in positions facing each other in the Y direction. The set of screws 209b1 and 209b2 and the set of screws 209d1 and 209d2 are arranged in positions facing each other in the X direction. The first connecting portion 204 is shaped like a pipe, and there is a slight gap between the column portion 201 and the first connecting portion 204. Each screw of the adjustment screw group 209 is configured to abut against the side surface of the column portion 201 or the pile portion 203 through a hole provided in the first connecting portion 204.

[0043] Therefore, by tightening or loosening the set of screws 209a1 and 209a2 and the set of screws 209c1 and 209c2, it is possible to adjust the rotation around the axis in the X direction, which is the vertical scanning direction of camera 112. On the other hand, since the set of screws 209b1 and 209b2 and the set of screws 209d1 and 209d2 are located at opposite positions in the X axis direction, it is possible to adjust the rotation around the axis in the Y direction of camera 112 by adjusting these screws.

[0044] 4(B), the above-mentioned positioning mark 201a is provided on the surface of the pillar portion 201 at a position extending in the longitudinal direction of the pillar portion 201. The positioning mark 201a may be provided without interruption in the longitudinal direction of the pillar portion 201, or may be omitted in the middle. In addition, a positioning mark 204a for alignment in the Y direction is also provided on the surface of the first connecting portion 204.

[0045] Therefore, with the positioning marks 201a and 204a aligned, the first connecting portion 204 is connected to the column portion 201. As a result, the set of screws 209a1 and 209a2 and the set of screws 209c1 and 209c2, which are arranged opposite each other in the Y direction, face each other in a direction parallel to the Y direction, which is the horizontal scanning direction of the camera 112.

[0046] In the above example, the positioning marks 112b, 202a, 210a, 201a, 204a, etc. are provided for alignment in the Y direction, but positioning marks may be provided for alignment in the X direction instead of the Y direction. Alternatively, positioning marks may be provided for alignment in both the X and Y directions.

[0047] Because of this configuration, for example, when screws 209b1 and 209d2 are tightened and 209b2 and 209d1 are loosened, pillar 201 rotates left as viewed from the -Y direction in Figure 4(B) (rotates right as viewed from the +Y direction). Similarly, for example, when screws 209c1 and 209a2 are tightened and screws 209c2 and 209a1 are loosened, pillar 201 rotates right as viewed from the +X direction in Figure 4(B).

[0048] It is desirable to indicate that the set of screws 209b1 and 209b2 and the set of screws 209d1 and 209d2 are for adjustment around the axis in the Y direction (horizontal scanning direction), and it is also desirable to indicate that the set of screws 209a1 and 209a2 and the set of screws 209c1 and 209c2 are for adjustment around the axis in the X direction (vertical scanning direction).

[0049] Alternatively, it is desirable to indicate, for example, "X right rotation" (right rotation when viewed from the +X direction) near the screws 209a2 and 209c1 on the side of the first connecting part, and "X left rotation" (left rotation when viewed from the +X direction) for the screws 209a1 and 209c2. Also, it is desirable to indicate, for example, "Y right rotation" (right rotation when viewed from the +Y direction) near the screws 209b1 and 209d2 on the side of the first connecting part, and "Y left rotation" (left rotation when viewed from the +Y direction) on the screws 209b2 and 209d1.

[0050] In other words, "X right rotation" means rotating the light receiving surface of the image sensor built into camera 112 to the right when viewed from the +X direction, and "X left rotation" means rotating the light receiving surface of the image sensor to the left when viewed from the +X direction. On the other hand, "Y right rotation" means rotating to the right as viewed from the +Y direction, and "Y left rotation" means rotating to the left as viewed from the +Y direction.

[0051] Therefore, when guidance such as "X rotate right" is displayed on the screen of the mobile terminal described below, it is easy to see that tightening screws 209a2 and 209c1 and loosening screws 209a1 and 209c2 will rotate the device right as viewed from the +X direction. Also, when guidance such as "Y rotate left" is displayed, it is easy to see that tightening screws 209b2 and 209d1 and tightening screws 209b1 and 209d2 will rotate the device left as viewed from the +Y direction, making adjustments easier.

[0052] In this way, in this embodiment, the guidance in the mobile terminal includes information indicating which of the multiple adjustment screws to adjust. Furthermore, the multiple adjustment screws are arranged to rotate the light receiving surface 112a about the rotation axis in the horizontal scanning direction and the rotation axis in the vertical scanning direction of the image sensor, so that it is easy to respond to the guidance, which is information about the tilt, and it is easy to adjust the tilt of the light receiving surface.

[0053] In this embodiment, the camera optical system is an erect optical system, but in the case of an inverted optical system, the rotation direction for adjustment is reversed, so the rotation direction instruction in the guidance described below is switched to an instruction to rotate in the reverse direction.

[0054] 5 is a schematic diagram of the connection portion between camera 112 and arm 202 according to this embodiment, seen from above, in which camera 112 is attached to the underside of arm 202 with its optical system facing downward, and is fixed to arm 202 with screws 501 to 504. In addition, screws 501 to 504 function as screws for adjusting the rotation angle of light receiving surface 112a of the imaging element built into camera 112 around the X and Y directions.

[0055] The line connecting the screws 501 and 502 and the line connecting the screws 503 and 504 pass near the center of the light receiving surface 112a of the image sensor and are perpendicular to each other. The line connecting the screws 501 and 502 coincides with the Y direction (horizontal scanning direction) of the light receiving surface 112a of the image sensor, and the line connecting the screws 503 and 504 coincides with the X direction (vertical scanning direction) of the light receiving surface 112a of the image sensor. Therefore, as described above, by aligning the positioning mark 112b provided on the housing of the camera 112 with the positioning mark 202a provided on the arm portion 202, the Y direction can be made to coincide with the horizontal scanning direction.

[0056] In the above example, the group of adjustment screws 209 of the first connecting portion 204 has the functions of adjusting rotation around the X and Y directions, adjusting in the Z direction, and connecting and fixing the first pile portion 203 and the column portion 201. However, in addition to the group of adjustment screws 209, connecting screws for connecting and fixing the first connecting portion 204 may be provided.

[0057] 5 via a connecting member such as a flexible shaft, may be provided around the first connecting portion 204. By configuring in this manner, the screws 501 to 504 can be directly adjusted via the connecting member such as a flexible shaft to the adjusting screws, making the adjustment work easier.

[0058] FIG. 6 is a flowchart showing the process in the maintenance mode of the information processing apparatus. The computer in the main board 106 executes a computer program stored in the memory to perform the operations of the steps in the flowchart of FIG. The flow of FIG. 6 starts when a switch (not shown) or the like of the information processing apparatus 100 is turned on to activate the maintenance mode.

[0059] In step S601, the main board 106 of the information processing device 100 displays a maintenance homepage screen on a display device such as a liquid crystal display of the mobile terminal 114 or the server 113. If the information processing device 100 is not connected to the mobile terminal 114 or the server 113 when it is started up, it waits until it is connected. After it is connected, it executes the processing of this step. An example of a display on the mobile terminal 114 will be described below. 7(A) to 7(C) are diagrams showing examples of screens displayed on a mobile terminal, and FIG. 7(A) is a diagram showing an example of a homepage screen for maintenance.

[0060] 7(A), the homepage screen displays detailed data 700 such as the ID number of the camera 112 connected to the information processing device 100, the date and time, the remaining battery level, whether the optical system is an erect optical system or an inverted optical system, and whether the level adjustment method is manual or automatic. Information on whether the optical system is an erect optical system or an inverted optical system is stored in the memory of the information processing device 100, and the content of the guidance described below is changed based on this information.

[0061] 6 differs depending on whether the adjustment method is automatic or manual. When the adjustment method is displayed as automatic, for example, a motor for rotating the light receiving surface 112a of the image sensor in the camera 112 about the X axis and a motor for rotating it about the Y axis are provided inside the camera. Alternatively, a motor for rotating the camera 112 about the X axis and a motor for rotating it about the Y axis are provided at the connection between the camera and the arm 202. Furthermore, the main board 106 can perform horizontal adjustment of the light receiving surface of the image sensor by controlling the driving of these motors. Furthermore, information indicating that the horizontal adjustment of the camera 112 can be performed automatically is stored in the memory of the information processing device 100.

[0062] On the homepage screen, a "Homepage" tab 701, a "Camera Photo" tab 702, and a "Close" button 703 are displayed. In step S602, it is determined whether the "Close" button 703 has been clicked, and if the answer is Yes, the process proceeds to step S603, where the flow of Fig. 6 is terminated. If the answer is No in step S602, the process proceeds to step S604.

[0063] In step S604, it is determined whether the "camera capture" tab has been clicked. Note that although it is determined here whether a button or tab has been "clicked," it may alternatively be determined whether mobile terminal 114 has a touch panel and whether a touch operation has been performed. The same applies to the determination of whether a click has been performed in the following explanation.

[0064] If the answer is No in step S604, the process returns to step S601, where the maintenance homepage screen continues to be displayed. If the answer is Yes in step S604, the process proceeds to step S605. In step S605, an image acquisition step is executed to acquire an RGB image, which is an image signal obtained from the image sensor of the camera 112.

[0065] In the next step S606, an image processing step is executed to generate a depth image based on the imaging signal. The depth image is an image in which the distance is calculated pixel by pixel based on two images with parallax obtained from the camera, and the color is changed according to the distance for each pixel.

[0066] In step S607, the RGB image and depth image are displayed side by side as shown in Fig. 7(B). Fig. 7(B) shows the state in which the "Camera Capture" tab is clicked and the RGB image 705 and depth image 704 are displayed. Note that the descriptions "RGB Image" and "Depth Image" are displayed to the left of the RGB image 705 and depth image 704, respectively. Note that if any warning or the like is received in this state, the warning may be notified by, for example, blinking the RGB image 705 or depth image 704.

[0067] In step S608, it is determined whether the depth image has been clicked. If the depth image is clicked in the display state of Fig. 7(B), the result in step S608 is determined as Yes and the process proceeds to step S610. If the depth image has not been clicked in step S608, the result is determined as No and the process proceeds to step S609.

[0068] In step S609, it is determined whether or not the "Home" tab 701 has been clicked. If No, the process returns to step S604 and the camera shooting screen is displayed; if Yes, the process returns to step S601 and the home screen is displayed. When the depth image is clicked, the process proceeds to step S610, where the depth image is enlarged and displayed.

[0069] Fig. 7(C) is a diagram showing an example of an enlarged display of a depth image. As shown in Fig. 7(C), five frames 706-707, arranged at the top, bottom, left, right, and center, are superimposed on the enlarged display of the depth image. Frame 706 is arranged in the center of the screen, and frames 707 and 708 are arranged to the right and left of frame 706, respectively. Frames 709 and 710 are arranged above and below frame 7 connecting screw group 20906, respectively.

[0070] In step S611, the average value of the distance information within each frame is calculated and displayed superimposed on the screen. In Figure 7(C), the average distance value within a frame 706 in the center of the screen is displayed as C155 next to C, which means center, as in 706a. Similarly, the average distance value within a frame 707 on the right side is displayed as R160 next to R, which means right, as in 707a. Furthermore, the average distance value within a frame 708 on the left side is displayed as L150 next to L, which means left, as in 708a.

[0071] The average value of the distance in the upper frame 709 is displayed as T153 next to T, which means up, as in 709a. The average value of the distance in the lower frame 710 is displayed as B157 next to B, which means down, as in 710a. In step S611, the difference between the average value of the distance within the frame 706 in the center of the screen and the average value of the distance within the other frames is calculated.

[0072] The result is then displayed, for example, as a numerical value 707b next to the right-hand frame 707, which is the difference between the average value of the distances in the central frame 706 and the average value of the distances in the right-hand frame 707. Also, based on the information that the optical system is an erect optical system, guidance saying "X right rotation" is displayed as in 707b. Also, next to the left frame 708, a numerical value of -5, which is the difference between the average value of the distance within the central frame 706 and the average value of the distance within the left frame 708, is displayed as 708b.

[0073] Furthermore, next to the upper frame 709, a numerical value of -2, which is the difference between the average value of the distance in the central frame 706 and the average value of the distance in the frame of the upper frame 709, is displayed as in 709b. Furthermore, since the optical system is an erect optical system, a guidance "Y rotate left" is displayed as in 709b. Furthermore, next to the lower frame 710, a numerical value of +2, which is the difference between the average value of the distance in the central frame 706 and the average value of the distance in the frame of the upper frame 710, is displayed as in 709b. Note that if information is acquired that the optical system is an inverted optical system, the rotation direction indicated by the guidance in 707b and 709b will be reversed.

[0074] In this way, in step S611, the average value of the distance within the frame 706 in the center of the screen and the average value of the distance within the other frames are displayed, and the difference between the average value of the distance within the frame 706 in the center of the screen and the average value of the distance within the other frames is also displayed. Furthermore, guidance on how to adjust the adjustment screws 209 based on the above average values ​​and differences of distance is displayed, for example, as "X right rotation" in 707b and "Y left rotation" in 709b.

[0075] That is, step S611 functions as a tilt detection means (tilt detection step) that detects the tilt of the light receiving surface of the image sensor of the camera from the horizontal based on the distance image. It also functions as a control means (control step) that notifies the smartphone of tilt information. Note that the tilt information here includes, for example, tilt information based on the difference between the average value of the distance within the central frame 706 and the average value of the distance within the other frames, and guidance information regarding adjustment of the tilt of the light receiving surface.

[0076] Note that, here, the tilt is indicated by displaying the difference between the average value of the distance within the frame 706 in the center of the screen and the average value of the distance within the other frames, but this method is not limited to this. For example, the tilt to the left or right may be indicated by displaying the difference between the average values ​​of the distance within the left and right frames. Similarly, the tilt to the up or down may be indicated by displaying the difference between the average values ​​of the distance within the top and bottom frames.

[0077] The guidance information only needs to include at least one of the adjustment amount and adjustment direction. In this embodiment, five frames are used to detect tilt based on distance information from multiple positions in the horizontal scanning direction of the distance image (frames 706-708) and distance information from multiple positions in the vertical scanning direction of the distance image (frames 706, 709, 710). However, the number of frames may be six or more.

[0078] In the above, the five frames 706 to 710 and information regarding the tilt are superimposed on the depth image, but some or all of the frames 706 to 710 and information regarding the tilt do not have to be superimposed on the depth image. For example, they may be superimposed on the RGB image, or information regarding the tilt may be displayed in a position separate from the image.

[0079] As described above, for example, when guidance "Rotate X right" is displayed, it is sufficient to rotate right as viewed from the +X direction by tightening screws 209a2 and 209c1 and loosening screws 209a1 and 209c2. Also, when guidance "Rotate Y left" is displayed, it is sufficient to rotate left as viewed from the +Y direction by tightening screws 209b2 and 209d1 and tightening screws 209b1 and 209d2, which makes the adjustment work more efficient.

[0080] Note that 711 is a button for turning on / off the superimposed display of these frames 706 to 710, the average distance values, etc., and Fig. 7(C) shows that the superimposed display is on. In response to clicking this button, a page corresponding to the image that is not superimposed is requested from the mobile terminal 114, and in response to this, the information processing device 100 sends to the mobile terminal 114 an HTML file including the enlarged depth image that is not superimposed.

[0081] Upon receiving this, the mobile terminal 114 interprets the HTML file and displays the screen without any overlapping display. Also, 712 is a button for storing the average value in the center of Fig. 7(C) as the camera height, 713 is a return button for returning to the camera shooting screen. The positions and sizes of the five frames 706 to 707 may be changed by the user as appropriate.

[0082] After the difference value and guidance are displayed in step S612, it is determined in step S613 whether the difference value is abnormal. For example, if the difference value is larger than a predetermined threshold, it is determined to be abnormal. Alternatively, for example, if the sum of the left and right difference values ​​or the sum of the top and bottom difference values ​​is approximately zero, it is considered normal, but if at least one of the sums is equal to or greater than a predetermined value, it is determined to be abnormal. If it is determined to be abnormal in step S613, the process proceeds to step S614, and if it is not determined to be abnormal, the process proceeds to step S615.

[0083] FIG. 8(A) is a diagram showing an example in which the camera 112 is tilted normally, and FIG. 8(B) is a diagram showing an example in which the planarity is abnormal. FIG. 8(A) shows the same screen as that shown in FIG. 7(C). When such a screen is displayed, horizontal adjustment is possible, and guidance can be displayed to provide instructions for doing so.

[0084] On the other hand, in Figure 8(B), for example, the sum of the difference value +5 displayed on the right 707b and the difference value +3 displayed on the left 708b does not equal 0. Also, the sum of the difference value -2 displayed on the top 709b and the difference value 0 displayed on the bottom 710b does not equal 0. In such a case, adjusting the tilt of the camera 112 will not make it level, so for example, "Error" is displayed as guidance in 707b and 709b. The information about the tilt mentioned above also includes information about such abnormalities in the tilt.

[0085] In step S614, the fact that an abnormality has occurred is displayed on the screen of the mobile terminal. For example, on the screen of Fig. 8(B), the fact that an abnormality has occurred is notified by blinking five frames 706 to 710. In addition, in step S614, the fact that an abnormality has occurred may be notified to an external server 113 or the like. After displaying or notifying the user of an abnormality in step S614, the process proceeds to step S617. On the other hand, if it is determined in step S613 that no abnormality has occurred, the process proceeds to step S615, where it is determined whether the adjustment method is automatic or not.

[0086] If a motor that rotates the light receiving surface 112a of the image sensor or the camera 112 about the X axis and a motor that rotates it about the Y axis are provided, it is determined in step S615 that the system is automatic. If it is determined that the system is automatic, in step S616, those motors are driven to rotate the camera about the X axis and the Y axis, and a predetermined amount of adjustment is made in the direction in which the difference between the average value of the distance within the frame 706 in the center of the screen and the average value of the distance within the other frames becomes zero.

[0087] For example, in the state shown in Figure 7(C), the motor is driven by a predetermined amount so that it rotates clockwise around the X axis as viewed from the +X direction, and the motor is driven by a predetermined amount so that it rotates counterclockwise around the Y axis as viewed from the +Y direction. Then, the process returns to step S610.

[0088] On the other hand, if such a motor is not provided within the camera 112 or between the camera 112 and the arm 202, it is determined in step S615 that the camera is not automatic, and the process proceeds to step S617. In step S617, it is determined whether or not the return button 713 in FIG. 7C has been clicked. If No, the process returns to step S610, and if Yes, the process proceeds to step S609.

[0089] If the home tab is clicked in step S609, the process returns to step S601 and the home screen is displayed; if the home tab is not clicked, the process returns to step S604 and it is determined whether the camera tab is clicked. As described above, in this embodiment, the display screen (display device) of a mobile terminal or the like displays information about whether the light receiving surface 112a of the image sensor is horizontal (inclination) based on the depth image, and information about the inclination, such as guidance for level adjustment. Furthermore, a distance image is also displayed on the mobile terminal. Therefore, the user of the mobile terminal can easily adjust the level of the camera 112.

[0090] In the above explanation, an example has been described in which each process is executed independently by the information processing device 100. However, some of the processes may be executed independently by the mobile terminal 114. In this case, the information processing device 100 uses HTTP for communication with the mobile terminal 114 and functions as an HTTP server.

[0091] For example, when S601 is executed on the mobile terminal 114, the information on the homepage screen is acquired by accessing the URL of the homepage using the browser of the mobile terminal 114. Alternatively, when a specific application is started, the information is acquired by the mobile terminal 114 accessing the URL of the homepage of the information processing device 100 in accordance with the program of the specific application.

[0092] When information processing device 100 receives an HTTP request from mobile terminal 114 requesting a maintenance homepage screen, it returns an HTML file for the homepage screen as an HTTP response. Upon receiving this, the mobile terminal interprets the HTML file, constructs the homepage screen, and displays it on the display. This is how this step is achieved.

[0093] Furthermore, when S602 is executed on the mobile terminal 114, if YES, the mobile terminal 114 notifies the information processing device 100 that the processing will be terminated, and terminates the application. In this case, the information processing device 100 may wait for a request from the mobile terminal 114 in a standby state, assuming that the application will be executed on the mobile terminal again.

[0094] Furthermore, when S604 is executed by the mobile terminal 114, if YES, the mobile terminal 114 requests the information processing device 100 to acquire a page corresponding to the camera shooting tab. Upon receiving this, the information processing device returns an HTML file indicating the screen layout of the camera shooting tab to the mobile terminal 114 in S607 via S605 and S606. By interpreting and displaying this, the screen of the camera shooting tab is displayed on the screen of the mobile terminal 114.

[0095] Furthermore, when S608 is executed by the mobile terminal 114, if the answer is YES, the mobile terminal 114 requests acquisition of a page corresponding to an enlarged display screen of the depth image. In response to this, the information processing device 100 generates an enlarged depth image in S610, and returns an HTML file corresponding to the page of the enlarged display screen including the calculated value and the guidance superimposed on the enlarged depth image in S611 to the mobile terminal 114. In S612, the mobile terminal 114 interprets and displays the received HTML file, thereby displaying an enlarged screen of the depth image on the screen of the mobile terminal 114.

[0096] Furthermore, when S609 is executed on the mobile terminal 114, if the result is YES, the process returns to S601 and requests the HTML of the home page screen, and if the result is NO, the process returns to S604 and requests the display screen of the camera image. This results in new image and status information being acquired again, and as a result, a screen updated with the latest information is displayed.

[0097] Furthermore, when S613 is executed by the mobile terminal 114, the mobile terminal 114 determines whether the value is abnormal by receiving the difference value from the information processing device 100 after S611. If the answer is YES, in S614 the mobile terminal 114 displays a notification to warn that an abnormal value exists, in accordance with the application program. Also, when S617 is executed by the mobile terminal 114, if NO, the process returns to S610, and the mobile terminal 114 requests the page corresponding to the enlarged display screen again according to the application program, and if YES, the process returns to S609.

[0098] FIG. 9 is a flowchart for explaining the operation of the information processing device 100 during normal operation. The computer in the main board 106 executes a computer program stored in the memory to perform the operations of the steps in the flowchart of FIG.

[0099] In step S901, images are captured at a predetermined cycle (for example, at preset times in the morning and afternoon of each day) using the camera 112. The outputs of the color sensors 109 to 111 are also acquired. Alternatively, sensors for temperature, humidity, air pressure, etc. may be provided in the vicinity of the information processing device 100, and their outputs may be acquired.

[0100] Next, in step S902, an RGB image and a depth image are created. In step S903, the images, the color detection results of the color sensor, and information such as temperature, humidity, and air pressure are sent to an external server 113. Based on the received RGB images, depth images, and color detection results of the color sensor, the server 113 analyzes the growth status of the crops (plants), such as leaf color, number of stems, and plant height, and stores the results in a database together with information such as temperature, humidity, air pressure, date and time.

[0101] In step S904, the information processing device 100 calculates the average value of the distances within the frames 706 to 710 shown in Fig. 7C, and also calculates the difference between the average value of the distances within the frame 706 in the center of the screen and the average values ​​of the distances within the other frames. Furthermore, the screens of Fig. 8A and 8B are generated. The processing in step S904 may be the same as the processing in step S611.

[0102] In step S905, it is determined whether the difference value is abnormal. The determination method may be the same as that in step S613. Note that the result of the abnormality determination in step S905 is determined to be Yes if an abnormality is determined a predetermined number of times. This is because an abnormality occurs in the measurement of whether the camera is horizontal when, for example, a person or animal temporarily enters the shooting angle of view, but such a case is ignored.

[0103] If it is determined in step S905 that an abnormality has occurred, the process proceeds to step S906, where a warning signal is generated, and the process proceeds to step S909. If no abnormality is determined in step S905, the process proceeds to step S907. In step S907, it is determined whether or not the tilt of the horizontal plane of light receiving surface 112a needs to be adjusted. If the difference between the average value of the distance within frame 706 in the center of the screen and the average value of the distance within the other frames is smaller than a predetermined threshold, the result is determined to be No. If the result of step S907 is determined to be Yes, the process proceeds to step S908, and if the result of step S907 is determined to be No, the process returns to step S901.

[0104] In step S908, it is determined whether the adjustment method is automatic. If a motor for rotating the light receiving surface 112a of the image pickup element or the camera 112 around the X axis and a motor for rotating it around the Y axis are provided, it is determined to be automatic in step S908 and the process proceeds to step S909.

[0105] In step S909, these motors are driven to rotate around the X axis and the Y axis, and the difference between the average value of the distance within frame 706 in the center of the screen and the average value of the distance within the other frames is made zero, and the process is then returned to step S901.

[0106] On the other hand, if the above-mentioned automatic adjustment motor is not provided inside the camera 112 or between the camera 112 and the arm 202, it is determined in step S908 that the adjustment is not automatic, and the process proceeds to step S910.

[0107] In step S910, an adjustment instruction signal is sent to the mobile terminal indicating that the tilt of the horizontal plane of the light receiving surface 112a of the image sensor needs to be adjusted, and a screen like the one shown in Fig. 8(A) created in step S904 is also sent. When the mobile terminal receives the adjustment instruction signal, it automatically starts the maintenance mode shown in Fig. 6 and displays the screen shown in Fig. 8(A) created in step S904.

[0108] Furthermore, if a warning signal is generated in step S906, the warning signal is transmitted to the mobile terminal, and a screen such as that shown in Fig. 8(B) created in step S904 is also transmitted. When the mobile terminal receives the warning signal, the maintenance mode shown in Fig. 6 is automatically activated, and the screen shown in Fig. 8(B) created in step S904 is displayed. Also, similar to step S614, five frames, for example, are displayed flashing. Note that some of the steps shown in FIG. 9, for example, steps S904 to S910, may be performed on the server 113 side.

[0109] The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments, and various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.

[0110] A computer program that realizes all or part of the control functions of the present embodiment may be supplied to a computer constituting an information processing system via a network or various storage media. The computer (or CPU, MPU, etc.) in the information processing system may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0111] 100 Information processing device 103 Solar Panels 109~111 Color sensor 112 Camera 113 Server 200 Information Processing Systems 209 Adjustment screws

Claims

1. an image acquisition means for acquiring an image signal from an image acquisition means for capturing an image of a subject from above; an image processing means for generating a distance image based on the imaging signal; tilt detection means for detecting a tilt of the light receiving surface of the imaging means from the horizontal based on the distance image; a control means for notifying information about the tilt detected by the tilt detection means, The information regarding the tilt includes information regarding adjustment of the tilt of the light receiving surface and information regarding an abnormality in the tilt. An information processing system comprising:

2. 2. The information processing system according to claim 1, wherein the information relating to the adjustment includes information relating to at least one of an amount of adjustment and an adjustment direction of the inclination of the light receiving surface.

3. 3. The information processing system according to claim 1, wherein the control means displays the information about the tilt on a display device.

4. 4. The information processing system according to claim 3, wherein the control means causes the display device to display the distance image.

5. 5. The information processing system according to claim 3, wherein the display device is provided in a server or a mobile terminal.

6. An information processing system according to any one of claims 1 to 5, characterized in that the tilt detection means detects the tilt based on distance information at multiple positions in the horizontal scanning direction of the distance image and distance information at multiple positions in the vertical scanning direction of the distance image.

7. the imaging means; a support member that supports the imaging means; 7. The information processing system according to claim 1, further comprising: an inclination adjustment member for adjusting the inclination of the light receiving surface of the imaging means.

8. the tilt adjustment member includes a plurality of screws; 8. The information processing system according to claim 7, wherein the plurality of screws are arranged at positions that allow the light receiving surface to rotate about a rotation axis in the horizontal scanning direction and a rotation axis in the vertical scanning direction of the imaging means.

9. 9. The information processing system according to claim 8, wherein the information regarding the tilt includes information indicating which of the plurality of screws is to be adjusted.

10. 10. The information processing system according to claim 7, further comprising a color sensor for detecting the color temperature of the surroundings.

11. 11. The information processing system according to claim 7, wherein the subject includes a plant.

12. 12. The information processing system according to claim 11, further comprising a measuring means for measuring the height of the plant based on the distance image.

13. 13. The information processing system according to claim 12, wherein the measuring means measures the number of stems of the plant.

14. 13. The information processing system according to claim 12, wherein the measuring means is provided in a server.

15. A computer program for causing a computer to function as each means of the information processing system described in any one of claims 1 to 14.

16. an image acquisition step of acquiring an image signal from an image capturing means that captures an image of a subject from above; an image processing step of generating a distance image based on the imaging signal; an inclination detection step of detecting an inclination of the light receiving surface of the imaging means from the horizontal based on the distance image; a control step of notifying information about the tilt detected by the tilt detection step, The information regarding the tilt includes information regarding adjustment of the tilt of the light receiving surface and information regarding an abnormality in the tilt.

1. A method for controlling an information processing system, comprising:

Images