Crop observation device, control method for crop observation device, and control program for crop observation device

Abstract

A crop observation device according to the present invention includes a camera unit that is installed on a mobile body that moves through a field, and that performs image capturing of a crop that is an observation object, in accordance with movement of the mobile body, and an exposure adjustment unit that adjusts exposure of the camera unit. The exposure adjustment unit outputs a speed change signal for changing movement speed of the mobile body on the basis of an adjustment state of the exposure, or an adjustment state signal regarding the adjustment state of the exposure. According to this configuration, a crop observation device and so forth that can prevent overlooking in observation of an object crop, even in widely varied environments, can be provided.

Description

Crop observation device, control method for crop observation device, and control program for crop observation device

[0001] The present invention relates to a crop observation device, a control method for a crop observation device, and a control program for a crop observation device.

[0002] Recently, the use of IT in agriculture has been rapidly progressing. As part of this IT use, development of a technology for timely understanding the growth status of agricultural crops planted in a field by capturing images of the crops using a camera unit is underway (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2023-095736

[0004] If multiple camera units are installed in a field where the crops to be observed are planted, images suitable for observation can be obtained by setting imaging conditions specialized for a specific observation target. However, if you try to grasp the growth status of crops in a particularly large field without missing anything, a huge number of camera units will be required, which not only increases operating costs but also increases the scale of the system.

[0005] To address this issue, a method is being developed in which a camera unit is mounted on a mobile robot that moves autonomously within a field, and images of the object being observed are captured sequentially as the robot moves autonomously. However, with this method, the object being observed changes sequentially as the mobile robot moves, so the imaging conditions of the camera unit must be adjusted sequentially. For example, when capturing an image of fruit hanging from a branch from above, the image capture range may include the fruit as well as the sky, resulting in a significant difference in brightness within the angle of view, and it may take a long time to adjust the exposure before capturing an appropriate image.

[0006] If it takes a long time to adjust the exposure, the mobile robot will continue to move during that time, which will result in the robot passing over the crops being observed and not being able to capture images of them.If it is not possible to capture images of even a part of the crops being observed, it will be impossible to accurately grasp the growth status of the entire field or estimate the harvest yield, or it will be impossible to discover poor growth in specific areas.

[0007] The present invention has been made to solve such problems, and provides a crop observation device etc. that can prevent target crops from being overlooked even under diverse and changing environments.

[0008] A first aspect of the present invention provides a crop observation device mounted on a mobile body moving through a field. The device includes a camera unit that captures images of the crops being observed as the mobile body moves, and an exposure adjustment unit that adjusts the exposure of the camera unit. The exposure adjustment unit outputs a speed change signal that changes the speed of the mobile body based on the exposure adjustment status. With a crop observation device having this configuration, if exposure adjustment takes a long time, the mobile body's speed can be slowed or stopped, ensuring that images of the target crops are captured without missing any. Furthermore, if exposure adjustment is successful, the mobile body's speed can be restored, thereby contributing to shortening work time.

[0009] In the above-described crop observation device, the exposure adjustment unit may recognize the sky area captured in the captured image output by the camera unit and adjust the exposure while allowing for overexposure in the sky area. The crop observation device according to the present invention only requires that the target crop be captured with the correct exposure, so even if the sky area is included in the overhead image, it is acceptable for that area to be overexposed. Relaxing the exposure adjustment conditions in this way is expected to speed up exposure adjustment. Conversely, it is desirable for the exposure adjustment unit to adjust the exposure without allowing for overexposure in non-sky areas other than the sky area. In many cases, areas other than the sky area are occupied by the crops being observed. Therefore, adjusting the non-sky areas to the correct exposure can ultimately be expected to speed up exposure adjustment, even without determining that the non-sky areas are strictly crop areas.

[0010] The exposure adjustment unit may also recognize the sky area by detecting the difference between multiple captured images continuously output by the camera unit while changing the imaging conditions. Since exposure adjustment images can be continuously captured during exposure adjustment, the sky area can be accurately recognized by using these images. The exposure adjustment unit may also correct the recognition of the sky area based on a designated area indicated by a user in the captured image. This correction can prevent, for example, a crop area illuminated by direct sunlight from being recognized as the sky area. The exposure adjustment unit may also correct the recognition of the sky area based on an index specified by a user regarding the crops in the captured image. For example, if the index is the number of fruits within the field of view, and the number of fruits counted by the exposure adjustment unit in the captured image differs from the number of fruits designated by the user after observing the captured image, the exposure adjustment unit corrects the recognition by scaling the sky area so that the two counts match. This correction allows the exposure adjustment unit to recognize a more appropriate range as the sky area.

[0011] The crop observation device may also have a plurality of camera units, an image processing unit that sets a common fixed white balance for the plurality of camera units, and an exposure adjustment unit that adjusts the exposure of each camera unit individually. Even when a plurality of camera units are provided, it is preferable that the white balance of each captured image be the same, taking into consideration subsequent analysis processing, etc. However, since each camera unit captures a different crop, it is preferable that the exposure adjustment be adjusted individually.

[0012] The crop observation device may also include an illuminance sensor that acquires illuminance information in the direction of movement of the moving object, and the exposure adjustment unit may adjust the exposure using the illuminance information. Moving objects often include sensors, such as cameras, that acquire environmental information in the direction of movement. These sensors can essentially function as illuminance sensors, so by using this illuminance information, the exposure adjustment unit can provisionally set an exposure adjustment value that matches the environmental illuminance in advance. This ultimately allows exposure adjustment to be completed more quickly.

[0013] The crop observation device may further include a position information acquisition unit that acquires position information of the mobile object, and the exposure adjustment unit may determine initial conditions for exposure adjustment based on the current position information acquired from the position information acquisition unit and past exposure adjustment information for the same position. In this way, by using past adjustment information, exposure adjustment can be completed as quickly as possible.

[0014] A second aspect of the present invention provides a control method for a crop observation device, the control method including a camera unit mounted on a mobile body moving through a field and capturing images of crops as the mobile body moves, and an exposure adjustment unit that adjusts the exposure of the camera unit, the control method including an output step in which the exposure adjustment unit outputs a speed change signal that changes the speed of movement of the mobile body based on the exposure adjustment status. A third aspect of the present invention provides a control program for a crop observation device, the control program including a camera unit mounted on a mobile body moving through a field and capturing images of crops as the mobile body moves, and an exposure adjustment unit that adjusts the exposure of the camera unit, the control program including an output step in which the exposure adjustment unit outputs a speed change signal that changes the speed of movement of the mobile body based on the exposure adjustment status. The second and third aspects of the present invention are also expected to provide the same effects as the first aspect.

[0015] The present invention can provide a crop observation device and the like that can prevent target crops from being overlooked even in diversely changing environments.

[0016] FIG. 1 is a conceptual diagram illustrating an overall picture of farm field management incorporating IT technology. FIG. 2 is a conceptual diagram showing how a field mobile robot equipped with a crop observation device according to the present embodiment moves through a field. FIG. 3 is a system configuration diagram of the crop observation device. FIG. 4 is a diagram schematically showing an example of a captured image. FIG. 5 is a flow diagram explaining the processing of the crop observation device. FIG. 6 is a conceptual diagram showing how a field mobile robot equipped with a crop observation device according to a first modified example moves through a field. FIG. 7 is a system configuration diagram of a crop observation device according to a second modified example. FIG. 8 is a conceptual diagram showing how a field mobile robot equipped with a crop observation device according to a third modified example moves through a field. FIG. 9 is a system configuration diagram of a crop observation device according to the third modified example.

[0017] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0018] Figure 1 is a conceptual diagram illustrating the overall picture of farm field management incorporating IT technology. In the figure, the farm field is a cultivation area where agricultural products are grown, and it is not limited to outdoor cultivation areas, but may also be a facility with a roof, such as a greenhouse or plant factory.

[0019] The work robot 10 is placed in a field and moves autonomously within the field, operating in response to operator input. The work robot 10 may be equipped with a sensor unit 20 for observing the growth status of agricultural crops, and a working unit 30 for performing tasks on the crops and the field. The sensor unit 20 may be, for example, a crop observation device for observing the target crops, as described below, or a GPS sensor for identifying the position of the work robot 10. The position of the work robot 10 may be identified using a method such as SLAM, which compares detection results from LiDAR or the like with map data. The working unit 30 may be, for example, a harvesting device for picking fruit that has reached the harvesting stage, or a spraying device for spraying pesticides.

[0020] The work robot 10 is equipped with a control unit 11 for controlling its own movement and for processing in cooperation with the onboard sensor unit 20 and work unit 30. It also has a communications interface (hereinafter referred to as "communication I / F") 12 for exchanging information and instructions with a server 50 that manages the field and a user terminal 60 carried by the user. The control unit 11 receives work instructions sent over a network 90 via the communications I / F 12, and also transmits information about the field and crops, such as image data obtained from the sensor unit 20, along with time and location information, over the network 90.

[0021] The sensor 40 is a measuring device that measures meteorological data, such as field temperature, humidity, illuminance, wind direction, and wind speed, as well as soil data, such as soil moisture, temperature, nutrients (EC value), and pH value, and is used to observe the condition of the field and crops. An appropriate number of sensors 40 are installed in the field depending on the object of observation. The sensor 40 may also be linked to the work robot 10. The sensor 40 receives sensing instructions from the server 50 or the user terminal 60 or acquires observation data according to a set sensing cycle. The observation data acquired by the sensor 40 is transmitted to the server 50 or the user terminal 60 via the network 90 as information related to the field, the environment, and the crops.

[0022] The server 50 functions as a management device for comprehensively managing the entire farm field. Specifically, it issues work instructions and provides requested information to the work robot 10, sensors 40, and user terminal 60 via the communication I / F 51 and network 90. ​​It also acquires information about the farm field, crops, and environment from the work robot 10, sensors 40, and user terminal 60. The calculation unit 52 is responsible for control calculations for the entire system, and in particular serves as an instruction unit that executes instructions to the work robot 10, etc., a processing unit that performs various processes on acquired information, and a prediction unit that predicts crop yields and workloads.

[0023] The value database 53 is a database for systematically storing data such as various information and analysis results accumulated in farm field management. For example, it stores crop data, which is data on the growth rate and yield of target crops associated with past dates and positions within the farm field, and operational data, which is data on the amount of labor required to perform specific farm work. The server 50 also serves as a value data platform that manages the value database 53.

[0024] The user terminal 60 is a tablet terminal or smartphone carried by a worker who performs agricultural work in the field or a manager who manages the field. The user terminal 60 issues work instructions and provides requested information to the work robot 10, sensor 40, and server 50 via the network 90. ​​Similarly, the user terminal 60 obtains information about the field, crops, and environment from the work robot 10, sensor 40, and server 50.

[0025] By adopting a farm management system constructed in this way, producers can achieve more productive farm operations. Specifically, they can automate crop management and processing to increase yields, and they can adjust growth and allocate work loads to reduce the workload of workers.

[0026] 2 is a conceptual diagram showing a field mobile robot 200 equipped with a crop observation device 100 according to this embodiment moving through a field. The crop observation device 100 is one aspect of the sensor unit 20, and is mounted on the field mobile robot 200. The field mobile robot 200 is one aspect of the work robot 10, and is a mobile body that can move autonomously through a field.

[0027] The crop observation device 100 is a device that captures images of agricultural crops grown in a field in order to observe the growth status of the crops, more specifically, for the server 50 to analyze the growth status. In this embodiment, an example of an observation target in which the problem is particularly pronounced is assumed to be a crop in which the fruit hangs from a cultivation shelf, such as grapes or kiwi. When observing such a crop, the camera unit 110 of the crop observation device 100 is installed relative to the field mobile robot 200 so as to overlook the crop. In other words, the camera unit 110 is installed so that the optical axis of the camera unit 110 forms an elevation angle with respect to the direction of movement of the field mobile robot 200 (indicated by the white arrow in the figure).

[0028] During the growth process, crops grow stems, grow leaves, and mature fruits. Therefore, assuming that the crop observation device 100 periodically captures images of the crops, the proportion of the crops occupying the field of view changes each time. Furthermore, especially when capturing images of crops from above, the field of view is dominated by the sky, with the exception of the crops. In other words, the image area of ​​the captured image generated by the crop observation device 100 is mostly occupied by the crop area and the sky area, and this proportion changes with each capture. Furthermore, crops do not grow uniformly on the cultivation shelves but often grow unevenly to some extent. Therefore, as shown in the figure, areas where the crops are densely packed are dark and areas where they are sparse are bright. Naturally, the brightness of the sky area also changes depending on the time of day when the image is captured and the weather on that day. In other words, the subject captured by the crop observation device 100 has a large difference in brightness between areas within the field of view, and this difference in brightness changes from moment to moment depending on the situation.

[0029] The crop observation device 100 performs exposure adjustment to adjust the exposure of the camera unit 110 prior to imaging processing in order to obtain a captured image with a proper exposure. However, when capturing such agricultural crops as the subject, it may take more time than usual to determine the appropriate exposure value. The exposure values ​​to be determined include, for example, the aperture value, shutter speed, and ISO sensitivity, and the proper exposure is achieved by combining these. The aperture value corresponds to the size of the aperture opening set when the camera unit 110 has a variable aperture. The shutter speed specifies the exposure time, which is the time it takes for the photoelectric conversion unit of the image sensor to accumulate subject light. The ISO sensitivity is converted into the amplification factor of the photoelectrically converted image signal.

[0030] The crop observation device 100 continuously acquires multiple preliminary images while varying the imaging conditions and calculates the appropriate exposure value by detecting changes in brightness between the preliminary images. However, when a bright sky area and a dark crop area coexist within the field of view, performing a normal exposure adjustment that averages the brightness values ​​within the field of view results in the crop area being adjusted to be significantly darker due to the influence of the brightness of the sky area. Captured images in which the crop area is darkened are undesirable for subsequent analysis and may lead to erroneous analysis results. Therefore, exposure adjustment is required to ensure that the crop area is appropriately bright. However, the crop observation device 100 requires more preliminary images to perform such exposure adjustment. Therefore, the crop observation device 100 acquires more preliminary images and performs appropriate exposure adjustment, which requires more time than exposure adjustment when the brightness difference within the field of view is within a certain range.

[0031] If it takes a long time to adjust the exposure, the mobile robot will continue to move during that time, which will result in the robot passing over the crops being observed and not being able to capture images of them.If it is not possible to capture images of even a part of the crops being observed, it will be impossible to accurately grasp the growth status of the entire field or estimate the harvest yield, or it will be impossible to discover poor growth in specific areas.

[0032] Therefore, the crop observation device 100 in this embodiment outputs a speed change signal to the speed control unit 220 of the field mobile robot 200 to change the movement speed of the field mobile robot 200 based on the exposure adjustment status. In other words, if the crop observation device 100 determines that exposure adjustment will take time, it outputs a speed change signal to reduce or stop the speed V of the field mobile robot 200, thereby preventing the field mobile robot 200 from passing over the crops included in the observation target. When the speed control unit 220 receives the speed change signal from the crop observation device 100, it adjusts the drive signal to the drive wheels 230 to reduce or stop the field mobile robot 200.

[0033] After determining the appropriate exposure, the crop observation device 100 sets the camera unit 110 accordingly, captures an image, and acquires the captured image. The image data of the acquired captured image is transmitted to the server 50 and the user terminal 60 via the communication I / F 210 of the field mobile robot 200.

[0034] In this embodiment, the field mobile robot 200 is assumed to be a wheeled mobile body that employs drive wheels 230, but it is not limited to wheeled mobile bodies, and may be a crawler-type or leg-walking mobile body, or may be an airborne mobile body such as a drone.

[0035] FIG. 3 is a system configuration diagram of the crop observation device 100. The crop observation device 100 is mainly composed of a camera unit 110, a control unit 120, and a storage unit 130. The camera unit 110 includes a lens 111, a variable aperture 112, and an image sensor 113. The lens 111 is a lens for forming an image of a subject on the imaging plane of the image sensor 113, and is generally composed of multiple lenses. The lens 111 may include a focus lens that moves for focus adjustment, or may include a zoom lens that changes the focal length. If the lens includes a focus lens, it may be adjusted to focus only on the crop to be observed, and if the lens includes a zoom lens, it may be adjusted to exclude the sky.

[0036] The variable aperture 112 is a mechanism that limits the amount of light entering the lens 111 according to a set aperture value. The image sensor 113 is, for example, a CMOS sensor that photoelectrically converts a subject image to generate an image signal. The image sensor 113 changes the amplification gain used to generate the image signal according to the set ISO sensitivity.

[0037] The control unit 120 is a processor (CPU: Central Processing Unit) that controls the crop observation device 100 and executes programs. The processor may be configured to work in conjunction with a processing chip such as an ASIC (Application Specific Integrated Circuit) or a GPU (Graphics Processing Unit). The control unit 120 reads out a control program stored in the storage unit 130 and executes control to appropriately acquire and output crop images of the target crop.

[0038] The control unit 120 also serves as a functional calculation unit that executes various calculations in response to processing instructed by a control program. The control unit 120 can function as an image processing unit 121, a position information acquisition unit 122, and an exposure adjustment unit 123, among others.

[0039] The image processing unit 121 receives the image signal output by the imaging sensor 113 and performs interpolation processing, brightness adjustment, white balance adjustment, etc. as necessary to generate a captured image. When the image processing unit 121 generates a preliminary image for adjusting exposure, it passes the preliminary image to the exposure adjustment unit 123. When the image processing unit 121 generates a crop image of the crop to be observed, it generates image data converted into a predetermined format and outputs it to the field mobile robot 200. Note that if the crop observation device 100 is equipped with a communication I / F, it may output the image data directly to the server 50 or the user terminal 60.

[0040] The position information acquisition unit 122 acquires position information relating to the current position of the farmland mobile robot 200 in the farmland from a GPS sensor or the like provided in the farmland mobile robot 200. The acquired position information is passed to the exposure adjustment unit 123.

[0041] The exposure adjustment unit 123 adjusts the exposure of the camera unit 110. Specifically, the camera unit 110 continuously executes the image capture process while changing the image capture conditions, sequentially acquires preliminary images from the image processing unit 121, and calculates an appropriate exposure value by detecting changes in brightness between the preliminary images. At this time, a speed change signal is output to change the movement speed of the field mobile robot 200 based on the exposure adjustment status. Specifically, for example, when the normal movement speed V ０ On the other hand, the image capturing interval to obtain image data for all the crops to be observed is T ０ If T ０ If it is determined that the exposure adjustment is not completed within the time limit, a speed change signal is output. The output speed change signal is a signal indicating the movement speed V to be reduced depending on the exposure adjustment status. Ｌ It can be a signal that specifies a stop or a stop signal that causes a stop.

[0042] The exposure adjustment unit 123 sends an aperture control signal that adjusts the aperture value to the variable aperture 112 according to the determined exposure value, and sends a sensor control signal that adjusts the shutter speed and ISO sensitivity to the image sensor 113. Note that the camera unit 110 may not be equipped with the variable aperture 112, in which case the exposure adjustment unit 123 determines appropriate values ​​for the shutter speed and ISO sensitivity to be adjusted, assuming a fixed aperture value.

[0043] In addition to storing the control program described above, the storage unit 130 also stores adjustment information 131 related to past exposures at each position set in the field. Each piece of adjustment information 131 describes exposure values ​​determined at a particular position and time in the past, associated with that position and time, and an exposure adjustment database is constructed in the storage unit 130. The exposure adjustment unit 123 references the exposure adjustment database using the position information acquired via the position information acquisition unit 122, the current time, and other parameters, and extracts corresponding adjustment information. Then, by referencing the exposure values ​​described in the extracted adjustment information, the storage unit 130 determines the initial conditions for adjusting exposure to obtain a preliminary image, i.e., the initial values ​​for aperture value, shutter speed, and ISO sensitivity. At this time, the exposure values ​​determined to obtain the immediately preceding captured image may be taken into consideration.

[0044] Next, a more specific description of exposure adjustment will be given. Fig. 4 is a diagram schematically illustrating an example of a captured image. As described above, when an image of a crop to be observed is captured from above, the captured image may include a sky area in which the sky is captured. The crop area in which the crop to be observed is captured is included in a non-sky area, which is an area other than the sky area, and substantially coincides with the non-sky area.

[0045] In general exposure adjustment, the exposure value is often determined by referring to the average brightness value of the entire preliminary image so that the brightness of the entire captured image is balanced. If an image is captured using the exposure value determined by such exposure adjustment, the resulting captured image will be affected by the brightness of the sky area, resulting in a relatively underexposed crop area.

[0046] Because the crop observation device 100 observes agricultural crops, it is important that the crop region in the captured image is properly exposed. Therefore, the exposure adjustment unit 123 recognizes sky regions in the captured image output by the camera unit 110 and adjusts the exposure so as not to allow crushed black in non-sky regions other than the sky region. In other words, the exposure adjustment unit 123 determines an exposure value so that crushed black in non-sky regions does not occur. By preventing crushed black in non-sky regions, the crop observation device 100 can provide captured images suitable for the purpose of observing agricultural crops to the server 50 and the user terminal 60. Such captured images significantly contribute to the accuracy of counting, for example, when the server 50 counts the number of observed fruits using image discrimination processing.

[0047] The exposure adjustment unit 123 may employ various methods for recognizing a sky region from a preliminary image output by the camera unit. For example, a region including pixels exhibiting a brightness value exceeding a threshold ratio relative to the average brightness value of the entire preliminary image may be defined as a sky region. Alternatively, the camera unit 110 may detect differences between multiple preliminary images successively output by changing the imaging conditions, and define, for example, a region that is continuously blown out as a sky region. Alternatively, the exposure adjustment unit 123 may transmit the preliminary image to the user terminal 60, have the user indicate a sky region via the user terminal 60, and then determine the sky region by receiving the indicated region.

[0048] The exposure adjustment unit 123 may correct the recognized sky area by acquiring additional information to more accurately recognize the sky area. Specifically, the exposure adjustment unit 123 can correct the recognized sky area based on an index specified by the user regarding the observed crops shown in the preview image. For example, if the index is set to the number of observed fruits to be counted, the exposure adjustment unit 123 can correct the width of the sky area so that the number automatically counted by the image discrimination process matches the specified number counted by the user by visually recognizing the preview image displayed on the user terminal 60.

[0049] The exposure adjustment unit 123 may perform exposure adjustment so as not to allow crushed black in non-sky regions as described above, but may also focus on sky regions and perform exposure adjustment while allowing blown-out highlights in the sky regions, thereby suppressing crushed black in the non-sky regions. That is, the exposure adjustment unit 123 may determine the exposure value by relaxing the exposure adjustment conditions to allow blown-out highlights in the sky regions. Furthermore, in the above example, the exposure adjustment unit 123 performed exposure adjustment so as not to cause crushed black in non-sky regions. However, when the observation target within the non-sky regions is further narrowed down, for example, when the observation target is fruit to be harvested or a discolored area causing growth disorders, the exposure adjustment may be performed so as not to cause crushed black in the observation target region.

[0050] Furthermore, while the above example assumes that crops are imaged during the day, crops may also be imaged at night. When imaging at night, the crops being observed are illuminated by lighting, so there is a large difference in brightness within the angle of view, and it is expected that it will take time to adjust the exposure before imaging in order to obtain an appropriate image. When imaging at night, exposure adjustment may be performed while allowing for blackout in the sky area. Conversely, while illuminated crops are prone to whiteout, it is preferable to adjust the exposure so that whiteout in non-sky areas does not occur.

[0051] Next, we will explain the processing procedure executed by the control unit 120 of the crop observation device 100. Figure 5 is a flow diagram explaining the processing procedure. The flow starts when the farmland mobile robot 200 starts moving and the crop observation device 100 is about to start capturing images of the crops to be observed.

[0052] In step S101, the position information acquisition unit 122 acquires current position information and passes it to the exposure adjustment unit 123. In step S102, the exposure adjustment unit 123 extracts adjustment information 131 corresponding to the received position information and the current time from the storage unit 130 and determines initial conditions for adjusting exposure to obtain a preliminary image. At this time, if a captured image to be output as image data to the field mobile robot 200 has just been processed, the initial conditions may be determined taking into account the exposure value determined to obtain that captured image.

[0053] In the subsequent step S103, the exposure adjustment unit 123 sequentially changes the imaging conditions from the determined initial conditions, causes the camera unit 110 to continuously output preliminary images, and adjusts the exposure according to the brightness information obtained from the preliminary images. Specifically, it executes a calculation process to determine an appropriate value. After a preset time has elapsed, the exposure adjustment unit 123 proceeds to step S104 to check whether the exposure adjustment is complete. If not, the exposure adjustment unit 123 proceeds to step S105, where the exposure adjustment unit 123 outputs a speed change signal to the speed control unit 220 of the field mobile robot 200 to request a reduction in the movement speed. Thereafter, the exposure adjustment unit 123 returns to step S103 to continue the exposure adjustment. If it is determined in step S104 that the exposure adjustment is complete, the exposure adjustment unit 123 proceeds to step S106. Note that, after completing the exposure adjustment in step S103, the exposure adjustment unit 123 may proceed to step S104 and immediately proceed to step S106 without waiting for the preset time to elapse.

[0054] As described above, by repeating step S105 in response to delays in exposure adjustment, the movement speed of the field mobile robot 200 can be gradually reduced. By gradually reducing the movement speed in this way, a balance can be achieved between the acquisition of complete image data and work efficiency. However, the request to reduce the movement speed is not limited to this gradual request. For example, if emphasis is placed on the reliability of image data acquisition, a speed change signal requesting an immediate stop may be output when it is determined that exposure adjustment is taking a long time.

[0055] In step S106, the exposure adjustment unit 123 controls the variable aperture 112 and the image sensor 113 according to the determined exposure value, causing the camera unit 110 to capture an observation image. If the acquired observation image is not properly exposed, the exposure adjustment may be performed again. Next, in step S107, the exposure adjustment unit 123 determines whether the field mobile robot 200 is still moving at a low speed, having previously output a speed change signal requesting a speed reduction. If it determines that the field mobile robot 200 is moving at a low speed, the process proceeds to step S108, where the exposure adjustment unit 123 outputs a speed change signal to the speed control unit 220 of the field mobile robot 200 to gradually restore the moving speed until the field mobile robot 200 reaches the predetermined speed set for the field mobile robot 200. Then, the process proceeds to step S109. If it is determined in step S107 that the farmland mobile robot 200 is moving at the predetermined speed set for the farmland mobile robot 200, step S108 is skipped and the process proceeds to step S109.

[0056] In step S109, the control unit 120 checks whether image data for all areas of the imaging target specified by the server 50 or the field mobile robot 200 has been output. If there are areas to be imaged, the process returns to step S101 and continues the series of processes. If image data output for all areas has been completed, the process ends. Note that the control unit 120 may end the series of processes when it receives an end instruction signal from the server 50 or the field mobile robot 200.

[0057] Next, another embodiment that is a modification of the crop observation device 100 will be described. Fig. 6 is a conceptual diagram showing a field mobile robot 200 equipped with a crop observation device 100' according to a first modified embodiment moving through a field. While Fig. 2 shows the field mobile robot 200 observed from the side, Fig. 6 shows the field mobile robot 200 observed from the front. Note that, apart from the components described below, the configuration of the crop observation device 100' is the same as that of the crop observation device 100, and therefore, description thereof will be omitted unless otherwise noted.

[0058] When operating a work robot 10 in a field, an aisle is typically provided for the work robot 10 to move through. The aisle is also used as a path for workers to travel through. Crops are placed on one or both sides of the aisle. When crops are placed on both sides of the aisle as shown in FIG. 6 , if images of the crops on the right and left sides of the field mobile robot 200 traveling along the aisle can be captured in parallel, image data of all the crops can be acquired in a shorter time or with higher resolution. Therefore, the crop observation device 100′ includes two camera units: one for capturing images of the crops on the right side of the traveling direction, and one for capturing images of the crops on the left side.

[0059] In such a configuration with two camera units 110, the exposure adjustment unit 123 individually adjusts the exposure of each camera unit 110. That is, the exposure adjustment unit 123 performs exposure adjustment so that the crops as subjects captured by each camera unit 110 are properly exposed. At this time, if the exposure adjustment unit 123 is taking a long time to adjust the exposure for at least one of the camera units 110, it outputs a speed change signal to slow down the movement speed of the field mobile robot 200.

[0060] Meanwhile, the image processing unit 121 generates captured images using a common fixed white balance for the image signals sent from each camera unit 110. That is, captured images are generated using a preset fixed white balance value. By processing in this manner, the captured images obtained from each camera unit 110 are properly exposed for the respective crops as subjects and are adjusted to a common color tone. If the image data used by the server 50 to analyze the growth status, etc., is generated from such captured images, it is expected that the error rate will be low and more accurate analysis results will be obtained. Note that the number of camera units 110 included in the crop observation device 100' is not limited to two and may be three or more. By sharing the image capture of the crops to be observed among many camera units 110, image data of the entire crop can be obtained in a shorter time or with higher resolution.

[0061] The crop observation device 100′ may also include an illuminance sensor 140 installed in front of the field mobile robot 200. By including such an illuminance sensor 140, information about the brightness in the direction of movement of the field mobile robot 200 can be acquired in advance, allowing the exposure adjustment unit 123 to more accurately determine the initial conditions for exposure adjustment to obtain a preliminary image. This in turn reduces the time required for exposure adjustment. The illuminance sensor 140 may be a forward observation camera unit that the field mobile robot 200 is equipped with to acquire information about the forward environment. In this case, the exposure adjustment unit 123 may directly receive the captured image output by the forward observation camera unit, or may receive information about brightness extracted from the captured image.

[0062] In the crop observation devices 100, 100' described above, the control unit 120 and memory unit 130, along with the camera unit 110, are mounted on the field mobile robot 200. However, at least some of these components, excluding the camera unit 110, may be incorporated into the server 50 or other external device. In this case, the camera unit 110 may exchange control signals and image signals with the control unit 120 via wireless communication, for example. Alternatively, the crop observation devices 100, 100' may be configured integrally with the field mobile robot 200. In this case, the camera unit 110 may function as a camera unit that recognizes the surrounding environment for the field mobile robot 200 to move.

[0063] Furthermore, the exposure adjustment unit 123 described above outputs a speed change signal to reduce the movement speed when it determines that exposure adjustment will take a long time, but this is not limited to this. The speed change signal may also be output to reduce the movement speed when the determined shutter speed is slower than a predetermined shutter speed. By reducing the movement speed of the field-mobile robot in this way, it is expected that blurring of the captured images will be reduced.

[0064] Another embodiment will now be described. Fig. 7 is a system configuration diagram of a crop observation device 100a according to a second modified example. The crop observation device 100a differs from the crop observation device 100 in that, while the exposure adjustment unit 123 of the crop observation device 100 outputs a speed change signal that changes the movement speed of the field mobile robot 200 based on the exposure adjustment status, the crop observation device 100a includes an exposure adjustment unit 123a that outputs an adjustment status signal related to the exposure adjustment status. The other configurations are the same as those described using Fig. 3.

[0065] The exposure adjustment unit 123a adjusts the exposure of the camera unit 110. Specifically, the exposure adjustment unit 123a continuously executes image capture processing while changing the image capture conditions of the camera unit 110, sequentially acquires preliminary images from the image processing unit 121, and calculates an appropriate exposure value by detecting changes in brightness between the preliminary images. At this time, an adjustment status signal relating to the exposure adjustment status is output to the speed control unit 220 of the field mobile robot 200.

[0066] The adjustment status signal is, for example, "0" indicating that the exposure value has been adjusted if it has been determined within a specified time that does not interfere with the image capture process, and "1" indicating that the exposure value has not been determined within the specified time. The specified time may be a fixed time that is set in advance, or may be a time that can change depending on the movement speed of the field mobile robot 200.

[0067] When the speed control unit 220 receives an adjustment status signal of "0," it adjusts the movement speed to a preset speed or a speed appropriate to the surrounding environment, regardless of the operating status of the crop observation device 100. On the other hand, when it receives an adjustment status signal of "1," it slows the movement speed to a preset speed or stops the movement. The adjustment status signal may be set in stages when the exposure value cannot be determined within a specified time. For example, the exposure adjustment unit 123a may determine the difficulty of determining the exposure value based on conditions such as low contrast in the preview image or a sudden change in brightness, and output an adjustment status signal in five levels from "1" to "5" depending on the difficulty. The speed control unit 220 can reduce the movement speed more as the value of the received adjustment status signal increases.

[0068] The exposure adjustment unit 123a is similar to the exposure adjustment unit 123 in the process of transmitting an aperture control signal that adjusts the aperture value to the variable aperture 112 and transmitting a sensor control signal that adjusts the shutter speed and ISO sensitivity to the image sensor 113 in accordance with the determined exposure value. The processing steps for adjusting the exposure value, which have already been described, are also similar to those of the exposure adjustment unit 123.

[0069] 8 is a conceptual diagram showing how a field mobile robot 200 equipped with a crop observation device 100b according to a third modified example moves through a field. The field mobile robot 200 is equipped with an auxiliary camera unit 150 that captures images of crops located in the direction of movement (forward as indicated by the white arrow) before the camera unit 110. The auxiliary camera unit 150 has its own exposure adjustment section, determines an exposure value for capturing images of crops located in the direction of movement, executes the image capture process, and transmits the exposure value to the crop observation device 100b as exposure adjustment information.

[0070] The crop observation device 100b acquires speed information relating to the current movement speed from the speed control unit 220 of the field mobile robot 200. The crop observation device 100b also transmits the same adjustment status signal as the crop observation device 100a to the speed control unit 220. The rest of the configuration is the same as the configuration described using Figure 2, so a detailed description will be omitted.

[0071] 9 is a system configuration diagram of a crop observation device 100b according to the third modified example. The crop observation device 100b differs from the crop observation device 100a according to the second modified example in that the exposure adjustment unit 123b adjusts exposure using exposure adjustment information sent from the auxiliary camera unit 150 and movement speed information sent from the speed control unit 220. The other configurations are the same as those described using FIGS. 3 and 7.

[0072] Specifically, the exposure adjustment unit 123b uses the movement speed information sent from the speed control unit 220 to calculate the timing at which the camera unit 110 will capture an image of the crop captured by the auxiliary camera unit 150. The exposure reference value for exposure adjustment of the image capture process at that timing is then set to the exposure value included in the exposure adjustment information sent from the auxiliary camera unit 150. In this way, by referring to the exposure value of the auxiliary camera unit 150, which captures an image of the observation target first, the exposure value to be applied to the camera unit 110 can be determined more quickly and appropriately.

[0073] The exposure adjustment information does not have to be the exposure value itself, but may be, for example, an EV value calculated by the auxiliary camera unit 150 or sample image data with thinned pixels. Such information can also serve as reference information for determining the above-mentioned exposure reference value in advance. In the illustrated example, the exposure adjustment unit 123b outputs an adjustment status signal to the speed control unit 220, similar to the above-mentioned exposure adjustment unit 123a, but may also output a speed change signal similar to that of the exposure adjustment unit 123. In this case, the speed control unit 220 changes the movement speed in accordance with the received speed change signal.

[0074] The exposure adjustment unit 123b determines an exposure value by adjusting it within a limited range (for example, within a range of plus or minus two stops) relative to the determined exposure reference value. The exposure adjustment unit 123b is similar to the exposure adjustment unit 123 in the process of transmitting an aperture control signal that adjusts the aperture value to the variable aperture 112 and transmitting a sensor control signal that adjusts the shutter speed and ISO sensitivity to the image sensor 113 in accordance with the determined exposure value. The processing steps for adjusting the exposure value, which have already been described, are also similar to those of the exposure adjustment unit 123.

[0075] Here, the main components of the crop observation device, crop observation device control method, and crop observation device control program described above are summarized. [Appendix 1] A crop observation device (100, 100', 100a, 100b) includes: a camera unit (110) mounted on a mobile object (200) that moves through a field and captures images of a crop being observed in accordance with the movement of the mobile object; and an exposure adjustment unit (123, 123a, 123b) that adjusts the exposure of the camera unit, wherein the exposure adjustment unit outputs a speed change signal that changes the movement speed of the mobile object based on the exposure adjustment status or an adjustment status signal related to the exposure adjustment status. [Appendix 2] The crop observation device described in Appendix 1, wherein the exposure adjustment unit recognizes a sky area captured from the captured image output by the camera unit and adjusts the exposure while allowing for overexposure in the sky area. [Supplementary Note 3] The crop observation device according to Supplementary Note 1, wherein the exposure adjustment unit recognizes a sky region captured from the captured image output by the camera unit, and adjusts the exposure without allowing crushed black in non-sky regions other than the sky region. [Supplementary Note 4] The crop observation device according to Supplementary Note 2, wherein the exposure adjustment unit recognizes the sky region by detecting a difference between the multiple captured images continuously output by the camera unit while changing the imaging conditions. [Supplementary Note 5] The crop observation device according to Supplementary Note 2, wherein the exposure adjustment unit corrects the recognition of the sky region based on a designated region designated by a user in the captured image. [Supplementary Note 6] The crop observation device according to Supplementary Note 2, wherein the exposure adjustment unit corrects the recognition of the sky region based on an index designated by a user regarding the crops appearing in the captured image. [Supplementary Note 7] The crop observation device according to Supplementary Note 1, wherein the crop observation device has a plurality of camera units, and comprises an image processing unit (121) that sets a common fixed white balance for the multiple camera units, and the exposure adjustment unit individually adjusts the exposure of each of the camera units. [Supplementary Note 8] The crop observation device according to Supplementary Note 1, further comprising an illuminance sensor (140) that acquires illuminance information in the moving direction of the moving body, and the exposure adjustment unit adjusts the exposure using the illuminance information.[Supplementary Note 9] The crop observation device according to Supplementary Note 1, comprising: a position information acquisition unit (122) that acquires position information of the moving body, and the exposure adjustment unit determines initial conditions for the exposure adjustment from current position information acquired from the position information acquisition unit and the exposure adjustment information (131) for the same position in the past. [Supplementary Note 10] A control method for a crop observation device (100, 100', 100a, 100b) that is mounted on a moving body (200) that moves in a field and comprises a camera unit (110) that captures images of a crop that is an observation target in accordance with the movement of the moving body, and an exposure adjustment unit (123, 123a, 123b) that adjusts the exposure of the camera unit, the control method comprising: an output step in which the exposure adjustment unit outputs a speed change signal that changes the movement speed of the moving body based on the exposure adjustment status, or an adjustment status signal related to the exposure adjustment status. [Supplementary Note 11] A control program for a crop observation device (100, 100', 100a, 100b) that is mounted on a mobile body (200) that moves in a field and includes a camera unit (110) that captures images of crops that are an observation target in accordance with the movement of the mobile body, and an exposure adjustment unit (123, 123a, 123b) that adjusts the exposure of the camera unit, the control program for the crop observation device causing a computer to execute an output step in which the exposure adjustment unit outputs a speed change signal that changes the movement speed of the mobile body based on the exposure adjustment status or an adjustment status signal related to the exposure adjustment status. [Supplementary Note 12] The crop observation device according to Supplementary Note 1, in which the exposure adjustment unit (123b) acquires exposure adjustment information related to exposure adjusted for an auxiliary camera unit (150) that captures images of the crops located in the movement direction of the mobile body before the camera unit, and adjusts the exposure of the camera unit based on the exposure adjustment information.

Claims

1. A crop observation device that is mounted on a moving body that moves in a field and comprises a camera unit that captures images of a crop to be observed in accordance with the movement of the moving body, and an exposure adjustment unit that adjusts the exposure of the camera unit, wherein the exposure adjustment unit outputs a speed change signal that changes the moving speed of the moving body based on the exposure adjustment status, or an adjustment status signal related to the exposure adjustment status.

2. A crop observation device as described in claim 1, wherein the exposure adjustment unit recognizes a sky area captured from the captured image output by the camera unit, and adjusts the exposure by allowing for blown-out highlights in the sky area.

3. A crop observation device as described in claim 1, wherein the exposure adjustment unit recognizes a sky area in which the sky is captured from the captured image output by the camera unit, and adjusts the exposure without allowing blackout of non-sky areas other than the sky area.

4. A crop observation device as described in claim 2, wherein the exposure adjustment unit recognizes the sky area by detecting the difference between a plurality of captured images successively output by the camera unit while changing the imaging conditions.

5. The crop observation device according to claim 2, wherein the exposure adjustment unit corrects the recognition of the sky area based on a designated area designated by a user in the captured image.

6. The crop observation device according to claim 2, wherein the exposure adjustment unit corrects the recognition of the sky area based on an index designated by a user with respect to the crop appearing in the captured image.

7. A crop observation device as described in claim 1, comprising a plurality of camera units, an image processing unit that sets a common fixed white balance for the plurality of camera units, and the exposure adjustment unit that individually adjusts the exposure of each of the camera units.

8. A crop observation device according to claim 1, further comprising an illuminance sensor that acquires illuminance information in a moving direction of the moving object, and the exposure adjustment unit adjusts the exposure using the illuminance information.

9. A crop observation device as described in claim 1, further comprising a position information acquisition unit that acquires position information of the moving body, and the exposure adjustment unit determines initial conditions for the exposure adjustment based on current position information acquired from the position information acquisition unit and exposure adjustment information at the same position in the past.

10. A control method for a crop observation device that is mounted on a moving body that moves in a field and has a camera unit that captures images of the crop being observed in accordance with the movement of the moving body, and an exposure adjustment unit that adjusts the exposure of the camera unit, the control method having an output step in which the exposure adjustment unit outputs a speed change signal that changes the movement speed of the moving body based on the exposure adjustment status, or an adjustment status signal related to the exposure adjustment status.

11. A control program for a crop observation device that is mounted on a moving body that moves in a field and has a camera unit that captures images of the crop being observed in accordance with the movement of the moving body, and an exposure adjustment unit that adjusts the exposure of the camera unit, the control program for a crop observation device that causes a computer to execute an output step in which the exposure adjustment unit outputs a speed change signal that changes the movement speed of the moving body based on the exposure adjustment status, or an adjustment status signal related to the exposure adjustment status.

12. A crop observation device as described in claim 1, wherein the exposure adjustment unit acquires exposure adjustment information regarding exposure adjusted for an auxiliary camera unit that captures images of the crops located in the direction of movement of the moving body before the camera unit, and adjusts the exposure of the camera unit based on the exposure adjustment information.