Image diagnostic apparatus, image diagnostic system, image diagnostic method, program, and storage medium

Abstract

To provide an image diagnostic apparatus capable of detecting a characteristic area of a crop plant highly accurately through the entire growth period.SOLUTION: An image diagnostic apparatus has: a holding part which holds a plurality of learned models; a selection part which selects a first learned model from the plurality of learned models; and a determination part which determines an object pixel including at least a part of a crop plant in an object image obtained by imaging the crop plant, by using the first learned model, wherein the selection part selects the first learned model corresponding to a growth index of the crop plant acquired on the basis of the object pixel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image diagnostic apparatus for diagnosing the growth state of crops using images of the crops.

Background Art

[0002] Conventionally, a method has been proposed for extracting characteristic regions of crops from images of the crops and diagnosing the growth state. Patent Document 1 discloses a configuration for recognizing target regions such as rice grains in a photographed image and further obtaining indices related to colors such as the color of ripe rice grains by machine learning or the like. The configuration of Patent Document 1 is suitable for accurately obtaining the color of rice grains at a time suitable for harvesting.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, there has been a need to accurately diagnose the growth state by detecting characteristic regions such as ears and grains throughout the entire growth period from early stages such as heading to the time suitable for harvesting, and applying it to work plans. Ears and grains in the early stages such as from heading to before ripening are lightly colored, for example, it is difficult to distinguish them from leaves, etc., and their shapes are also different from those at the harvesting time. Therefore, with a configuration optimized for the time suitable for harvesting as in Patent Document 1, it is difficult to accurately diagnose the growth state throughout the entire growth period.

[0005] An object of the present invention is to provide an image diagnostic apparatus capable of highly accurately detecting characteristic regions of crops throughout the entire growth period.

Means for Solving the Problems

[0006] An image diagnostic apparatus according to one aspect of the present invention includes a holding unit that holds a plurality of learned models, a selection unit that selects a first learned model from the plurality of learned models, and a determination unit that determines target pixels including at least a part of the crop in a target image obtained by imaging the crop using the first learned model. The selection unit is characterized by selecting the first learned model according to a growth index of the crop acquired based on the target pixels.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide an image diagnostic apparatus capable of detecting characteristic regions of crops with high accuracy throughout the growth period.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Modes for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant descriptions are omitted.

[0010] FIG. 1(a) shows an aspect of the configuration of an image diagnostic apparatus 100 according to an embodiment of the present invention. The image diagnostic apparatus 100 includes an arithmetic unit 101, a RAM 102, a storage 103, a network interface 104, an input unit 105, a display unit 106, and a ROM 107. The arithmetic unit 101 includes a CPU 101a and a GPU 101b. The GPU 101b is not necessarily required, but it is preferably used as appropriate according to the desired processing speed because the processing speed can be increased when performing the flow of FIG. 3 described later.

[0011] The input unit 105 acquires a captured image obtained by imaging means such as a camera (not shown) for crops such as growing rice and wheat, fungi, and crystals. In the present embodiment, a color image having three RGB information using wavelength information in the visible light region is used as the captured image. The ROM 107 stores the captured image. The captured image is stored in the storage 103 or an external storage device 160 or the like. The storage 103 can be arbitrarily selected from a hard disk, a solid state drive, or the like. The external storage device 160 can be arbitrarily selected from various storage media, and in the present embodiment, the captured image is read through the network interface 104. The display unit 106 is a display or various display devices, and displays information such as various calculation results. Note that the display unit 106 does not necessarily have to be in the same place or in the vicinity of other devices of the image diagnostic apparatus 100, and may be a remotely located display, a screen of a PC, a smartphone, or the like.

[0012] Fig. 1(b) shows a configuration for executing the flow of Fig. 3 described later. The arithmetic unit 101 includes a selection unit 201, a determination unit 202, a determination unit 203, and a diagnosis unit 204. Regarding each unit, it may be appropriately selected whether to select the CPU 101a or the GPU 101b as the arithmetic unit 101, and there is no constraint. The storage 103 includes a holding unit (storage unit) 205 that holds a learned model (machine learning model) and a phase determination result described later.

[0013] Fig. 2 is a diagram showing a configuration example of an image diagnosis system 1 including an image diagnosis apparatus 100. The image diagnosis system 1 includes an imaging device (imaging unit) 10 and an image diagnosis apparatus 100. The image diagnosis system 1 may have the external storage device 160 described above as a storage destination for various images and the like. Note that the imaging device 10 and the image diagnosis apparatus 100 do not necessarily need to be arranged in the vicinity. They may be connected by communication via the Internet 150, or various information acquired by the imaging device 10 may be stored in the external storage device 160 in advance, and image diagnosis by the image diagnosis apparatus 100 may be performed at an arbitrary timing.

[0014] The imaging device 10 is a device for imaging and diagnosing the growth status of crops and the like, and includes a camera unit 11 as an imaging means, a photometry unit 12 as a photometry means, and a control unit 14 that also serves as an exposure control means and a recording means.

[0015] The camera unit 11 includes a lens (not shown) for capturing a light beam from a subject and forming an image, and an image sensor (not shown) for converting and amplifying an optical signal into an electrical signal.

[0016] The photometry unit 12 is a means for making the ambient light color correspond to standard white, and is a color sensor for capturing the ambient light luminance and selectively receiving each ambient light of red (R band), green (G band), and blue (B band).

[0017] The camera unit 11 and the photometry unit 12 are installed in the field and are used to monitor the crops (such as rice, wheat, and vegetables, etc.) 2 that grow day by day by taking fixed-point pictures, and to diagnose information such as plant height information and color information as indicators of the growth state. In this embodiment, the imaging device 10 is fixed to a pole-shaped support member 13 that rises from the ground at a predetermined position within the field. The support member 13 is made of a corrosion-resistant material such as aluminum or stainless steel, and has an L-shaped bracket structure. The camera unit 11 is attached to the tip of the bracket structure and can image a specific target plant that is the imaging target.

[0018] Note that a power supply 15 may be attached to the support member 13. The power supply 15 may supply power via natural energy such as a solar panel or a wind power generation device, or may be a replaceable battery. Also, it may be powered by sharing power through an external electric wire or the like.

[0019] Also, when it is desired to perform imaging over a wider range or when a strict fixed viewpoint is not required, the imaging device 10 does not necessarily have to be fixed by various support means. For example, it may be carried by a person's hand or moved by a flying means such as a drone.

[0020] The control unit 14 performs various image processing and calculations on the collected digital image data, such as cutting out a predetermined part of the image, adjusting the white balance, and extracting the luminance value. Also, the control unit 14 can calculate an exposure value consisting of gain (Gain), aperture (Fno), and shutter speed (S / S), and can give instructions to the camera unit 11. In FIG. 2, the control unit 14 is attached to the support member 13 and is provided at a position close to the camera unit 11, but it may be arranged at a remote location via communication equipment (not shown). In this case, for example, a plurality of camera units 11 and photometry units 12 may be arranged at a remote location, and the control unit 14 may be arranged in a data center or the like to perform data input / output via cloud technology or the like.

[0021] In the following respective embodiments, a method for accurately detecting a feature region such as an ear in a target image obtained by photographing a crop using the image diagnostic apparatus 100 will be described. The target image may be, for example, an image obtained by photographing the crop from above, or may be one of the images obtained by continuously photographing the crop from a fixed viewpoint. In each of the embodiments, ears of paddy rice are detected and diagnosed, but the present invention is not limited to ears and can be applied to various crops such as wheat and fruit trees.

Embodiment

[0022] FIG. 3 is an image diagnostic flowchart F1 showing an image diagnostic method executed by the arithmetic unit 101 of the present embodiment. Among the arithmetic unit 101, mainly the CPU 101a is used in normal sequential processing, and mainly the GPU 101b is used in parallel processing such as image processing. However, since it is not particularly specified strictly for each process and can be appropriately selected, in the following description, the expression arithmetic unit 101 is unified.

[0023] In step S101, the arithmetic unit 101 acquires a target image (captured image) obtained by photographing paddy rice from the holding unit 205.

[0024] In step S102, the arithmetic unit 101 acquires the current determination stage from the holding unit 205. Here, the current determination stage is an index for representing step by step the degree of appearance and confirmation of ears in the feature region based on the ear growth index described later, and in this embodiment, it is represented by any one of the values 0, 1, 2. Note that when the device is first installed or when the installation location is changed, there may be no determination stage based on the growth index and the holding unit 205 may not hold the determination stage. In such a case, it is preferable to perform exception processing such as referring to the data of the determination stage of previous years from the time and environmental temperature, etc., and temporarily setting the determination stage.

[0025] In step S103, the arithmetic unit 101 determines whether the current determination stage "stage" is 0 based on the function of the selection unit 201. It is preferable to set the determination stage "stage" to 0 during a period when there is no possibility of ears emerging, such as immediately after transplanting or during the middle culm period. When the arithmetic unit 101 determines that the current determination stage "stage" is 0, it executes the process of step S114. By not performing the process of the determination unit 202 described later, the time taken for unnecessary processes can be reduced, which can also lead to data reduction. Further, processes not shown, such as setting the number of calculated pixels to 0 in the calculation of the target pixels described later, or outputting a completely black image with the target pixel area set to 0, may be added. When the arithmetic unit 101 determines that the current determination stage "stage" is not 0, it executes the process of step S104.

[0026] In step S104, the arithmetic unit 101 determines whether the current determination stage "stage" is 1 based on the function of the selection unit 201. It is preferable to set the determination stage "stage" to 1 during the period when young panicles are formed inside the rice plants, generally after the rice plants reach the maximum tillering stage. However, if there are needs such as collecting data from the initial stage, the determination stage "stage" may be set to the state of 1 from the installation stage of the image diagnosis system 1. When the arithmetic unit 101 determines that the current determination stage "stage" is 1, it executes the process of step S105a, and otherwise, that is, when it determines that the current determination stage "stage" is 2, it executes the process of step S105b.

[0027] In step S105a, the arithmetic unit 101 selects model A, which is a learned model generated by machine learning, for determining the target area (feature area), which is the area where ears exist in the image for ear detection, based on the target image acquired in step S101.

[0028] In step S105b, the arithmetic unit 101 selects model B, which is a learned model generated by machine learning, for determining a target area that is an area where ears exist in the image for ear detection based on the target image acquired in step S101.

[0029] In step S106, the arithmetic unit 101 extracts the area of the ears, which is a feature area in the target image, using the selected learned model based on the function of the determination unit 202, and determines it as target pixels that include at least a part of the crop and appear according to the growth state of the crop.

[0030] In this embodiment, step S105a and step S105b are the first step, and step S106 is the second step.

[0031] Figure 4 is an explanatory diagram of the target pixel determination process. The captured image 401 outputs an ear area image 404 using the Gray learning model 402a, which is model A, or the Color learning model 402b, which is model B, so as to solve a regression problem.

[0032] The Gray learning model 402a is a learned model obtained by learning using a Gray image group 403a, which is a group of captured images in grayscale. The Gray learning model 402a is mainly suitable for performing ear detection at a time immediately before or after the ears appear. In the heading where the ears appear outside the plant, the ears at a time immediately after that have a color close to that of the leaves and stems, and it is difficult to distinguish them using color information. Furthermore, since there are rice varieties in which the leaf tips turn yellow, a learned model using a color image of the ears is likely to misidentify such leaves as ears. Therefore, it is preferable to use the Gray learning model 402a for the determination process when the determination stage stage is 1.

[0033] The Gray image group 403a may be an image originally acquired as a grayscale image, or an image obtained by performing grayscale processing on a color-captured image. Also, instead of a complete grayscale image, a color image with a lower chroma than the Color learning model 402b may be appropriately used. Also, the combination of images in the Gray image group 403a used for the Gray learning model 402a may be arbitrarily selected, but by, for example, setting the proportion of grayscale images to 60% or more, it is possible to improve the detection performance for the period around the heading stage. Preferably, 80% or more of the combination of images in the Gray image group 403a used for the Gray learning model 402a is composed of grayscale images.

[0034] When the determination stage stage is 2, in the stage where the ears are out and growing, since the coloring of the ears gradually becomes darker day by day and the shape clearly shows the appearance of ears, it is preferably for detection performance to learn the model using the information obtained by color-capturing the ears. Therefore, the Color learning model 402b is used for the determination process. In the Color image group 403b, it is preferable that 60% or more of all the images are formed by color-captured images. More preferably, it is 80% or more, and the detection performance can be improved. Also, more images of ears at a more advanced growth stage than the Gray image group 403a may be used.

[0035] FIG. 5 is a diagram showing an example of learning using a CNN (Convolutional Neural Network). In the input layer 501, an input image 5011 which is an image of paddy rice as input data, and a teacher image 5012 in which the part where ears exist in the input image 5011 is labeled white (1) and the part where ears do not exist is labeled (0) are prepared. The output image 503 in the output layer output by the learning model 502 generated by the CNN using the input layer 501 is compared with the teacher image 5012, and a loss function 504 is calculated. Then, using various methods such as the error backpropagation method, the weight function of the connection between the nodes of the CNN is updated so as to reduce the obtained error. By repeating these generationally, a learning model 502 with appropriate updates is obtained.

[0036] In this embodiment, there are two learned models, but there may be three or more learned models.

[0037] In step S107, the arithmetic device 101 performs various processes for obtaining information on an appropriate ear region from the ear region image output by the determination process. For example, in the output ear region image, it is conceivable that leaves or soil parts reflected in addition to the ear are misjudged as ears. Therefore, it is preferable to perform noise processing according to the performance of the Gray learning model 402a or the Color learning model 402b. For example, first, a detection image actually detected as an ear is calculated using a plurality of images not included in the teacher image group, and the color histogram region of the incorrect part is defined as an abnormal region using the color histograms of the correct part and the incorrect part. Next, processes such as dispersing pixels having a color histogram region conforming to the incorrect part from the detection image can be considered. Note that other noise processing may be performed. Furthermore, noise reduction or the like based on an image processing method such as morphological processing may be executed, and the remaining pixels may be treated as an ear region image.

[0038] In step S108, the arithmetic device 101 counts the target pixels for the ear region image and calculates (acquires) the ear area ratio S as a growth index. The ear area ratio S is the ratio of the number of target pixels in an arbitrary region to the total number of pixels (total number of all pixels) in the same region in the ear region image. The arbitrary region is at least a part of the ear region image. That is, it may be the entire ear region image or a specific region to be diagnosed in the ear region image.

[0039] FIG. 6 is a diagram showing an example of ear area ratio calculation processing. The ear display image 601 is obtained by superimposing an input image on the ear region image. As the ear grows, as in the ear display image 601, the ears are unevenly distributed within the image, and due to the influence of wind, lodging, etc., the degree of uneven distribution changes, and the ear area ratio S may change significantly depending on the shooting timing. Therefore, instead of the ear area ratio S of the entire image, the ear display image 601 is divided into nine parts (nine sub-regions are set), and as shown in the calculation result example 602, the sub-ear area ratio Sg of each sub-region is calculated, and the average ear area ratio Save 603 calculated from their average may be used. By appropriately using such a method, the influence of the uneven distribution change of the ears within the screen can be reduced. Preferably, the screen is divided into six or more parts, and more preferably, nine or more parts. Hereinafter, assuming that an appropriate one is selected for the ear area ratio S and the average ear area ratio Save as appropriate, the expression is unified as the ear area ratio S.

[0040] FIG. 7 is a diagram showing an example of a detection result calculated based on the functions of the selection unit 201 and the determination unit 202, and shows an example of a detection result for the same image using the Gray learning model 402a and the Color learning model 402b. In FIG. 7, the horizontal axis represents the number of days elapsed since the start of shooting with the image diagnosis system 1 installed, and the vertical axis represents the average ear area ratio Save. The result of detection using the Gray learning model 402a is shown by a solid line, and the result of detection using the Color learning model 402b is shown by a broken line. In FIG. 7, the day when the ear was visually confirmed in the image is the 65th day from the start of shooting. Before the ear is confirmed, when the Gray learning model 402a is used, the overall ear area ratio S is low, there are few false detections, and appropriate results can be obtained. Also, after the ear is confirmed, when the Gray learning model 402a is used, the ear area ratio Save fluctuates greatly. On the other hand, when the Color learning model 402b is used, the ear area ratio S shows a growth state with little fluctuation, and a result closer to the actual growth state can be obtained by using the Color learning model 402b. Therefore, by the selection unit 201 selecting an appropriate learned model according to the determination stage stage and the determination unit 202 performing ear region detection, it is possible to achieve ear region detection with good performance throughout the entire growth period.

[0041] In step S109, the arithmetic unit 101 determines whether the ear area ratio S is smaller than a predetermined threshold TH based on the function of the determination unit 203. The threshold TH is preferably arbitrarily set according to the performance of the learned model, the strength of noise removal, and the situation of each variety. However, by setting it to approximately 5% or less as the ear area ratio S, good ear confirmation performance can be obtained. When the arithmetic unit 101 determines that the ear area ratio S is smaller than the threshold TH, it executes the process of step S110a, and when it determines otherwise, it executes the process of step S110b. Note that when the ear area ratio S is equal to the threshold TH, it is possible to arbitrarily set which process to execute.

[0042] In step S110a, the arithmetic unit 101 sets the determination stage stage to 1 based on the function of the determination unit 203.

[0043] In step S110b, the arithmetic unit 101 sets the determination stage stage to 2 based on the function of the determination unit 203.

[0044] In step S111, the arithmetic unit 101 updates the determination stage stored in the holding unit 205.

[0045] In this embodiment, in step S104, the learning model used for the calculation for obtaining the ear region image is selected according to the state where the determination stage stage is 1 or 2. However, the present invention is not limited to this. Depending on the variety and the situation of the field, it is also possible to calculate and hold both of the two learning models in advance, and select the ear region image for calculating the growth confirmation index described later according to the determination stage.

[0046] In step S112, the arithmetic unit 101 calculates (acquires) an ear confirmation index as a growth confirmation index based on the function of the determination unit 203. As the ear confirmation index, for example, it is preferable to use the ear area ratio S calculated in step S108 and the ear coverage rate ER (= S / Sp) expressed as a percentage of the day of the maximum value Sp of the ear area ratio S recorded in the same field or of the same variety before the previous year. In the first year when the measurement starts, it is assumed that the maximum value Sp of the ear area ratio S is not available as pre-data. In that case, for example, the maximum value Sp of the ear area ratio S may be set to a virtual assumed value or set to 1. Note that the values of the first flag or the second flag for notifying the user described later are preferably set appropriately according to the set maximum value Sp of the ear area ratio S. Further, in order to enhance the smoothness of the data and facilitate the analysis, the ear coverage rate ER may be calculated using a time-series moving average or various smoothing methods. Note that the ear confirmation index is obtained using the ear area ratio S which is a growth index in this embodiment, but may be obtained using the slope of the growth index. The slope of the growth index may be a differential value or a difference value in the time series.

[0047] FIG. 8 is a diagram showing an example of a notification process displayed on the display unit 106. The graph 801 shows the moving average of the ear coverage rate ER (= S / Sp) of the image diagnostic apparatus 100 installed in a specific farm field. Based on the function of the determination unit 203, the arithmetic unit 101 performs a two-stage determination as shown in the determination example 802. When the ear coverage rate ER exceeds 10%, the ear possibility recognition flag indicating the possibility of the appearance of the feature region (target pixel) is set to ON, a first flag for notifying the user is set, and the user is given a first notification. When the ear coverage rate ER exceeds 40%, the ear confirmation determination flag indicating the confirmation of the appearance of the feature region (target pixel) is set to ON, a second flag for notifying the user is set, and the user is given a second notification.

[0048] The display 803 shows an example of the display on the display unit 106. In the above case, it is preferable that the second notification is made at a time later than the first notification. Therefore, even if the ear coverage rate ER exceeds 40% before the first notification due to false detection, the convenience of the user can be improved by preventing the second notification from being made before the first notification through error processing or abnormal value exclusion. By performing such two-stage notification determination processing, it is possible to visualize the growth condition of the ears and improve the convenience of the user. Such notification flags and displays are not limited to two, and may be only one or three or more.

[0049] In step S113, based on the function of the diagnostic unit 204, the arithmetic unit 101 performs a growth diagnosis. Here, the growth diagnosis calculates an index (feature amount) related to an index other than the ear area ratio S among the indexes related to the growth of the ears, and examples thereof include an index related to color. Specifically, an index related to the color of the ear (ear color) may be calculated from the median values of the color histograms of the ratios R / G and B / G of the color luminances of R, G, and B of the target pixels in the ear region image.

[0050] In step S114, the arithmetic unit 101 causes the display unit 106 to display the diagnosis result. FIG. 9 is a diagram showing an example of the diagnosis result displayed on the display unit 106. The diagnosis result 900 is the diagnosis result for a specific day in a specific field. The ear coverage rate graph 901 shows the moving average of the ear coverage rate ER (= S / Sp) of the image diagnosis device 100 installed in a specific field. Note that various measures may be taken to enhance user convenience, such as appropriately weighting according to the growth status, making it non-displayed before ear confirmation, and displaying the ear confirmation date. The ear color graph 902 numerically displays the moving average of the ear color index of the image diagnosis device 100 installed in a specific field. At this time, in order to enhance the reliability regarding the detected color of the ear, for example, it is conceivable to display the graph from the day when the ear confirmation determination flag in step S112 is set to ON, which is referred to as the ear confirmation date. Further, as shown in the information display 903, dates, data, etc. regarding the ear may be appropriately displayed in text. For example, by displaying the ear coverage rate, ear color, ear confirmation date, and the number of days elapsed since the ear confirmation date in parallel, it is possible to assist the user in formulating a harvest plan. Also, by displaying the actual captured images in parallel like the image 904, the image of the growth state can be made clearer.

[0051] Note that the diagnosis result 900 may display not only the various types of information described above but also other growth indicators such as leaf color and plant height, and environmental information such as environmental temperature. Further, the image diagnosis device 100 may be provided with a notification function such as transmitting the diagnosis result 900 to, for example, a smartphone and notifying of ear confirmation.

[0052] Also, in this embodiment, although the determination stage stage can be set (determinable) using the ear area ratio S, which is a growth indicator, the determination stage stage may be set using the slope of the growth indicator. The slope of the growth indicator may be a differential value, a difference value in the time series, or the like.

Example

[0053] In this embodiment, the processing of the determination unit 203 is different from that in Embodiment 1. In this embodiment, only the configurations different from those in Embodiment 1 will be described, and the description of the same configurations will be omitted.

[0054] FIG. 10 is an image diagnosis flowchart F2 showing an image diagnosis method executed by the arithmetic unit 101 of this embodiment. Since the processing from step S201 to step S208 is the same as the processing from step S101 to step S108 in FIG. 2, the description thereof will be omitted.

[0055] In step S209, the arithmetic unit 101 reads out, from the holding unit 205, time-series data regarding the ear area ratio S and the ear possibility flag (growth flag) Flag recorded before the previous time, based on the function of the determination unit 203. The ear possibility flag Flag is a flag value generated in each round of this flow for each image executed in the same image diagnosis apparatus 100 in the same year, and each has a value of either 0 or 1.

[0056] In step S210, the arithmetic unit 101 calculates (acquires) a determination ear area ratio ΔS from the time-series data of the ear area ratio S, based on the function of the determination unit 203. The determination ear area ratio ΔS may be, for example, the difference ΔS (=Sm(3Days) - Sm(10Days)) between the average value Sm(3Days) of the ear area ratios for the past three days including the latest ear area ratio S and the average value Sm(10Days) of the ear area ratios for the past ten days. By performing such processing, even if the ear area ratio S temporarily increases due to noise depending on the performance of the learned model, or disturbances such as sunlight or insects, even though no ears have emerged, the determination ear area ratio ΔS can be smoothed and the possibility of misjudgment can be reduced. The calculation formula for the difference ΔS, the calculation range of the moving average parameter, etc. may be changed as appropriate.

[0057] In step S211, based on the function of the determination unit 203, the arithmetic unit 101 determines whether the determination ear area ratio ΔS is greater than a predetermined threshold TH'. For example, when it is desired to determine the presence of ears, it is preferable to set the predetermined threshold TH' to about 0.5% to 5.0%. If the arithmetic unit 101 determines that the determination ear area ratio ΔS is greater than the predetermined threshold TH', it executes the process of step S212a. If it determines that the determination ear area ratio ΔS is less than the predetermined threshold TH', it executes the process of step S212b. When the determination ear area ratio ΔS is equal to the predetermined threshold TH', it can be arbitrarily set which step's process to execute.

[0058] In step S212a, based on the function of the determination unit 203, the arithmetic unit 101 outputs a value of 1 as the ear possibility flag Flag, associated with the shooting date and time of the target image being executed in this flow.

[0059] In step S212b, based on the function of the determination unit 203, the arithmetic unit 101 outputs a value of 0 as the ear possibility flag Flag, associated with the shooting date and time of the target image being executed in this flow.

[0060] In step S213, based on the function of the determination unit 203, the arithmetic unit 101 determines whether the total value ΣFlag of the ear possibility flags Flag recorded in time series exceeds the ear confirmation determination flag THF. As the ear confirmation determination flag THF, a value between 3 and 10 can be taken to achieve good performance for ear confirmation. Also, operations such as limiting the period for counting the total value ΣFlag to the past three days may be performed. Further, as the total value ΣFlag, a value obtained by counting consecutive past ear possibility flags Flag may be compared with the ear confirmation determination flag THF. When the arithmetic unit 101 determines that the total value ΣFlag of the ear possibility flags Flag exceeds the ear confirmation determination flag THF, it executes the process of step S214a. Also, when the arithmetic unit 101 determines that the total value ΣFlag of the ear possibility flags Flag does not exceed the ear confirmation determination flag THF, it executes the process of step S214b. Note that when the total value ΣFlag of the ear possibility flags Flag is equal to the ear confirmation determination flag THF, it can be arbitrarily set which step's process to execute.

[0061] In step S214a, based on the function of the determination unit 203, the arithmetic unit 101 sets the determination stage stage to 1.

[0062] In step S214b, based on the function of the determination unit 203, the arithmetic unit 101 sets the determination stage stage to 2.

[0063] In step S215, the arithmetic unit 101 updates the determination stage stage stored in the holding unit 205.

[0064] In step S216, based on the function of the determination unit 203, the arithmetic unit 101 calculates (acquires) an ear confirmation index. As the ear confirmation index, it is preferable to use the ear coverage rate ER (=S / Sp) described in Example 1.

[0065] Since the processes of step S217 and step S218 are the same as the processes of step S113 and step S114 in FIG. 2, respectively, the description thereof is omitted.

[0066] FIG. 11 is a diagram showing an example of a diagnostic result displayed on the display unit 106. The graph 1111 shows the moving average of the ear coverage rate ER (= S / Sp) of the image diagnostic apparatus 100 installed in a specific field. The arithmetic unit 101 performs step-by-step determination as shown in determination example 1113 by processing data in time series based on the function of the determination unit 203. For example, when the ear coverage rate ER for which shooting is performed three times a day exceeds 3%, as shown in Table 1112, the past three shooting records are read from the holding unit 205. Further, when the ear coverage rate ER continuously exceeds 3%, a process of turning on the ear possibility recognition flag is performed as shown in determination example 1113 on the assumption that the possibility of ears is recognized. Similarly, when the ear coverage rate ER exceeds 10%, for example, the past ten shooting records are read from the holding unit 205. When the number of times the ear coverage rate ER exceeds 10% is nine or more out of the past ten times, a process of turning on the ear confirmation determination flag is performed on the assumption that the confirmation of ears is definitely determined, and the ear confirmation determination flag for notifying the user is set. The display 1114 is an example of the display on the display unit 106. [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0067] The disclosure of the present embodiment includes the following configurations and methods. [Configuration 1] A holding unit that holds a plurality of learned models; A selection unit that selects a first learned model from the plurality of learned models; A determination unit that determines target pixels including at least a part of the crop in a target image obtained by imaging the crop using the first learned model, wherein the selection unit selects the first learned model according to a growth index of the crop obtained based on the target pixels. An image diagnostic apparatus characterized by that. (Configuration 2) The image diagnostic apparatus according to Configuration 1, wherein the plurality of learned models includes a first model learned using at least one of a grayscale image and an image having a lower saturation than the target image. (Configuration 3) The image diagnostic apparatus according to Configuration 1 or 2, wherein the plurality of learned models includes a second model learned using a color image. (Configuration 4) The image diagnostic apparatus according to any one of Configurations 1 to 3, wherein the growth index is a first index that is a ratio of the number of pixels in the feature region in the region to the total number of pixels in at least a part of the region of the target image. (Configuration 5) When at least a part of the region of the target image is divided into a plurality of sub-regions including a first sub-region and a second sub-region, the growth index is a first ratio of the number of pixels in the feature region in the first sub-region to the total number of pixels in the first sub-region and a second ratio of the number of pixels in the feature region in the second sub-region to the total number of pixels in the second sub-region, and the image diagnostic apparatus according to any one of Configurations 1 to 3, wherein the growth index is a second index obtained using the second ratio. (Configuration 6) The image diagnostic apparatus according to Configuration 5, wherein the second index is obtained using an average of the first ratio and the second ratio. (Configuration 7) The image diagnostic apparatus according to any one of Configurations 1 to 6, further comprising a determination unit that determines the growth state of the crop step by step. (Configuration 8) The image diagnostic apparatus according to Configuration 7, wherein the determination unit determines the growth state step by step using the growth index. (Configuration 9) The plurality of learned models includes a first model learned using at least one of a grayscale image and an image having a lower saturation than the target image, the determination unit can determine whether the growth state is in a first stage or a second stage, When the determination unit determines that the growth state is in the first stage, the selection unit selects the first model. The image diagnostic apparatus according to Configuration 7 or 8, characterized in that. (Configuration 10) The holding unit holds at least one of the growth index and the growth flag generated based on the growth index as time-series data associated with the shooting date and time of the target image. The determination unit uses the time-series data to determine the growth state step by step. The image diagnostic apparatus according to Configuration 7, characterized in that. (Configuration 11) The selection unit selects one of the learned models corresponding to the stage of the growth state determined by the determination unit. The image diagnostic apparatus according to any one of Configurations 7 to 10, characterized in that. (Configuration 12) The determination unit notifies the user of the growth state according to the growth confirmation index obtained by using the growth index or the ratio of the growth index to the past growth index obtained before the growth index is obtained. The image diagnostic apparatus according to any one of Configurations 7 to 11, characterized in that. (Configuration 13) According to the growth index or the growth confirmation index, one of the first notification indicating the possibility of the appearance of the target pixel and the second notification indicating the confirmation of the appearance of the target pixel is given to the user. The second notification is performed after the first notification is notified. The image diagnostic apparatus according to Configuration 12, characterized in that. (Configuration 14) The past growth index is the maximum value of the growth index recorded before the year before last. The image diagnostic apparatus according to Configuration 12 or 13, characterized in that. (Configuration 15) The image diagnostic apparatus according to any one of Configurations 1 to 14, further comprising a diagnostic unit that acquires a feature amount related to the growth state of the crop using the target pixel. (Configuration 16) The feature amount includes a feature amount related to the color of the crop. The image diagnostic apparatus according to Configuration 15, characterized in that. (Configuration 17) The learned model is a learned model that has learned the target pixel using an image obtained by dividing a teacher image, which uses a previously prepared image of the crop, into a plurality of images, and is characterized by being any one of Configurations 1 to 16 of the image diagnostic apparatus according to the description. (Configuration 18) The target image is an image obtained by photographing the crop from above, and is characterized by being any one of Configurations 1 to 17 of the image diagnostic apparatus according to the description. (Configuration 19) The target image is one of the images obtained by continuously photographing the crop from a fixed viewpoint, and is characterized by being any one of Configurations 1 to 18 of the image diagnostic apparatus according to the description. (Configuration 20) The crop includes at least one of rice and wheat, and is characterized by being any one of Configurations 1 to 19 of the image diagnostic apparatus according to the description. (Configuration 21) An image diagnostic apparatus according to any one of Configurations 1 to 20, and An image diagnostic system characterized by having an imaging unit that images the crop. (Configuration 22) The imaging unit is disposed remotely from the image diagnostic apparatus, and is characterized by being the image diagnostic system according to Configuration 21. (Method 1) A first step of selecting a first learned model from a plurality of learned models, and A second step of determining a target pixel including at least a part of the crop in a target image obtained by imaging the crop using the first learned model, and In the first step, the first learned model is selected according to a growth index of the crop obtained based on the target pixel, and is characterized by being the image diagnostic method. (Configuration 23) A program characterized by causing a computer to execute the image diagnostic method according to Method 1. (Configuration 24) A storage medium characterized by storing the program according to Configuration 23.

[0068] As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

Explanation of Signs

[0069] 100 Imaging diagnostic apparatus 201 Selection unit 202 Determination unit 205 Holding unit

Claims

1. A holding unit that holds a plurality of learned models; A selection unit that selects a first learned model from the plurality of learned models; A determination unit that determines target pixels including at least a part of the crop in a target image obtained by imaging the crop using the first learned model, and The selection unit selects the first learned model according to a growth index of the crop obtained based on the target pixels. An image diagnostic apparatus characterized by that.

2. The image diagnostic apparatus according to claim 1, wherein the plurality of learned models include a first model learned using at least one of a grayscale image and an image having a lower saturation than the target image.

3. The image diagnostic apparatus according to claim 1 or 2, wherein the plurality of learned models include a second model learned using a color image.

4. The image diagnostic apparatus according to claim 1 or 2, wherein the growth index is a first index that is a ratio of the number of the target pixels in the region to the total number of pixels in at least a part of the region of the target image.

5. When at least a part of the region of the target image is divided into a plurality of sub-regions including a first sub-region and a second sub-region, the growth index is the first ratio of the number of the target pixels in the first sub-region to the total number of pixels in the first sub-region, and the second ratio of the number of the target pixels in the second sub-region to the total number of pixels in the second sub-region. The image diagnostic apparatus according to claim 1 or 2, characterized in that it is a second index obtained using.

6. The image diagnostic apparatus according to claim 5, wherein the second index is obtained using an average of the first ratio and the second ratio.

7. The image diagnostic apparatus according to claim 1 or 2, further comprising a determination unit that determines the growth state of the crop step by step.

8. The image diagnostic apparatus according to claim 7, wherein the determination unit determines the growth state step by step using the growth index.

9. The plurality of learned models include a first model learned using at least one of a grayscale image and an image having a lower saturation than the target image, and The determination unit can determine whether the growth state is in a first stage or a second stage. The image diagnostic apparatus according to claim 7, wherein the selection unit selects the first model when the determination unit determines that the growth state is in the first stage.

10. The holding unit holds at least one of the growth index and the growth flag generated based on the growth index as time-series data associated with the shooting date and time of the target image. The image diagnostic apparatus according to claim 7, wherein the determination unit uses the time-series data to determine the growth state step by step.

11. The image diagnostic apparatus according to claim 7, wherein the selection unit selects one of the learned models corresponding to the stage of the growth state of the crop determined by the determination unit.

12. The image diagnostic apparatus according to claim 7, wherein the determination unit notifies the user of the growth state according to the growth confirmation index obtained by using the growth index or the ratio of the growth index to the past growth index obtained before the growth index is acquired.

13. One of a first notification indicating the possibility of appearance of the target pixel and a second notification indicating the confirmation of appearance of the target pixel is given to the user according to the growth index or the growth confirmation index. The image diagnostic apparatus according to claim 12, wherein the second notification is given after the first notification is given.

14. The image diagnostic apparatus according to claim 12, wherein the past growth index is the maximum value of the growth index recorded before the year before last.

15. The image diagnostic apparatus according to claim 1 or 2, further comprising a diagnostic unit that acquires a feature amount related to the growth state of the crop using the target pixel.

16. The image diagnostic apparatus according to claim 15, wherein the feature amount includes a feature amount related to the color of the crop.

17. The learned model according to claim 1 or 2, wherein the learned model is a learned model that learns the target pixel using an image obtained by dividing a teacher image using a previously prepared image of the crop into a plurality of images.

18. The image diagnostic apparatus according to claim 1 or 2, wherein the target image is an image obtained by photographing the crop from above.

19. The image diagnostic apparatus according to claim 1 or 2, wherein the target image is one of the images obtained by continuously photographing the crop from a fixed viewpoint.

20. The image diagnostic apparatus according to claim 1 or 2, wherein the crop includes at least one of rice and wheat.

21. An image diagnostic system comprising the image diagnostic apparatus according to claim 1 or 2, and an imaging unit configured to image the crop.

22. The image diagnostic system according to claim 21, wherein the imaging unit is disposed remotely from the image diagnostic apparatus.

23. Determining target pixels that appear according to the growth of the crop in a target image obtained by photographing the crop, using at least one learned model selected from a plurality of learned models; and selecting the at least one learned model according to a growth index obtained using the target pixels.

24. A program for causing a computer to execute the image diagnostic method according to claim 23.

25. A storage medium storing the program according to claim 24.

Images