Image diagnostic apparatus, image diagnostic method, program, and storage medium

Abstract

To provide an image diagnostic method that enables highly accurate acquisition of a position of a plant stock.SOLUTION: An image diagnostic apparatus includes an estimation unit that estimates, from an image obtained by capturing a plurality of plants, a plurality of pixels that include an image of at least one of the plurality of plants. The image diagnostic apparatus includes a first acquisition unit that acquires a plurality of first coordinates corresponding to the plurality of plants, based on a connected component composed of two or more adjacent pixels among the plurality of pixels. The image diagnostic apparatus includes a second acquisition unit that acquires a plurality of second coordinates corresponding to the plurality of plants, based on information regarding distances between the plurality of first coordinates.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image diagnostic apparatus, an image diagnostic method, a program, and a storage medium for obtaining the position of a plant stock from a photographed image of a plant.

Background Art

[0002] Conventionally, a technique for determining the growth state from an image of a photographed plant is known. Patent Document 1 discloses a method for determining whether a stock has a specific trait by using the ratio of the area of each plant stock identified from an image of a field in which plants are planted to the area of the minimum circumscribed circle circumscribing each stock region.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, with the method of Patent Document 1, it is possible to identify the stocks of plants with large leaves such as pumpkins where the recognition of the stock region is easy, but for plants with thin leaves such as rice and wheat, there is a risk that the same stock region will be separated, making it difficult to identify the stocks.

[0005] An object of the present invention is to provide an image diagnostic method capable of accurately obtaining the position of a stock.

Means for Solving the Problems

[0006] As one aspect of the present invention, an image diagnostic apparatus includes an estimation unit that estimates a plurality of pixels including at least one image of a plurality of plants among images obtained by photographing the plurality of plants, a first acquisition unit that acquires a plurality of first coordinates corresponding to the plurality of plants based on a connected component composed of two or more adjacent pixels among the plurality of pixels, and a second acquisition unit that acquires a plurality of second coordinates corresponding to the plurality of plants based on information regarding distances between the plurality of first coordinates.

Advantages of the Invention

[0007] According to the present invention, it is possible to provide an image diagnostic method capable of accurately acquiring the positions of plants.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same elements are denoted by the same reference numerals, and redundant descriptions will be omitted as appropriate.

[0010] FIG. 1 is a hardware configuration diagram of an image diagnostic apparatus 100 according to an embodiment of the present invention. The image diagnostic apparatus 100 includes a system bus 101, a CPU 102, a ROM 103, a RAM 104, an HDD 105, a NIC 106, an input unit 107, and a display unit 108. The image diagnostic apparatus 100 performs stock recognition from a captured image obtained by photographing a plurality of plants (crops, objects) in a field from above. In the present embodiment, the case where the plant is rice (paddy rice) will be described.

[0011] An input image of rice photographed with a camera (not shown) is input to the input unit 107. The ROM 103 stores the captured image. The captured image may be stored in the HDD 105 or may be stored in an external storage device 110 via the Internet 109. The captured image stored in the ROM 103 or HDD 105 is read into the RAM 104. When the captured image is stored in the external storage device 110, it is read onto the RAM 104 via the NIC 106.

[0012] The CPU 102 executes each process of the flowchart of FIG. 2 described later.

[0013] The display unit 108 displays the information acquired by the CPU 102. When displaying each piece of information, the display unit 108 may display information such as each image and each value used for calculating each piece of information.

[0014] FIG. 2 is a flowchart showing an image diagnostic method executed by the CPU 102.

[0015] In step S10, the input unit 107 reads a photographed image of rice.

[0016] In step S20, the CPU 102 functions as a leaf region extraction unit (estimation unit), estimates pixels where rice exists from the photographed image, and extracts a leaf region composed of a plurality of pixels. In the present embodiment, the leaf region is extracted based on the color information of the photographed image. FIG. 3 is a diagram showing an example of the photographed image 301 and the leaf region image 302. When extracting a vegetation region, for example, RGBVI is used. RGBVI is a vegetation index focusing on the green color of leaves and is represented by the following formula (1).

[0017]

Equation

[0018] (1) In formula (1), when using a general digital camera, the values of each pixel of the two-dimensional image can be used as they are. Also, instead of obtaining the value of each pixel independently, the average value of a plurality of pixels within a predetermined range may be used. By performing threshold processing based on RGBVI, the leaf region is extracted and a leaf region mask is created. The leaf region mask is a binary image with the vegetation region in white (1) and the non-vegetation region in black (0).

[0019] Note that although RGBVI is used for the threshold processing, when using a camera capable of acquiring the infrared region, other indices such as NDVI may be used. Also, a combination of a plurality of indices and thresholds may be used.

[0020] Also, the acquisition of the leaf region is not limited to the extraction method using an index and a threshold, and other methods such as using a learned model generated by CNN may be used so that the input data is the photographed image 301 and the output data is the leaf region image 302.

[0021] In order to reduce errors due to noise in the captured image caused by the performance of the camera for the created leaf area image, noise processing such as morphological processing may be performed. Also, in order to prevent separation of the leaf areas, after performing convolution processing using an arbitrary filter such as a Gaussian distribution on the created leaf area image, threshold processing may be performed, and the image after a series of processing may be used as the leaf area image again. Not limited to these, other existing noise processing may be performed.

[0022] The captured image 301 shown in FIG. 3 is in monochrome display, but an RGB image is used for actual image diagnosis. When detecting leaves, a monochrome image in the absorption wavelength band of chloroplasts such as a wavelength of 700 nm may be used, but since detection accuracy is improved when using color information, it is preferable to use an image containing wavelength information of two or more bands.

[0023] Also, although it was determined that the leaf area was determined using color information from the RGB image, the leaf area may be determined from distance information. For example, by previously obtaining the camera height (distance between the camera and the ground) and setting pixels having values in the vicinity of the camera height in the distance image as outside the leaf area, the leaf area can be extracted from the distance image. When extracting the leaf area only from the distance image, since the RGB image is not used in subsequent processing, the image read into the input unit 107 in step S10 may be only the distance image.

[0024] Also, the range for analysis may be narrowed down to a part of the captured image 301 such as the analysis area 303 set at the center of the captured image 301.

[0025] In step S30, the CPU 102 functions as a first position information calculation unit (first acquisition unit) and calculates (acquires) the first position information (first coordinates) for each leaf area from the leaf areas extracted in step S20.

[0026] FIG. 4 is a diagram showing an example of processing by the CPU 102 that functions as the first position information calculation unit. First, the CPU 102 performs a process of assigning a label to each connected component composed of two or more adjacent pixels among a plurality of pixels estimated to include an image of at least one rice plant in the leaf region image 302. Here, as shown in FIG. 4(a), four connected components 401 are recognized in the leaf region image 302, and labels 1, 2, 3, and 4 are assigned to the four connected components 401 as shown in FIG. 4(b), respectively. Next, the CPU 102 calculates the area (region size) of each connected component and removes small connected components by threshold processing based on the area. Specifically, the CPU 102 leaves the connected components whose area is larger than a predetermined value (first predetermined value) and removes the connected components whose area is smaller than the predetermined value. The process when the area is equal to the predetermined value can be arbitrarily set (it may be set to leave or may be set to remove). Here, as shown in FIG. 4(c), the connected component 401 with label 4 is removed. By performing the threshold processing, it is possible to remove small weeds reflected in addition to the target plants such as rice and leaf regions erroneously detected due to image noise. Finally, the CPU 102 calculates the representative value of the obtained connected component 401 as the first position information. In the present embodiment, as shown in FIG. 4(d), as an example of the representative value, the center of gravity position of each connected component is calculated, and the calculated center of gravity position is acquired as the first position information 402.

[0027] In the case of thin-leaved plants such as rice targeted in this embodiment, the leaf region may be separated due to noise in the image itself. There is also a risk that the leaf region may be separated even by noise processing when calculating the leaf region such as morphological processing. In addition, when transplanting rice, several seedlings are planted in one place and treated as one plant. Therefore, even if the leaf region is completely acquired, there may be a case where one plant does not become a leaf region of the same connected component. For example, as shown in FIG. 5, in the leaf region image 502 (displayed as a grayscale image to indicate that it is a different connected component) corresponding to the captured image 501 in which one seedling is separated, it is separated into two connected components. The first position information 503 calculated in such a case is calculated as two, although there is only one plant in the captured image 501.

[0028] Therefore, in the present embodiment, in step S40, by correcting (combining) the first position information, the second position information (second coordinates), which is more accurate position information, is calculated.

[0029] In step S40, the CPU 102 functions as a second position information calculation unit (second acquisition unit), and calculates (acquires) the second position information by performing correction on the first position information based on the distance between the first position information calculated in step S30.

[0030] FIG. 6 is a diagram showing an example of processing by the CPU 102c functioning as the second position information calculation unit. First, the CPU 102 calculates the distance between the first position information 402 shown in FIG. 6(a). In FIG. 6(b), focusing on label 1, the distances between label 1 and 3 and between label 1 and 2 are calculated. Next, the CPU 102 determines whether the calculated distance is equal to or less than a threshold value (second predetermined value). Here, it is assumed that the distance between label 1 and 2 is equal to or less than the threshold value, and the distance between label 1 and 3 is greater than the threshold value. When the calculated distance is equal to or less than the threshold value, in the present embodiment, as shown in FIG. 6(c), the midpoint of the two pieces of first position information 402 is used as the second position information 601. When there is no other first position information with a distance equal to or less than the threshold value around the reference first position information, the reference first position information is directly treated as the second position information.

[0031] The above processing is performed for all the first position information. At this time, in order to prevent duplicate calculation of the same combination, it is preferable not to use the first position information that has already been used in the calculation. Also, when there are two or more pieces of first position information with a distance equal to or less than the threshold value, it is preferable to treat the average point of all of them as the second position information.

[0032] In this embodiment, the midpoint and the average point of the first position information are used as the second position information. However, position information weighted according to the area of each connected component may also be used. For example, as shown in FIG. 7, it is assumed that the second position information 701 is calculated based on three pieces of first position information 402. When the coordinates indicated by the three pieces of first position information 402 are (X1, Y1), (X2, Y2), (X3, Y3), respectively, and the areas of the connected components are S1, S2, S3, respectively, the coordinates (X est , Y est ) of the second position information 701 are calculated by the following formula (2).

[0033] [Number]

[0034] (2) In addition, not limited to the average value or the weighted position information, another calculation method may be used.

[0035] FIG. 8 is a diagram showing an example in which stock position detection is performed by calculating the second position information for the photographed image 301. As shown in FIG. 8, according to the configuration of this embodiment, the stock position can be accurately detected.

[0036] In step S50, the CPU 102 functions as a stock information output unit (third acquisition unit), and calculates and outputs information regarding the recognized stocks using the second position information calculated in step S40. Examples of the calculated and output information include stock interval, row interval, stock loss, individual difference of stocks (growth degree of stocks), planting variation, camera rotation, and measurement area (diagnosis area).

[0037] To calculate the plant spacing and row spacing, it is advisable to calculate the distance between two adjacent pieces of second position information. In plants such as rice, the plants are generally planted at equal intervals, and as shown in Fig. 9, since the plant spacing 901 is shorter than the row spacing 902, the intervals in two perpendicular directions can be calculated, and the one with the larger interval can be output as the row spacing. Also, by calculating a plurality of intervals and calculating their average, the average plant spacing and average row spacing can be calculated. Further, when the camera height is acquired in advance or a camera with a ranging function is used, the actual plant spacing and row spacing can be calculated from the camera height and calculated.

[0038] To calculate the missing plants, it is advisable to calculate the plant spacing and row spacing and detect a significantly large interval by comparing it with other plant spacings and row spacings. In the example shown in Fig. 10, since there is a plant missing in the plant spacing 1001, a significantly larger value is calculated compared to other plant spacings such as the plant spacing 901. By calculating the average plant spacing and detecting a significantly large plant spacing such as 1001, the missing plants can be detected. Note that significantly large means a value that is 1.5 times or more the average value of the plant spacing and row spacing.

[0039] To detect the growth degree of the plants, it is advisable to compare the areas of the connected components used when calculating the second position information. In the example shown in Fig. 11, the areas of the connected components are significantly different between the well-grown plant 1101 and the slow-growing plant 1102. By utilizing this and outputting the area of the connected component as the growth degree of the plant as it is, the individual differences of the plants can be provided to the user.

[0040] In order to calculate the planting variation and the camera rotation, it is advisable to calculate a straight line in the row-parallel direction based on the second position information in the row-parallel direction. In FIG. 12, the horizontal direction of the image is the row-parallel direction. In the example shown in FIG. 12, a straight line 1201 is calculated by the least squares method based on the second position information of the 7 plants in the second row from the lower side of the image. Since the distance between the straight line 1201 and the second position information of the 7 plants corresponds to the planting variation, the average value of this distance can be calculated and output to provide the planting variation to the user. In FIG. 12, one straight line is calculated, but a plurality of straight lines in the row-parallel direction may be calculated, the planting variation for each row may be calculated, output, and provided to the user. Also, the average value of the planting variation may be calculated and output.

[0041] Also, by calculating the angle between the straight line in the row direction and the horizontal direction of the captured image, the rotation angle of the camera may be calculated, output, and provided to the user.

[0042] In order to calculate the measurement area, it is advisable to set the area by connecting the midpoints of the second position information. In the example shown in FIG. 13, a 2×2 plant area is determined by setting the vertices of the measurement area 1301 as the midpoints of the 4 adjacent plants. This is not limiting, and other methods such as setting the vertices of the measurement area 1301 from the plant intervals and row intervals and the second position information may be used.

[0043] Also, it may be diagnosed whether appropriate camera installation can be performed from these information and presented to the user.

[0044] Note that the information calculated and output by the CPU is not limited to the above-mentioned ones, and other information may be calculated and output. Also, the calculation methods for each information are not limited to the methods listed in this embodiment, and other methods may be used.

[0045] As described above, according to the configuration of this embodiment, even in plants where one plant such as rice is separated, the recognition and estimation of the plant position can be performed with high accuracy from the captured image.

[0046] Note that the plants of the present invention are not limited to rice. Examples of plants other than rice include wheat. In the case of wheat as well, as with rice, often one plant separates, and the present invention can be applied. However, the plants are not limited to cases where they include at least one of rice and wheat, and if the present invention can be applied, the plants may be other plants (crops) other than rice and wheat. [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and having one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0047] The disclosure of this embodiment includes the following configurations and methods. (Configuration 1) An estimation unit that estimates a plurality of pixels including at least one image of the plurality of plants among the images obtained by photographing the plurality of plants, A first acquisition unit that acquires a plurality of first coordinates corresponding to the plurality of plants based on a connected component composed of two or more adjacent pixels among the plurality of pixels, An image diagnostic apparatus, comprising: a second acquisition unit that acquires a plurality of second coordinates corresponding to the plurality of plants based on information regarding distances between the plurality of first coordinates. (Configuration 2) The first acquisition unit according to Configuration 1, wherein the first acquisition unit removes the connected component having an area smaller than a first predetermined value and acquires the plurality of first coordinates using the connected component having an area larger than the first predetermined value. (Configuration 3) The plurality of first coordinates according to Configuration 2 are representative values for each of the connected components. (Configuration 4) The second acquisition unit according to any one of Configurations 1 to 3, wherein the second acquisition unit acquires the plurality of second coordinates based on threshold processing using distances between the first coordinates. (Configuration 5) When there is no first coordinate among the plurality of first coordinates for which the distance is less than a second predetermined value with respect to a predetermined first coordinate, the second acquisition unit acquires the predetermined first coordinate as the second coordinate, and when there is at least one first coordinate for which the distance is less than the second predetermined value with respect to the predetermined first coordinate, the second acquisition unit combines the predetermined first coordinate and the at least one first coordinate to acquire the second coordinate. The image diagnostic apparatus according to Configuration 4, characterized by the above. (Configuration 6) The image diagnostic apparatus according to any one of Configurations 1 to 5, further comprising a third acquisition unit that acquires at least one of the plant spacing and the row spacing using the distances between the plurality of second coordinates. (Configuration 7) When there is a plant spacing that is 1.5 times or more the average value of the plurality of plant spacings, or a row spacing that is 1.5 times or more the average value of the plurality of row spacings, the third acquisition unit determines that a plant is missing. The image diagnostic apparatus according to Configuration 6, characterized by the above. (Configuration 8) The image diagnostic apparatus according to any one of Configurations 1 to 7, further comprising a third acquisition unit that acquires the growth degree of the plant according to the area of the connected component corresponding to each of the plurality of second coordinates. (Configuration 9) The image diagnostic apparatus according to any one of Configurations 1 to 8, further comprising a third acquisition unit that acquires a diagnostic region using the plurality of second coordinates. (Configuration 10) The image is an image obtained by photographing the plant from above. The image diagnostic apparatus according to any one of Configurations 1 to 9, characterized by the above. (Configuration 11) The plant includes at least one of rice and wheat. The image diagnostic apparatus according to any one of Configurations 1 to 10, characterized by the above. (Method 1) Estimating a plurality of pixels including an image of at least one of the plurality of plants in an image obtained by photographing the plurality of plants; Obtaining a plurality of first coordinates corresponding to the plurality of plants based on a connected component composed of two or more adjacent pixels among the plurality of pixels; Obtaining a plurality of second coordinates corresponding to the plurality of plants based on information regarding distances between the plurality of first coordinates, the method for image diagnosis being characterized by including these steps. (Configuration 12) A program characterized by causing a computer to execute the method for image diagnosis according to Method 1. (Configuration 13) A computer-readable storage medium storing the program according to Configuration 12.

[0048] As described above, the preferred embodiments of the present invention have been explained, but the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

Explanation of Signs

[0049] 100 Image diagnosis apparatus 102 CPU (Estimation unit, First acquisition unit, Second acquisition unit)

Claims

1. An estimation unit that estimates a plurality of pixels including at least one image of the plurality of plants among images obtained by photographing the plurality of plants; A first acquisition unit that acquires a plurality of first coordinates corresponding to the plurality of plants based on a connected component composed of two or more adjacent pixels among the plurality of pixels; An image diagnostic apparatus, comprising: a second acquisition unit that acquires a plurality of second coordinates corresponding to the plurality of plants based on information regarding distances between the plurality of first coordinates.

2. The image diagnostic apparatus according to claim 1, wherein the first acquisition unit removes the connected component having an area smaller than a first predetermined value and acquires the plurality of first coordinates using the connected component having an area larger than the first predetermined value.

3. The image diagnostic apparatus according to claim 2, wherein the plurality of first coordinates are representative values for each of the connected components.

4. The image diagnostic apparatus according to claim 1 or 2, wherein the second acquisition unit acquires the plurality of second coordinates based on threshold processing using the distances.

5. The image diagnostic apparatus according to claim 4, wherein when there is no first coordinate having a distance smaller than a second predetermined value with respect to a predetermined first coordinate among the plurality of first coordinates, the second acquisition unit acquires the predetermined first coordinate as the second coordinate, and when there is at least one first coordinate having a distance smaller than the second predetermined value with respect to the predetermined first coordinate, the second acquisition unit synthesizes the predetermined first coordinate and the at least one first coordinate to acquire the second coordinate.

6. The image diagnostic apparatus according to claim 1 or 2, further comprising: a third acquisition unit that acquires at least one of an inter-stock distance and an inter-row distance using distances between the plurality of second coordinates.

7. The image diagnostic apparatus according to claim 6, wherein the third acquisition unit determines that a stock is missing when there is an inter-stock distance that is 1.5 times or more the average value of the plurality of inter-stock distances, or an inter-row distance that is 1.5 times or more the average value of the plurality of inter-row distances.

8. The image diagnostic apparatus according to claim 1 or 2, further comprising: a third acquisition unit that acquires a growth degree of a stock according to an area of the connected component corresponding to each of the plurality of second coordinates.

9. The image diagnostic apparatus according to claim 1 or 2, further comprising: a third acquisition unit that acquires a diagnostic region using the plurality of second coordinates.

10. The image diagnostic apparatus according to claim 1 or 2, wherein the image is an image obtained by photographing the plant from above.

11. The image diagnostic apparatus according to claim 1 or 2, wherein the plant includes at least one of rice and wheat.

12. A step of estimating a plurality of pixels including an image of at least one of the plurality of plants from among images obtained by photographing the plurality of plants; A step of obtaining a plurality of first coordinates corresponding to the plurality of plants based on a connected component composed of two or more adjacent pixels among the plurality of pixels; An image diagnostic method comprising: a step of obtaining a plurality of second coordinates corresponding to the plurality of plants based on information regarding distances between the plurality of first coordinates.

13. A program characterized by causing a computer to execute the image diagnostic method according to claim 12.

14. A computer-readable storage medium storing the program according to claim 13.

Images