Detection method, detection device, and program

Abstract

To provide a detection method capable of creating two binarized images from an image of a plant by using two threshold values and detecting lateral branches of the plant via the two binarized images.SOLUTION: A detection method (S1) for detecting lateral branches of a plant from an RGB image includes: a first generation step (S11) of generating a first gray scale image from the RGB image of the plant; second generation steps (S12 to S16) of creating two binarized images for the first gray scale image by using two predetermined threshold values, and generating a second gray scale image from the two binarized images and the first gray scale image; an enhancement step (S17) of converting the second gray scale image into an edge image enhancing contours of a region including the lateral branches; and a straight line acquisition step (S18) of acquiring a formula of a straight line indicative of at least a part of the lateral branches from the edge image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a detection method, a detection device, and a program. [Background technology]

[0002] A technique for creating a binary image from an image of a plant using a single threshold value and detecting the plant through the binary image is known as a conventional technique. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-008382 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional techniques described above have the problem that they use only one threshold value when creating a binary image and target the entire plant body as the detection target, making it impossible to detect only some components of the plant, such as side branches.

[0005] An aspect of the present invention aims to create two binarized images from an image of a plant using two threshold values, and to detect lateral branches of the plant through the two binarized images. [Means for solving the problem]

[0006] In order to solve the above problems, a detection method according to one embodiment of the present invention is a detection method for detecting side branches of a plant from an RGB image, and includes: a first generation step of generating a first grayscale image from the RGB image of the plant; a second generation step of creating two binary images for the first grayscale image using two predetermined thresholds and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement step of converting the second grayscale image into an edge image in which the contour of an area including the side branches is enhanced; and a straight line acquisition step of acquiring an equation for a straight line indicating at least a part of the side branches from the edge image.

[0007] In order to solve the above problems, a detection device according to one embodiment of the present invention is a detection device that detects side branches of a plant from an RGB image, and includes a first generation unit that generates a first grayscale image from the RGB image of the plant, a second generation unit that creates two binary images for the first grayscale image using two predetermined thresholds and generates a second grayscale image from the two binary images and the first grayscale image, an enhancement unit that converts the second grayscale image into an edge image that enhances the contours of areas including the side branches, and a straight line acquisition unit that acquires an equation for a straight line that indicates at least a portion of the side branches from the edge image.

[0008] In order to solve the above-mentioned problems, a program according to one embodiment of the present invention causes a computer to execute a first generation process of generating a first grayscale image from an RGB image of a plant; a second generation process of creating two binary images for the first grayscale image using two predetermined thresholds and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement process of converting the second grayscale image into an edge image that enhances the contours of areas including side branches; and a straight line acquisition process of acquiring an equation for a straight line that indicates at least a portion of the side branches from the edge image. Effect of the Invention

[0009] According to one aspect of the present invention, two binary images are created from an image of a plant using two threshold values, and lateral branches of the plant can be detected via the two binary images. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a detection device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a flow chart showing the flow of a detection method according to the first embodiment of the present invention. [Diagram 3] This is an example of a grayscale image generated from an RGB image of a plant. [Figure 4] 13 is an example of a process for creating a mask image via a binarized image. [Diagram 5] 13 is an example of mask processing for a grayscale image. [Figure 6] 13 is an example of an image in which edges are enhanced by filter processing. [Figure 7] 11 is a diagram illustrating an example of a flow of a straight line acquisition process. [Figure 8] FIG. 5 is a block diagram showing a configuration of a detection device according to a second embodiment of the present invention. [Figure 9] 10 is a flowchart showing the flow of a point detection method according to a second embodiment of the present invention. [Figure 10] FIG. 6 is a flowchart showing the flow of a detection method according to a second embodiment of the present invention. [Figure 11] 1 is an example of an image of a node and tip on a lateral branch of a plant. [Figure 12] This is an example of grouping straight lines indicating at least a portion of a lateral branch and points indicating nodes and tips. [Figure 13] FIG. 13 is a diagram illustrating an example of a group detection range. [Figure 14] 13 is an example of a search range for points indicating adjacent nodes. [Figure 15] This is an example of a group in which lines are detected overlapping each other. [Figure 16] 13 is an example image of a side branch with the addition of lines and points detected as a group. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [Embodiment 1] Hereinafter, an embodiment of the present invention will be described in detail. This embodiment is a basic form of the embodiments described later.

[0012] (Overview of detection device 1) In recent years, with the decline in the agricultural population, automation is also desired in plant cultivation. For example, in the cultivation of fruit trees and other plants, many tasks are required before harvest, such as pinching off the tips of the side branches of fruit trees to concentrate nutrients in the fruit, training the extended side branches to fix them to the trellis, and pruning the side branches.

[0013] Here, a lateral branch is a shoot, or new branch, that grows from the side of a main branch or sub-main branch in early spring. Such lateral branches are often green and have high brightness (luminance). Furthermore, lateral branches tend to grow in an almost straight line. Also, a related term is a node, which is a point from which a new branch or leaf grows from an existing branch, that is, a growth point. As a branch grows, the number of nodes on the branch can increase. Additionally, an internode is the part of a branch between nodes.

[0014] For example, to automate the aforementioned tasks using a robot, etc., technology is needed that can detect plant lateral branches from captured images. However, because lateral branches, for example, have a green color and high brightness (luminance) as mentioned above, they are difficult to distinguish from the background of surrounding plant organs in the image, and are difficult to detect even in color spaces such as HSV (Hue, Saturation, Value) and HIS (Hue, Intensity, Saturation).

[0015] The detection device 1 according to the present embodiment is, for example, a device for detecting such side branches from an image. Specific examples of plants to be detected include grapevines as vines, persimmons as deciduous trees, and apples, pears, peaches, plums, apricots, and plums as fruit trees of the Rosaceae family.

[0016] (Configuration of detection device 1) The configuration of the detection device 1 according to this embodiment will be described. Fig. 1 is a block diagram showing the configuration of the detection device 1 according to this embodiment. The detection device 1 is a detection device that detects lateral branches of a plant from an RGB image. The detection device 1 includes a control unit 10, an input unit 11, a storage unit 12, and an output unit 13.

[0017] (Control unit 10) The control unit 10 comprehensively controls each unit of the detection device 1. The control unit 10 includes a first generation unit 101, a second generation unit 102, an emphasis unit 103, and a straight line acquisition unit 104.

[0018] (First generation unit 101) The first generating unit 101 generates a first grayscale image from an RGB (Red, Green, Blue) image of a plant. The RGB image of a plant is, for example, an image of an outdoor plant captured using visible light.

[0019] (Grayscale conversion of RGB plant images) Figure 3 shows examples of grayscale images generated from an RGB image of a plant using different methods. Image 301 in Figure 3 is an example of an image obtained by converting an RGB image of a plant into a grayscale image using RGB values. With this method, color information specific to the plant, mainly green in the RGB image, is not reflected in the grayscale image, which may make it difficult to distinguish the plant area from the grayscale image.

[0020] In order to reflect the color information specific to plants in an RGB image in a grayscale image, for example, a vegetation index is used. The vegetation index is an index for converting a captured RGB image into a grayscale image so that the plants covering the ground can be detected. In this case, the background of the plants is mainly low-luminance soil.

[0021] The image 302 in FIG. 3 is a vegetation index conversion formula, which is a known technique, for the RGB image of the plant. (3*G-2.4*RB) / (R+G+B) This is an example of an image that has been grayscaled using the vegetation index. As mentioned above, the vegetation index is an index for detecting plants that cover the ground. However, side branches often grow above the plant, and when a plant that includes side branches is photographed from the ground, the background of the plant is mainly the bright sky. If the vegetation index is then applied directly to plants that exist in the air, the boundary between the aerial plant and the sky in the background will be unclear, as in this image, meaning there will be a lack of contrast.

[0022] Therefore, in order to make the boundary between the aerial plant and the sky clear, for example, the first generating unit 101 may generate the first grayscale image using a conversion formula that converts the RGB value of one pixel in the area of ​​the side branches in the RGB image to a single positive integer. Also, for example, the conversion formula may be a conversion formula that adds a predetermined constant term to a value obtained by adding, subtracting, and multiplying a real value indicating the RGB value of one pixel, and processes fractions after the decimal point to convert the value to a single positive integer. As a specific example of the conversion formula, in the image 303 of FIG. 3, the RGB image of the plant is converted to the following while referring to the above-mentioned publicly known vegetation index: (3*G-2.4*R-B+100) The result of is converted to integer values ​​and converted to grayscale. In this example image, the boundary between the plant and the sky in the background is clear. Using the above conversion formula, plants in the air can be detected from the image with high accuracy.

[0023] First, in the above-mentioned conversion method, by avoiding division by (R+G+B), it becomes easier to detect plants even when the image has a bright sky in the background.

[0024] In the above-mentioned conversion formula method, by adding 100 as a constant term, the converted numerical value in, for example, the side branch becomes a positive value. In the above-mentioned conversion formula method, the conversion result is converted into an integer value. By using these methods, the image of the side branch after conversion can be converted into an image format such as JPEG, in which the RGB value is an integer value in the range of 0 to 255.

[0025] (Second generation unit 102) The second generating unit 102 generates two binary images from the first grayscale image using two predetermined thresholds, and generates a second grayscale image from the two binary images and the first grayscale image. The second generating unit 102 may include, for example, a mask creating unit 1021 and a mask processing unit 1022 as shown in FIG.

[0026] (Mask Creation Unit 1021) The mask creation unit 1021 may create a mask image from, for example, two binarized images.

[0027] (The process of creating a mask image via two binary images) FIG. 4 shows an example of a process for creating a mask image from a first grayscale image via two binarized images.

[0028] For example, in the flow of Fig. 4, it is assumed that the range of brightness values ​​including branches is predetermined as 50 to 120 by numerical analysis or the like. Then, it is assumed that the lower and upper limits of the brightness values, 50 and 120, are predetermined as two thresholds in the binarization process described below. Note that the range of brightness values ​​including branches may vary depending on, for example, the color of the branch, specifically, the amount of chlorophyll contained in the branch.

[0029] In the example of Fig. 4, the mask creation unit 1021 creates two binary images from the first grayscale image 401 in Fig. 4 using the two thresholds described above, as in image 402 or image 403 in Fig. 4. These two binary images are created by coloring pixels with luminance values ​​greater than the thresholds white and pixels with luminance values ​​less than the thresholds black for each of the two thresholds described above for the first grayscale image 401.

[0030] Image 402 in Fig. 4 is an example of a binarized image when the threshold is 50. In this binarized image, branches and leaves are extracted as white areas. Image 403 in Fig. 4 is an example of a binarized image when the threshold is 120. In this binarized image, leaves are extracted as white areas, but branches are black areas.

[0031] 4, the mask creation unit 1021 performs an exclusive OR (XOR) operation on the two binary images, images 402 and 403, as shown in a truth table 404, to create a mask image 405 in which branches are extracted as white parts. In the truth table 404 in FIG. 4, for the color of each pixel in the image, white corresponds to a truth value of 1, and black corresponds to a truth value of 0. By performing the XOR operation in this manner, for example, a branch region that is white only in image 402 of the two binary images becomes white in the mask image 405, and a leaf region that is white in both the two binary images becomes black in the mask image 405.

[0032] (Mask processing unit 1022) For example, the mask processing unit 1022 may perform mask processing on the first grayscale image using a mask image to generate a second grayscale image.

[0033] (Mask processing for grayscale images) Fig. 5 is an example of mask processing for a grayscale image. In the example of Fig. 5, the mask processing unit 1022 performs mask processing on the first grayscale image 401 to leave only the branch region, which is white in the mask image 405, and generates a second grayscale image 501.

[0034] (Emphasis part 103) The highlighting unit 103 converts the second grayscale image into an edge image in which the contour of the region including the lateral branch is highlighted. The highlighting unit 103 may include, for example, a filter unit 1031 and a luminance extraction unit 1032 as shown in FIG.

[0035] (Filter section 1031) The filter unit 1031 may perform, for example, Laplacian filtering, which is a known technique, to emphasize the contour of the region including the side branches.

[0036] (Example of an image with contours emphasized through filtering) Fig. 6 shows examples of images in which the contours of the region including the side branches in the second grayscale image are enhanced by various types of filtering processes that are known techniques. The examples in Fig. 6 show images in which the Sobel x, canny, or Laplacian filtering methods are used as filtering methods.

[0037] Images 601, 603, and 605 in the upper part of Fig. 6 are examples of images obtained by enlarging the areas including side branches in the second grayscale image 501 obtained by filtering the second grayscale image 501 using the above-mentioned methods. Images 602, 604, and 606 in the lower part of Fig. 6 are examples of images obtained by further enlarging the contours of the side branches in the images obtained by enlarging the areas including the side branches.

[0038] After the second grayscale image is filtered to enhance the contours, the contours of the side branches are approximated by straight lines, as described below.

[0039] In the Sobel x and canny filter processes, the contour of the side branch is divided into segments, as shown in the images 602 and 604 in the lower part of Fig. 6. In this case, the contour of the side branch cannot be accurately approximated by a straight line.

[0040] In Laplacian filtering, the contour of the side branch is continuous and not divided, as shown in the image 606 in the lower part of Fig. 6. In this case, the contour of the side branch can be accurately approximated by a straight line.

[0041] The filtering method is not limited to the Laplacian filter, and any other known filtering method may be used as long as it can accurately approximate the contour line of the side branch into a straight line.

[0042] (Luminance extraction unit 1032) The luminance extraction unit 1032 may extract, for example, an area having a luminance value equal to or greater than a predetermined value from an image that has been subjected to Laplacian filter processing, and may use this extraction processing to enhance the contour of an area that includes a lateral branch.

[0043] For example, as shown in FIG. 6, the luminance extraction unit 1032 may extract only a region having a luminance value of 200 to 255 as a predetermined luminance value from an image that has been subjected to Laplacian filtering.

[0044] (Line acquisition unit 104) The straight line acquisition unit 104 acquires an equation of a straight line indicating at least a part of a side branch from the edge image. Here, at least a part of a side branch may be, for example, an internode in the side branch. Furthermore, for example, a side branch may have a portion where two or more internodes are consecutive. For example, the straight line acquisition unit 104 may include a straight line extraction unit 1041 and an equation acquisition unit 1042 as shown in FIG. 1.

[0045] (Line extraction part 1041) The line extracting unit 1041 may extract a line indicating at least a part of the lateral branch from the edge image using, for example, a line extraction technique.

[0046] The line extraction unit 1041 may use, for example, a known technique, LSD (Line Segment Detector), as a line extraction method. For example, in the case of LSD, a part approximated by a straight line is extracted, including information on the gradient of the line, so that the accuracy of line extraction is high.

[0047] The line extraction method is not limited to LSD, and any known line extraction method may be used, such as ED (Edge Drawing) Lines or MCMLSD (Markov Chain Marginal Line Segment Detector).

[0048] (The process of obtaining a straight line that shows at least a part of a side branch from the original image of the plant) Fig. 7 shows a flow of obtaining an equation of a straight line indicating at least a part of a side branch from an original image of a plant. In the example of Fig. 7, the straight line extraction unit 1041 extracts lines such as the contours of side branches that are emphasized on an image 702 with emphasized contours, i.e., an edge image, from an original image 701 using a straight line extraction method.

[0049] In the example of Fig. 7, information on straight lines extracted by a straight line extraction method is included in image 703. In the case of this image, approximate straight lines of straight structures such as the iron wire of a greenhouse may be extracted in addition to approximate straight lines related to side branches, but the invention according to the embodiment described below makes it possible to select only approximate straight lines related to side branches.

[0050] (Formula acquisition unit 1042) The equation obtaining unit 1042 may obtain the equation of the straight line extracted using, for example, a straight line extraction method.

[0051] 7, the equation acquiring unit 1042 acquires the equations of the lines extracted using the line extraction method, as shown in image 704. The equations of the lines may be acquired as an equation, for example, in the form of Ax+By+C=0.

[0052] (Input section 11) The input unit 11 is an interface that acquires various data referred to by the control unit 10. For example, the input unit 11 acquires image data referred to by the control unit 10. The input unit 11 may be configured to include at least one of various input devices such as a keyboard, a mouse, and a touch pad. The input unit 11 may also be a wired or wireless interface for connecting such an input device to the detection device 1.

[0053] (Storage unit 12) The storage unit 12 stores various types of data referenced by the control unit 10. For example, the storage unit 12 stores image data and data on equations of straight lines.

[0054] (Output section 13) The output unit 13 is an interface that outputs various result data generated or detected by the control unit 10. For example, the output unit 13 outputs result data of a detection process.

[0055] (Effect of detection device 1) As described above, the detection device 1, as a detection device for detecting side branches of a plant from an RGB image, includes a first generation unit that generates a first grayscale image from the RGB image of the plant, a second generation unit that creates two binary images for the first grayscale image using two predetermined thresholds and generates a second grayscale image from the two binary images and the first grayscale image, an enhancement unit that converts the second grayscale image into an edge image that enhances the contours of areas including the side branches, and a straight line acquisition unit that acquires an equation for a straight line that indicates at least a part of the side branch from the edge image.

[0056] According to the detection device 1 configured as described above, two binary images can be created from an image of a plant using two threshold values, and side branches of the plant can be detected via the two binary images.

[0057] (Flow of detection method S1) The flow of the detection method S1 executed by the detection device 1 will be described with reference to Fig. 2. The detection method S1 is a detection method for detecting lateral branches of a plant from an RGB image. Fig. 2 is a flow diagram showing the flow of the detection method S1 according to this embodiment.

[0058] (Step S11) In step S11, the first generating unit 101 generates a first grayscale image from the RGB image of the plant. For example, the first generating unit 101 may generate the first grayscale image using a conversion formula that converts the RGB value of one pixel in the region of the side branch in the RGB image to a single positive integer. Also, for example, the conversion formula may be a conversion formula that adds a predetermined constant term to a value obtained by adding, subtracting, and multiplying a real value indicating the RGB value of one pixel, and processes fractions after the decimal point to convert the value to a single positive integer.

[0059] (Step S12) In step S12, the second generating unit 102 generates two binarized images for the first grayscale image by using two predetermined threshold values.

[0060] (Steps S13 to S14) In steps S13 and S14, the second generating unit 102 generates a second grayscale image from the two binarized images and the first grayscale image.

[0061] (Step S13) In step S13, the mask creation unit 1021 may create a mask image for extracting side branches from, for example, two binarized images. For example, the mask creation unit 1021 may create the mask image by performing an exclusive OR (XOR) operation on the two binarized images.

[0062] (Step S14) In step S14, the mask processing unit 1022 may, for example, perform mask processing on the first grayscale image using a mask image to generate a second grayscale image as a region including side branches.

[0063] (Step S15) In step S15, the highlighting unit 103 converts the second grayscale image into an edge image in which the contour of the region including the side branch is highlighted. For example, in step S15, the filter unit 1031 may highlight the contour of the region including the side branch by performing Laplacian filter processing. Also, for example, the brightness extraction unit 1032 may extract a region having a brightness value equal to or greater than a predetermined brightness value from the image after the Laplacian filter processing.

[0064] (Step S16) In step S16, the straight line acquisition unit 104 acquires an equation of a straight line indicating at least a part of the side branch from the edge image. For example, in step S16, the straight line extraction unit 1041 may use a straight line extraction method to extract a straight line indicating at least a part of the side branch from the edge image. Also, the straight line extraction unit 1041 may use, for example, a Line Segment Detector (LSD), Edge Drawing Lines (ED), or Markov Chain Marginal Line Segment Detector (MCMLSD) as the straight line extraction method. Also, for example, the equation acquisition unit 1042 may acquire an equation of the straight line extracted using the straight line extraction method.

[0065] (Effect of detection method S1) As described above, detection method S1, as a detection method for detecting side branches of a plant from an RGB image, employs the following configuration: a first generation step of generating a first grayscale image from the RGB image of the plant; a second generation step of creating two binary images for the first grayscale image using two predetermined thresholds and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement step of converting the second grayscale image into an edge image in which the contours of areas including the side branches are enhanced; and a straight line acquisition step of acquiring an equation for a straight line indicating at least a part of the side branch from the edge image.

[0066] According to the detection method S1 configured as described above, two binary images are created from an image of a plant using two threshold values, and side branches of the plant can be detected via the two binary images.

[0067] [Embodiment 2] Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

[0068] (Overview of detection device 2) In the cultivation of plants such as fruit trees, for example, when a side branch grows and the number of nodes on the side branch reaches a predetermined number, it is determined that it is the right time to perform the above-mentioned work such as pinching, training, pruning, etc. For example, when the number of nodes on the side branch reaches five or more, it may be determined that it is the right time to perform the above-mentioned work. It is a heavy burden for workers to go around and judge the right time for such work by looking at each tree in an orchard. In order to judge the right time for such work from an image, it is necessary to group the nodes and tips contained in one side branch and count the number of nodes present on one side branch.

[0069] The detection device 2 according to this embodiment is a device for detecting, for example, the nodes and tips of one side branch on an image as one group. The generation device according to this embodiment is a device for executing the same process as the detection device 2 described above and displaying the process result on an image.

[0070] (Configuration of detection device 2) The configuration of the detection device 2 according to this embodiment will be described. FIG. 8 is a block diagram showing the configuration of the detection device 2 according to this embodiment. The detection device 2 is a detection device that detects the nodes and tips of one side branch of a plant as one side branch group. A generation device according to this embodiment will also be described. The generation device is a generation device that generates an image in which the nodes and tips of one side branch of a plant are displayed as one side branch group. The detection device 2 and the generation device each include a control unit 20, an input unit 11, a memory unit 12, and an output unit 13.

[0071] (Control unit 20) The control unit 20 comprehensively controls each unit of the detection device 2. In the detection device 2, the control unit 20 includes a point detection unit 201, a line detection unit 202, a group detection unit 203, and a point addition unit 204. In the detection device 2, the control unit 20 may further include, for example, an overlapping group selection unit 205 and an overlapping point selection unit 206.

[0072] In the generating device, the control unit 20 includes an image adding unit 207 in addition to the same components as those in the detecting device 2 .

[0073] (Point detection unit 201) The point detection unit 201 detects points indicating the nodes and tips of the side branches of the plant from an image of the plant.

[0074] 11 shows example images of nodes and tips of a lateral branch of a plant. In FIG. 11, image 1101 is an example image of a lateral branch. Image 1102 is an enlarged view of the tip of the lateral branch in image 1101. In image 1102, a is an example of the tip, and b and c are examples of nodes in the lateral branch. Such tips and nodes are the detection targets of point detection unit 201.

[0075] The point detection unit 201 may detect the coordinates of points indicating the nodes and tips of the lateral branches of the plant, for example, by using a machine learning model. A specific example of the machine learning model used for the detection is YOLOv4, but other known machine learning models may also be used for the detection.

[0076] (Straight line detector 202) The line detection unit 202 detects a line indicating at least a part of a side branch of a plant from an image of the plant. The line detection unit 202 has the same function as the detection device 1 described in the first embodiment, and therefore a description thereof will be omitted.

[0077] (Group detection unit 203) The group detection unit 203 calculates the distance between the points and the straight line, and detects, as one group, a straight line indicating at least a part of one side branch and one or more points existing within a predetermined distance from the straight line.

[0078] Fig. 12 is an example of grouping a straight line indicating at least a part of a side branch with points indicating a node and a tip. In the example of Fig. 12, in an image 1201, in a straight side branch, point a indicating the tip is indicated by a black cross, and points b and c indicating the nodes are indicated by white crosses. In the example of Fig. 12, image 1201 includes information on the points indicating the detected nodes and tips, and the straight line indicating at least a part of the side branch.

[0079] In the example of Figure 12, the group detection unit 203 calculates the distance between points and lines in an image 1201, and detects a line indicating at least a part of a side branch and one or more points that are within a predetermined distance from the line as one group.

[0080] In this embodiment, when points that have been detected as one group during group detection processing are to be distinguished from undetected points, the detected points are denoted as P1, P2, P3, ... Pn. In the example of Fig. 12, points b and c in image 1201 are detected as one group in the above-mentioned group detection processing, and are denoted as points P1 and P2, respectively, in image 1202 after the group detection. Group detection of lines will be described later in the explanation of Fig. 15.

[0081] Furthermore, for example, the group detection section 203 may detect a point and a line as one group when the distance between the point and the line and a predetermined designated range from the line satisfy the following formula.

number

[0082] Furthermore, the group detection unit 203 may detect, for example, shoots with three or more nodes, that is, side branches, as one group, as shown in FIG.

[0083] (An example of a grouping process) For example, when the diameter of one side branch is K and a value determined in advance with reference to K is X, the group detection unit 203 calculates a set of straight lines detected as one group together with the detected point as a rotation center by using the detected point as a rotation center. Angle θ=tan -1 (3*K / X)*180 / π It is also possible to search for undetected points within the range.

[0084] In this case, the value of X is, for example, K:X=1:8~1:13 may be determined in advance to satisfy the following:

[0085] Fig. 13 shows an example of a group detection range. In the example of Fig. 13, a straight line indicating at least a part of one side branch is L, a point that has been detected as one group together with the straight line L is node 2, and an undetected point that exists next to node 2 is node 1. In this case, the length between nodes 1 and 2 is assumed to be X.

[0086] Fig. 14 shows an example of a search range for a point indicating an adjacent node. In the example of Fig. 14, the center of rotation of the above-mentioned angle θ, i.e., the point below the triangle in Fig. 14, is taken as node 2, and a straight line passing through the center of the triangle in the vertical direction is taken as line L. In this case, for example, searching for an undetected point within the range of angle θ from line L corresponds to searching the upper side of the triangle in Fig. 14.

[0087] For example, in the examples of Figs. 13 and 14, If K=1 and X=8, the angle θ is approximately ±20.55 degrees. If K = 1 and X = 13, the angle θ is approximately ±12.99 degrees.

[0088] Using the above-mentioned equation for the angle θ, it is possible to calculate the angle θ and search for undetected points even if the diameter of the side branch and the assumed internode length are different.

[0089] (Point addition part 204) The point adding unit 204 adds a point that was not detected in any group by the group detecting unit 203 to one of the groups. For example, the tip of a side branch is thin and soft, and therefore often has a curved shape. In this case, the distance between the point indicating the tip of the side branch and the straight line indicating at least a part of the side branch does not satisfy the above-mentioned specified range, and the point may not be detected in any group by the group detecting unit 203. For example, the point adding unit 204 adds such a point to one of the groups.

[0090] 14, it is assumed that the straight line L, node 1, and node 2 indicating at least a part of the side branch are detected as one group, and the point indicating the tip of the side branch is not detected in any group. In this case, for example, the point adding unit 204 sets the point indicating the tip of the side branch as point A, the point indicating node 1 as the point closest to point A as point B, the point indicating node 2 as the point closest to point B excluding point A as point C, and the distance from point B to point C as X, and satisfies the following conditional expressions (1) and (2). Distance from line L, which includes point B, to point A <X / 2 …(1) Distance from point A to point B <X / 2 …(2) If the above equation is satisfied, point A may be added to a group including point B. Also, for example, half the distance X / 2 from point B to point C may be calculated using the following formula, with point B as node p1 and point C as node p2.

number

[0091] (Duplicate Group Selection Unit 205) The overlapping group selection unit 205 may select, for example, a group in which points are detected as overlapping by the group detection unit 203 .

[0092] As described in the first embodiment, a straight line indicating at least a part of a side branch is detected from the contour of the side branch. However, the contour of the side branch may be divided into multiple parts at a node or the like of the side branch, even after filtering. In this case, multiple straight lines may be detected from one side branch. Data of the detection result by the detection device 2 may be used for automation of work on the side branch using a robot or the like, as described in the first embodiment. In this case, if multiple straight lines corresponding to one side branch are detected, the work of the robot or the like on one side branch may be duplicated. In order to prevent such duplicated work by the robot or the like from occurring, for example, when multiple straight lines are detected from one side branch, the overlapping group selection unit 205 may select the multiple straight lines so that only one straight line corresponds to the one side branch.

[0093] Figure 15 shows an example of a group where overlapping straight lines are detected. In the example in Figure 15, four straight lines (L1, L2, L3, L4) are detected from a single central side branch based on the outline of the side branch that is divided near the node points (P1, P2, P3).

[0094] 15, if we consider the line L1 as an extension in both the up and down directions, then when P1, P2, and P3 are within a predetermined distance from line L1, (L1, P1, P2, P3) are detected as one group. Lines L2, L3, and L4 can also be considered in the same way as line L1.

[0095] As a result of considering L1, L2, L3, and L4, in the example of FIG. 15, (L1, P1, P2, P3) (L2, P1, P2, P3) (L3, P1, P2, P3) (L4, P1, P2, P3) It is assumed that the following four groups are detected. These groups are referred to as (group L1, group L2, group L3, group L4) from the top.

[0096] These four groups all share the same points (P1, P2, P3) that indicate nodes. Since these points are considered to have all been detected from the same branch, it is determined that the L1, L2, L3, and L4 groups are groups that have been detected in duplicate from the same branch. At this time, the duplicate group selection unit 205 may select from the four groups, for example, to leave only the L1 group.

[0097] (Duplicate point selection unit 206) The overlapping point selection section 206 may select, for example, points that are detected overlappingly in multiple groups by the group detection section 203 .

[0098] For example, if the same point is detected in multiple different groups, the number of points in each group may be checked, and the point may be determined to be in the group with the largest number of points.

[0099] Now consider the following example. The group to which the line L1 belongs (hereafter referred to as the L1 group) is composed of (L1, P1, P2, P3, P4, P6, P10) → the number of points in the group is 6: (P1, P2, P3, P4, P6, P10) The group to which the line L7 belongs (hereafter referred to as the L7 group) is composed of (L7, P1, P7, P11) → the number of points in the group is three: (P1, P7, P11) In these examples, point P1 is detected in two groups, group L1 and group L7. In this case, since the number of points in the group L1 is greater, for example, the overlapping point selection unit 206 may determine that point P1 in group L7 is a overlapping detected point, and select points in group L7 so that only (P7, P11) are included.

[0100] As a specific example corresponding to the above-mentioned examples of the L1 group and the L7 group, a state in which a side branch corresponding to the L7 group branches off from a side branch corresponding to the L1 group at a node corresponding to point P1 is conceivable. By selecting points that are detected in duplicate as described above and correctly grasping the number of nodes in one side branch, it becomes possible to determine the optimum time to perform work on that side branch based on the number of nodes in the side branch, for example, as described in the overview of the detection device 2.

[0101] (Image Addition Section 207) The image adding unit 207 adds lines indicating lateral branches and points indicating the nodes and tips of the lateral branches to the image of the plant.

[0102] Fig. 16 is an example of an image of a side branch to which a line and a point detected as one side branch group have been added. In the example of Fig. 16, a straight line Lx within one side branch group is drawn from point a at the tip of the group to point c at the bottom in the side branch image.

[0103] (Effect of detection device 2) As described above, detection device 2 is a detection device that detects the nodes and tips on a side branch of a plant as a side branch group, and includes a point detection unit that detects points indicating the nodes and tips on the side branch of the plant from an image of the plant, a line detection unit that detects a straight line indicating at least a part of the side branch of the plant from the image of the plant, a group detection unit that calculates the distance between the point and the line and detects, as a group, the line indicating at least a part of a side branch and one or more points that are within a predetermined distance from the line, and a point addition unit that adds points that were not detected in any group by the group detection unit to one of the groups.

[0104] According to the detection device 2 configured as described above, the nodes and tips present on one side branch of a plant can be detected as one side branch group.

[0105] Furthermore, as described above, the generating device generates an image in which the nodes and tips of a side branch of a plant are displayed as one side branch group, and the generating device adopts configurations similar to the respective configurations of the detection device 2, as well as an image adding unit that adds straight lines indicating the side branches and points indicating the nodes and tips of the side branches to the image of the plant.

[0106] According to the generating device configured as above, it is possible to generate an image in which the nodes and tips on one side branch of a plant are displayed as one side branch group.

[0107] (Flow of point detection method SA) The flow of the point detection method SA executed by the point detection unit 201 will be described with reference to Fig. 9. The point detection method SA is a point detection method for detecting points indicating nodes and tips present on side branches of a plant from an image of the plant. Fig. 9 is a flow diagram showing the flow of the point detection method SA according to this embodiment.

[0108] (Step SA1) In step SA1, the point detection unit 201 equalizes the luminance value of the plant image.

[0109] (Step SA2) In step SA2, the point detection unit 201 creates learning data.

[0110] (Step SA3) In step SA3, the point detection unit 201 extracts plant organ information using AI.

[0111] (Step SA4) In step SA4, the point detection unit 201 determines whether the accuracy rate exceeds a predetermined value. If it is below the predetermined value (step SA4: NO), the process proceeds to step SA5. If it is above the predetermined value (step SA4: YES), the process proceeds to step SA6.

[0112] (Step SA5) In step SA5, the point detection unit 201 increases the learning data. After the process in step SA5, the process returns to step SA2.

[0113] (Step SA6) In step SA6, the point detection unit 201 acquires the coordinates of the shoots and nodes.

[0114] (Flow of detection method S2) The flow of the detection method S2 executed by the detection device 2 will be described with reference to FIG. 10. The detection method S2 is a detection method for detecting the nodes and tips of one side branch of a plant as one side branch group. A generation method executed by the generation device according to this embodiment will also be described. The generation method is a generation method for generating an image in which the nodes and tips of one side branch of a plant are displayed as one side branch group. FIG. 10 is a flow diagram showing the flow of the detection method S2 according to this embodiment. In the detection method S2, the detection device 2 executes each process of steps SA, S1, and S21 to S24. In addition to each process executed by the detection device 2 described above, the generation device executes a process of step S25 in the generation method. Note that, as for step SA and step S1 in the flow diagram of FIG. 10, the processes are the same as those of the point detection method SA and detection method S1 described above, respectively, and therefore their description will be omitted.

[0115] (Step S21) In step S21, the group detection unit 203 calculates the distance between the points and the straight line, and detects a straight line indicating at least a part of one side branch and one or more points that exist within a predetermined distance from the straight line as one group.

[0116] For example, group detection unit 203 may detect a point and a line as one group if the distance between the point and the line and a predetermined specified range from the line satisfy the formula in the above description of group detection unit 203.

[0117] In addition, for example, when the diameter of one side branch is K and a predetermined value is X with reference to K, the group detection unit 203 calculates a rotation center of a detected point and a straight line detected as one group together with the detected point as follows: Angle θ=tan -1 (3*K / X)*180 / π Alternatively, the range may be searched for undetected points.

[0118] (Step S22) In step S22, the overlapping group selection unit 205 may, for example, select a group in which overlapping points are detected in step S21. For example, when a plurality of groups including all the same points are detected in step S21, the overlapping group selection unit 205 may keep one of the plurality of groups and delete the other groups.

[0119] (Step S23) In step S23, the overlapping point selection unit 206 may, for example, select points that are detected overlappingly in multiple groups in step S21. For example, when the same point is detected overlappingly in multiple groups in the group detection step, the overlapping point selection unit 206 may leave one of the multiple groups that contains the largest number of points and delete the same point from the other groups.

[0120] (Step S24) In step S24, point adding section 204 adds the points that were not detected in any group in step S21 to one of the groups.

[0121] For example, the point adding unit 204 may set a point that is not detected in any group as point A, a point that is closest to point A as point B, a point that is closest to point B excluding point A as point C, and a distance from point B to point C as X, and may satisfy the following conditional expressions (1) and (2). Distance from point A to a line including point B <X / 2 …(1) Distance from point A to point B <X / 2 …(2) If the above relation is satisfied, point A may be added to the group including point B. Also, for example, half the distance X / 2 from point B to point C may be calculated by the formula in the above description of point adding unit 204, with point B as node p1 and point C as node p2.

[0122] (Step S25) In step S25, the image adding unit 207 adds, to the image of the plant, straight lines indicating the lateral branches and points indicating the nodes and tips of the lateral branches.

[0123] (Effect of detection method S2) As described above, detection method S2 employs a detection method for detecting the nodes and tips present on a side branch of a plant as a side branch group, which includes a point detection process for detecting points indicating each of the nodes and tips present on the side branch of the plant from an image of the plant, a line detection process for detecting a line indicating at least a part of the side branch of the plant from the image of the plant, a group detection process for calculating the distance between the point and the line and detecting, as a group, the line indicating at least a part of a side branch and one or more points present within a predetermined distance from the line, and a point addition process for adding a point that was not detected in any group in the group detection process to one of the groups.

[0124] According to the detection method S2 configured as described above, the nodes and tips present on one side branch of a plant can be detected as one side branch group.

[0125] As described above, the generation method employs configurations similar to those in detection method S2 for generating an image in which the nodes and tips of a side branch of a plant are displayed as one side branch group, and an image addition process for adding straight lines indicating the side branch and points indicating the nodes and tips of the side branch to the image of the plant.

[0126] According to the generating method configured as described above, it is possible to generate an image in which the nodes and tips on one side branch of a plant are displayed as one side branch group.

[0127] [Software implementation example] The functions of detection devices 1 and 2 (hereinafter referred to as the "devices") can be realized by a program for causing a computer to function as the devices, and a program for causing a computer to function as each control block of the devices (particularly each part included in control units 10 and 20).

[0128] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0129] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0130] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0131] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be executed by the control device or another device (for example, an edge computer or a cloud server).

[0132] 〔summary〕 A detection method according to a first aspect of the present invention is a detection method for detecting side branches of a plant from an RGB image, and includes a first generation step of generating a first grayscale image from the RGB image of the plant; a second generation step of creating two binary images for the first grayscale image using two predetermined thresholds and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement step of converting the second grayscale image into an edge image in which the contours of areas including the side branches are enhanced; and a straight line acquisition step of acquiring an equation for a straight line indicating at least a part of the side branches from the edge image.

[0133] With the above configuration, two binary images can be created from an image of a plant using two threshold values, and lateral branches of the plant can be detected via the two binary images.

[0134] In the detection method according to aspect 2 of the present invention, in aspect 1, the first generation step is a step of generating the first grayscale image using a transformation formula that converts the RGB value of one pixel in the side branch region in the RGB image into a single positive integer.

[0135] With the above configuration, plants present in the air can be detected from an image with high accuracy.

[0136] A detection method according to aspect 3 of the present invention is a conversion formula in aspect 2, which adds a predetermined constant term to a value obtained by adding, subtracting, and multiplying real values ​​indicating the RGB values ​​of the one pixel, and processes any fractional parts after the decimal point to convert the result into the one positive integer.

[0137] With the above configuration, the image of the side branch or the like after conversion can be in an image format in which the RGB values ​​are integer values ​​in the range of 0-255.

[0138] In the detection method according to aspect 4 of the present invention, in any one of aspects 1 to 3, the second generation step includes a mask creation step of creating a mask image from the two binarized images, and a mask processing step of performing mask processing on the first grayscale image using the mask image to generate the second grayscale image.

[0139] With the above configuration, it is possible to extract the area of ​​the lateral branches of a plant from a grayscale image of the plant.

[0140] A fifth aspect of the present invention relates to a detection method in accordance with the fourth aspect, wherein the mask creating step is a step of creating the mask image by performing an exclusive OR (XOR) operation on the two binarized images.

[0141] With the above configuration, it is possible to accurately extract the area of ​​the lateral branches of a plant from a grayscale image of the plant.

[0142] In the detection method according to aspect 6 of the present invention, in any one of aspects 1 to 5, the enhancement step includes a filtering step of performing Laplacian filter processing to enhance the contour of an area including side branches, and a brightness extraction step of extracting an area having a brightness value equal to or greater than a predetermined value from the image after the Laplacian filter processing.

[0143] With the above configuration, the contour line of the side branch can be approximated to a straight line with high accuracy.

[0144] In the detection method according to aspect 7 of the present invention, in any one of aspects 1 to 6, the straight line acquisition step includes a straight line extraction step of extracting a straight line indicating at least a part of a side branch from the edge image using a straight line extraction technique, and an equation acquisition step of acquiring an equation of the extracted straight line.

[0145] With the above configuration, it is possible to obtain an equation for a straight line that represents at least a portion of a lateral branch from an edge image of a plant.

[0146] A detection method according to an eighth aspect of the present invention is the same as that of the seventh aspect, wherein the line extraction step uses a Line Segment Detector (LSD), Edge Drawing Lines (ED) or Markov Chain Marginal Line Segment Detector (MCMLSD) as the line extraction method.

[0147] With the above configuration, it is possible to accurately obtain an equation of a straight line representing at least a portion of a lateral branch from an edge image of a plant.

[0148] A detection device according to a ninth aspect of the present invention is a detection device for detecting side branches of a plant from an RGB image, and includes a first generation unit that generates a first grayscale image from the RGB image of the plant, a second generation unit that creates two binary images for the first grayscale image using two predetermined thresholds and generates a second grayscale image from the two binary images and the first grayscale image, an enhancement unit that converts the second grayscale image into an edge image that enhances the contours of areas including the side branches, and a straight line acquisition unit that acquires an equation for a straight line that indicates at least a portion of the side branches from the edge image.

[0149] The above configuration provides the same effects as in the first aspect.

[0150] A program according to a tenth aspect of the present invention causes a computer to execute a first generation process of generating a first grayscale image from an RGB image of a plant; a second generation process of creating two binary images for the first grayscale image using two predetermined thresholds and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement process of converting the second grayscale image into an edge image that enhances the contours of areas including side branches; and a straight line acquisition process of acquiring, from the edge image, an equation for a straight line that indicates at least a portion of the side branches.

[0151] The above configuration provides the same effects as in the first aspect.

[0152] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0153] 1, 2 Detection device 10, 20 Control section 11 Input section 12 Storage section 13 Output section 101 First Generation Unit 102 Second Generation Section 103 Emphasis part 104 Straight line acquisition part 201 Point detection unit 202 Straight line detector 203 Group detection section 204 points additional section 205 Duplicate Group Selection Department 206 Duplicate point selection section 207 Image Addition Section 1021 Mask Making Department 1022 Mask processing section 1031 Filter section 1032 Luminance Extraction Unit 1041 Straight line extraction part 1042 Expression acquisition part

Claims

1. A method for detecting lateral branches of a plant from an RGB image, comprising the steps of: a first generation step of generating a first grayscale image from an RGB image of the plant; a second generating step of generating two binary images for the first grayscale image by using two predetermined thresholds, and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement step of converting the second grayscale image into an edge image in which the contour of an area including side branches is enhanced; a line acquisition step of acquiring an equation of a line representing at least a part of a side branch from the edge image; A detection method comprising:

2. The first generation step includes: generating the first grayscale image using a conversion equation in which the RGB value of one pixel in the region of the side branch in the RGB image is converted to a positive integer; The detection method according to claim 1 .

3. The conversion formula is: adding a predetermined constant term to a value obtained by adding, subtracting, or multiplying a real value indicating the RGB value of the one pixel; A conversion formula for converting a fraction after the decimal point into the single positive integer. The detection method according to claim 2.

4. The second generation step includes: a mask creation step of creating a mask image from the two binarized images; a mask processing step of performing a mask processing on the first grayscale image using the mask image to generate the second grayscale image; The method of claim 1 , comprising:

5. The mask creating step includes: creating the mask image by performing an exclusive OR (XOR) operation on the two binarized images; The detection method according to claim 4.

6. The highlighting step includes: A filtering step of performing Laplacian filtering to enhance the contour of the region including the side branches; a brightness extraction step of extracting an area having a brightness value equal to or greater than a predetermined brightness value from the image that has been subjected to the Laplacian filter processing; The method of claim 1 , comprising:

7. The straight line acquisition step includes: a line extraction step of extracting a line indicating at least a part of a side branch from the edge image using a line extraction technique; An equation acquisition step of acquiring an equation of the extracted straight line; The method of claim 1 , comprising:

8. 8. The detection method according to claim 7, wherein the line extraction step uses a line segment detector (LSD), an edge drawing (ED) lines, or a Markov chain marginal line segment detector (MCMLSD) as the line extraction technique.

9. A detection device for detecting lateral branches of a plant from an RGB image, comprising: a first generating unit that generates a first grayscale image from an RGB image of the plant; a second generating unit that generates two binary images for the first grayscale image by using two predetermined thresholds, and generates a second grayscale image from the two binary images and the first grayscale image; an enhancement unit that converts the second grayscale image into an edge image in which an outline of an area including a side branch is enhanced; a straight line acquisition unit that acquires an equation of a straight line representing at least a part of a lateral branch from the edge image; A detection device comprising:

10. On the computer, a first generation process for generating a first grayscale image from an RGB image of the plant; a second generation process for generating two binary images for the first grayscale image using two predetermined thresholds, and generating a second grayscale image from the two binary images and the first grayscale image; an enhancement process for converting the second grayscale image into an edge image in which the contour of an area including the side branches is enhanced; A line acquisition process for acquiring an equation of a line representing at least a part of a side branch from the edge image; A program for executing the above.

Images