A method and device for identifying and locating ipomoea nil flowers based on non-parallel camera placement

Abstract

This invention provides a method for identifying and locating the maturity of okra flowers based on non-parallel-placed cameras, comprising the following steps: two non-parallel-placed cameras capture images of the plant from different directions; color blocks in both cameras that reach a set threshold are marked as flowers; each image marked as a flower is binarized and edge extracted; the variance of the distance from each pixel on the edge of the closed shape to its geometric center is calculated; the flower shape is determined based on the variance; and the location of mature flowers is determined. This invention also provides a device for detecting and locating the maturity of okra flowers based on non-parallel-placed cameras. The method and device of this invention utilize non-parallel-placed cameras to acquire images of the target crop, calculate its three-dimensional coordinates using the two-dimensional spatial coordinates of the flower obtained from each of the two cameras, and simultaneously use the flower's morphology and color information as a basis for determining whether further processing is required.

Description

Technical Field

[0001] This invention provides a low-cost target recognition method and device for okra flowers based on orthogonally placed cameras, belonging to the agricultural field. Background Technology

[0002] Yellow okra [Abelmoschus manihot (L.) Medic.] is an annual or perennial erect herb belonging to the Malvaceae family and the Abelmoschus genus. Due to indiscriminate harvesting and ecological degradation, wild yellow okra resources in my country were on the verge of extinction in the 1980s. Currently, artificial cultivation of yellow okra is mainly distributed in Jiangsu, Anhui, Shandong, Hainan, Guangxi and other regions. The entire yellow okra plant is valuable. Its leaves, young pods, and flowers are rich in various vitamins and nutrients such as iron, calcium, and crude protein, and have effects such as preventing arteriosclerosis, hair loss, prolonging puberty, and enhancing internal organ function. The seeds of yellow okra have an oil yield of 25%–32%, and the extracted oil is clear and light yellow with strong antioxidant properties, making it a new type of edible oil crop. Yellow okra flowers, as a medicinal material, have been included in the Chinese Pharmacopoeia.

[0003] With the advancement of large-scale agricultural planting and industrialization, traditional manual harvesting techniques are far from meeting the needs of modern agriculture, and robotics has begun to be applied to the agricultural field. However, currently, harvesting robots are mainly used for crops with clearly defined targets, such as apples and oranges. To ensure that these crops are not damaged during transportation, they are often harvested before they are fully ripe. For example, patent CN 103279762 B provides a method for determining common fruit growth morphologies in natural environments, and patent CN 101828469 B provides a binocular vision information acquisition device for cucumber harvesting robots. This device uses two parallel cameras for binocular recognition. After obtaining two sets of images with similar content but slight differences in shooting angle, the images are processed to obtain target position information, including depth information (i.e., the distance of the target from the camera). When using parallel cameras for binocular recognition, the depth information of the target can only be obtained by transforming the coordinates after processing the differences in the pixels of the obtained images. To achieve high positioning accuracy, it is necessary to increase the resolution of the camera images and perform multiple positioning operations. This approach requires high computer processing power and is susceptible to interference. The problem is the increased hardware costs, which hinders large-scale, low-cost deployment.

[0004] For flowers, especially okra flowers used medicinally, maturity affects the content of their active ingredient, flavonoids. Therefore, determining maturity is crucial when using robots to harvest okra flowers. Patent CN111723863A discloses a method, apparatus, computer equipment, and storage medium for identifying and acquiring the location of fruit tree flowers. This method inputs a local image of a flower cluster into a flower detection model to obtain the flower type and corresponding flower location of all flowers in the local image of the flower cluster. The flower detection model is trained based on local sample images of flower clusters and the flower location and type labels within those images. This method is used to automatically identify the type and location of flowers in fruit trees to guide subsequent flower thinning operations by robots. It involves artificial intelligence, requires the collection of a large number of images for model training, and has high hardware requirements. While this AI-based method can only solve flower identification, it is difficult to handle spatial positioning. Furthermore, training the model and performing the identification process requires high hardware specifications and a large amount of training data must be prepared beforehand.

[0005] In addition, existing technologies also employ a single camera plus a distance sensor, which is a very early concept. The distance sensor is easily interfered with and loses distance information, thus failing to solve the practical problem. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a low-cost target recognition device and method based on a non-parallel-placed camera. This device and method enable the construction of a hardware solution at the lowest possible cost while achieving high-precision positioning of the detected target, such as the flower of the okra.

[0007] This invention first provides a method for identifying and locating the maturity of okra flowers based on a non-parallel-placed camera, comprising the following steps:

[0008] Step S10: Two non-parallel cameras take pictures of the plant from different directions, extract all pixels with yellow color image peak features in the image, form color blocks to be judged, and mark the color blocks in both cameras that reach the set threshold as flowers.

[0009] Step S20: After binarizing each image marked as a flower, edge extraction is performed to obtain closed edges. The area of ​​the closed shape enclosed by the edges is calculated. Closed shapes with suitable areas are selected and the coordinates of their respective geometric center points are calculated. Then, the variance of the distance from each pixel point of the closed shape edge to its geometric center point is calculated.

[0010] S30. Determine the flower shape based on the variance. If the variance meets the set value, mark it as a mature flower and proceed to step S40. If the variance does not meet the set value, adjust the angle of the two non-parallel cameras to take pictures of the plant from other different directions. Repeat step S10 and subsequent operations until the camera rotates one full circle. If the flower shape still does not meet the standard after rotating one full circle, abandon the subsequent operations on the flower.

[0011] S40: The position of the mature flower is determined by using the XY plane position and XZ plane position provided by the two cameras.

[0012] Preferably, in step S10, the non-parallel placement refers to the two cameras having an angle greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal placement. Preferably, one of the two cameras is located above the okra plant being detected.

[0013] Preferably, in step S10, the method for determining whether the yellow color reaches the threshold is as follows: extract the information of the red channel and green channel of each pixel of the color camera, and determine that the pixel is yellow when the output values ​​of the two channels are simultaneously greater than their respective set thresholds.

[0014] Preferably, in step S20, the edge extraction method is as follows: the marked pixels in step S10 are set to black, and the unmarked pixels are set to white. When there is a white pixel among the eight pixels around a black pixel, or when the black pixel is at the edge of the image, the black pixel is marked as an edge. After performing the above operation on each pixel of the image, a closed shape formed by several closed edges will be obtained.

[0015] In step S20, the closed shapes with suitable areas are those whose areas are greater than or equal to a preset threshold after eliminating closed shapes with too small areas and closed shapes located at the edge of the image.

[0016] In step S30, when determining the flower shape based on the variance, if the variance is small, it means the flower is close to round, and the flower shape seen from this angle is acceptable for picking; if the variance is large, it means the shape does not meet the standard, and there may be the following situations: a. The flower is not fully open and grows in a strip shape, which does not meet the picking standard and can be abandoned; b. The shooting angle is incorrect, and the flower is trumpet-shaped, which does not meet the picking angle, and the camera can be rotated for further shooting processing; c. The working environment is complex, with several flowers growing together, making it inconvenient to perform the positioning and picking operation.

[0017] In step S40, the XY plane refers to a plane parallel to the ground, and the Z axis refers to an axis orthogonal to the XY plane. The mature flower is located by resolving this position.

[0018] Another aspect of the present invention provides a method for recognizing and locating the color and shape of a target object based on a non-parallel-placed camera, comprising the following steps:

[0019] Step S1: Two cameras, which are not placed in parallel, take pictures of the object to be detected from different directions. Extract the pixel points with target peak characteristics in the color image of the picture and form a color block to be judged. Mark the color block in the two cameras that reaches the set threshold as the initial target.

[0020] Step S2: After binarizing the image of each marked preliminary target, edge extraction is performed to obtain closed edges. The area of ​​the closed shape enclosed by the edges is calculated. Closed shapes with suitable areas are selected and the coordinates of their respective geometric center points are calculated. Then, the variance of the distance from each pixel point of the closed shape edge to its geometric center point is calculated.

[0021] Step S3: Determine whether the shape of the preliminary target meets the standard based on the variance. If the variance meets the set value, mark it as a target object and proceed to step S40. If the variance does not meet the set value, adjust the angle of the two non-parallel cameras to take pictures of the detection object from other different directions. Repeat step S10 and subsequent operations until the camera rotates one full circle. If it still fails to be successfully identified as a target object after rotating one full circle, abandon the subsequent operation on the detection object.

[0022] Step S4: The position of the target object is determined by using the XY plane position and XZ plane position provided by the two cameras.

[0023] Preferably, in step S1, the non-parallel placement refers to the two cameras having an angle greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal placement. One of the two cameras is preferably located above the object being detected.

[0024] Preferably, in step S1, the method for determining that the selected color has reached the threshold is as follows: extract information of two specified color channels for each pixel of the color camera, and when the output values ​​of the two channels are simultaneously greater than their respective set thresholds, the pixel is determined to be the selected color.

[0025] Preferably, in step S1, the detection object and the target object have different colors, especially contrasting colors, such as red and green, yellow and green, white and other colors, etc. Preferably, the target object is located outside the detection object. More preferably, the detection object is conical or similar to a cone shape with a pointed top and a wider bottom. Preferably, the detection object is a plant, more preferably a Malvaceae plant, especially a yellow hibiscus plant. The target object is more preferably the flower or fruit of a Malvaceae plant, more preferably the flower of a yellow hibiscus.

[0026] Preferably, in step S2, the edge extraction method is as follows: the marked pixels in step S1 are set to black, and the unmarked pixels are set to white. When there is a white pixel among the eight pixels around a black pixel, or when the black pixel is at the edge of the image, the black pixel is marked as an edge. After performing the above operation on each pixel of the image, a closed shape formed by several closed edges will be obtained.

[0027] In step S2, the closed shapes with suitable areas are those whose areas are greater than or equal to a preset threshold after eliminating closed shapes with too small areas and closed shapes located at the edge of the image.

[0028] In step S3, the XY plane refers to a plane parallel to the ground, and the Z axis refers to an axis orthogonal to the XY plane. The location of the target object and the camera position is achieved by resolving this position.

[0029] In another aspect, the present invention provides a target object recognition and positioning device based on non-parallel-placed cameras, including a frame (1) fixed on a mobile vehicle and a rotating platform (2); two non-parallel-placed cameras are fixed on the rotating platform (2); the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can rotate on the frame (1).

[0030] Preferably, in the target object recognition and positioning device, the non-parallel placement means that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal placement.

[0031] Preferably, in the target object identification and positioning device, the rotating platform (2) is a gantry-type bracket or an inverted L-shape. When the rotating platform (2) is an inverted L-shape, a Y-axis camera (3) and a Z-axis camera (4) are fixed on the two sides of the L-shaped rotating platform (2), respectively; a motor assembly (5) with a position encoder is fixed to the inner side of the top edge of the frame (1), and the connection between the rotating shaft of the motor assembly (5) and the outer side of the top edge of the inverted L-shaped rotating platform (2) is used to drive the rotating platform (2) to rotate.

[0032] The present invention also provides a maturation and positioning detection device for okra flowers based on non-parallel cameras, including a frame (1) fixed on a mobile carrier and a rotating platform (2); two non-parallel cameras are fixed on the rotating platform (2); the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can rotate on the frame (1).

[0033] Preferably, in the okra flower maturity and positioning detection device, the non-parallel placement means that the angle between the two cameras is greater than 15 degrees, more preferably greater than 45 degrees, and even more preferably orthogonal placement.

[0034] Preferably, in the okra flower maturity and positioning detection device, the rotating platform (2) is a gantry frame or an inverted L-shape. When the rotating platform (2) is an inverted L-shape, a Y-axis camera (3) and a Z-axis camera (4) are fixed on the two sides of the L-shaped rotating platform (2), respectively; a motor assembly (5) with a position encoder is fixed to the inner side of the top edge of the frame (1), and the connection between the rotating shaft of the motor assembly (5) and the outer side of the top edge of the inverted L-shaped rotating platform (2) is used to drive the rotating platform (2) to rotate.

[0035] The device provided by this invention uses non-parallel cameras to acquire images of the target object. By obtaining two spatial coordinates of the target object in two dimensions from each of the two cameras, its three-dimensional coordinates are calculated. At the same time, the shape and color information of the target itself are used as the basis for whether it needs to be processed, such as picking flowers.

[0036] The method provided by this invention can obtain the result by performing color image pixel threshold judgment, edge extraction after binarization, and variance calculation on the target image. The amount of computation required is extremely low. It can even combine a low-resolution color camera with a microcontroller to achieve the recognition of target objects such as hibiscus flowers, which greatly reduces the difficulty of solution development and the cost of actual application.

[0037] The method provided by this invention can solve the problem of maturity detection and positioning of okra flowers during automated harvesting. By screening the flowers on the plant, the spatial position and harvesting angle of the okra flowers that can be harvested are determined, and the method is provided to the harvesting robotic arm to guide it to perform automated harvesting. This invention relates to the fields of agricultural robotics and image recognition, and in particular to a method for determining the maturity and spatial positioning of flowers of fruits, vegetables or economically valuable crops before harvesting through image recognition. However, the application scenarios of the results are not limited to this. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of the low-cost target recognition device based on orthogonally placed cameras according to the present invention. Detailed Implementation

[0039] The present invention will be further described below. In the present invention, the XY plane refers to a plane parallel to the ground, and the Z axis refers to an axis orthogonal to the XY plane.

[0040] Example 1

[0041] Low-cost target recognition device based on orthogonally placed cameras, see [link / reference] Figure 1 The system includes a frame (1) fixed on a mobile vehicle and a rotating platform (2); two cameras that are not placed in parallel are fixed on the rotating platform (2); the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can rotate on the frame (1).

[0042] The following is a more specific example to illustrate the low-cost target recognition device based on orthogonally placed cameras of the present invention:

[0043] See Figure 1 A frame (1) fixed to a mobile vehicle (e.g., a tracked vehicle) is connected to an L-shaped rotating platform (2) via a motor assembly (5), allowing the rotating platform (2) to rotate around the axis of the motor assembly (5). A Z-axis camera (4) is fixed to the inner side of the top edge of the rotating platform 2, with its image frame parallel to the ground, providing image information in the XY plane direction. A Y-axis camera (3) is fixed to the inner side of the rotating platform (2), with its image frame perpendicular to the ground, providing image information in the XZ plane direction.

[0044] Since the flowers of the yellow hollyhock plant, which are valuable for harvesting, are yellow, there is a significant color difference between them and the green of the plant itself. Moreover, they are distributed on the outer part of the plant and are not easily obscured by the branches and leaves of the plant. This provides an advantageous condition for the implementation of the present invention.

[0045] After the harvesting robot stops next to the plant according to the designed route, the motor assembly (2) fixed on the frame (1) is almost directly above the plant. At this time, the Z-axis camera (4) takes pictures from the top and the Y-axis camera (3) takes pictures from the side, extracting all the pixels with yellow RGB image peak characteristics in the picture to form a color block to be judged.

[0046] Then the motor assembly (5) drives the rotating platform (2) which is fixed to the Z-axis camera (4) and the Y-axis camera (3) to rotate, so that the Y-axis camera (3) can collect images of the flowers on the okra plant from multiple angles.

[0047] During the rotation, color blocks that reach a certain color threshold are marked as flowers. Simultaneously, the image of each marked flower is binarized, and edge extraction is performed to determine the coordinates of the geometric center point of the edge. Then, the variance of the distance (which can be understood as the radius) from each point on the edge to the geometric center point is calculated. If the flower is more circular, the variance is smaller, meaning the flower shape seen from this angle is acceptable for picking. If the variance is larger, it means:

[0048] a. The flowers are not fully open and are elongated, not meeting the picking standards;

[0049] b. If the shooting angle is incorrect at this time, forming a trumpet shape, and the picking angle is not reached, you need to wait for the rotating platform to rotate to the appropriate angle for recognition before the robotic arm fixed on the rotating platform can pick the fruit.

[0050] Through the above steps, the three XYZ coordinates of the flowers determined to be pluckable at the current position of the rotating platform can be solved by the XY plane position and XZ plane position provided by the two cameras. At this time, the rotating platform (2) temporarily stops rotating, and the robotic arm fixed on it picks the flowers according to the XYZ coordinates provided above.

[0051] Example 2

[0052] The same as in Example 1, except that: the rotating platform (2) is a gantry-type support rotating platform, and the top and bottom of the rotating platform (2) are respectively fixed with a Y-axis camera (3) and a Z-axis camera (4) on the inner wall of one side; the motor assembly (5) with a position encoder is fixed to the inner side of the top edge of the frame (1), and the rotating shaft of the motor assembly (5) is connected to the outer side of the top edge of the rotating platform (2) to drive the rotating platform (2) to rotate.

[0053] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for identifying and locating the maturity of okra flowers based on a non-parallel-placed camera, characterized in that: Includes the following steps: S10. Two non-parallel cameras take pictures of the plant from different directions, extract all pixels with yellow color image peak features in the image, form color blocks to be judged, and mark the color blocks in both cameras that reach the set threshold as flowers. S20. After binarizing each image marked as a flower, perform edge extraction to obtain closed edges. Calculate the area of ​​the closed shape enclosed by the edges. Select closed shapes with suitable areas and calculate the coordinates of their respective geometric center points. Then calculate the variance of the distance from each pixel point of the closed shape edge to its geometric center point. S30. Determine the flower shape based on the variance. If the variance meets the set value, mark it as a mature flower and proceed to step S40. If the variance does not meet the set value, adjust the angle of the two non-parallel cameras to take pictures of the plant from other different directions. Repeat step S10 and subsequent operations until the camera rotates one full circle. S40. The position of the mature flower is determined by using the XY plane position and XZ plane position provided by the two cameras; The appropriate closed shape is a closed shape whose area is greater than or equal to a preset threshold after removing closed shapes with too small an area and closed shapes located at the edge of the image.

2. The method according to claim 1, characterized in that: In step S10, the method for determining whether the yellow color reaches the threshold is as follows: extract the information of the red channel and green channel of each pixel of the color camera, and determine that the pixel is yellow when the output values ​​of the two channels are simultaneously greater than their respective set thresholds.

3. The method according to claim 1, characterized in that: In step S20, the edge extraction method is as follows: the marked pixels in step S10 are set to black, and the unmarked pixels are set to white. When there is a white pixel among the eight pixels around a black pixel, or when the black pixel is at the edge of the image, the black pixel is marked as an edge. After performing the above operation on each pixel of the image, a closed shape enclosed by several closed edges will be obtained.

4. The method according to claim 1, characterized in that: One of the two cameras is located above the okra plant being monitored.

5. A method for recognizing and locating the color and shape of a target object based on a non-parallel-placed camera, characterized in that: Includes the following steps: S1. Two non-parallel cameras capture images of the object to be detected from different directions. Pixels with target peak characteristics in the color images are extracted and used to form color blocks to be judged. Color blocks in the two cameras whose colors reach the set threshold are marked as preliminary targets. S2. After binarizing the image of each marked preliminary target, perform edge extraction to obtain closed edges. Calculate the area of ​​the closed shape enclosed by the edges, select a closed shape with a suitable area and calculate the coordinates of its geometric center point. Then calculate the variance of the distance from each pixel point of the closed shape edge to its geometric center point. S3. Determine whether the shape of the preliminary target meets the standard based on the variance. If the variance meets the set value, mark it as the target object and proceed to step S4. If the variance does not meet the set value, adjust the angle of the two non-parallel cameras to take pictures of the object from other different directions. Repeat step S1 and subsequent operations until the camera rotates one full circle. S4. The position of the target object is determined by combining the XY plane position and the XZ plane position provided by the two cameras; The appropriate closed shape is a closed shape whose area is greater than or equal to a preset threshold after removing closed shapes with too small an area and closed shapes located at the edge of the image.

6. The method according to claim 5, characterized in that: The angle between two non-parallel cameras is greater than 15 degrees.

7. The method according to claim 5, characterized in that: The two cameras, which are not placed parallel to each other, are placed orthogonally.

8. The method according to claim 5, characterized in that: The angle between two cameras that are not placed parallel is greater than 45 degrees.

9. The method according to claim 5, characterized in that: The object to be detected has a different color from the target object. The target object is located outside the object to be detected. The object to be detected is conical or generally conical.

10. The method according to claim 6, characterized in that: The sample tested was a plant.

11. The method according to claim 6, characterized in that: The sample tested was a plant from the Malvaceae family.

12. The method according to claim 6, characterized in that: The sample tested was a plant of Okra (Hippophae rhamnoides).

13. The method according to claim 6, characterized in that: The target is the flower or fruit of a plant in the Malvaceae family.

14. The method according to claim 6, characterized in that: The target object is the flower of the yellow hollyhock.

15. A target object recognition and positioning device based on a non-parallel-placed camera, which uses the method described in claim 5 to recognize and locate the color and shape of the target object, characterized in that: Includes a frame (1) fixed on a mobile vehicle and a rotating platform (2); two cameras that are not placed in parallel are fixed on the rotating platform (2); the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can rotate on the frame (1).

16. The device according to claim 15, characterized in that: The rotating platform (2) is an inverted L-shape. A Y-axis camera (3) and a Z-axis camera (4) are fixed on the two sides of the L-shaped rotating platform (2). A motor assembly (5) with a position encoder is fixed to the inner side of the top edge of the frame (1). The shaft of the motor assembly (5) is connected to the outer side of the top edge of the inverted L-shaped rotating platform (2) to drive the rotating platform (2) to rotate.

17. A device for detecting the maturity and location of okra flowers based on a non-parallel-placed camera, wherein the maturity of okra flowers is identified and located using the method described in any one of claims 1 to 4, characterized in that: Includes a frame (1) fixed on a mobile vehicle and a rotating platform (2); two cameras that are not placed in parallel are fixed on the rotating platform (2); the frame (1) and the rotating platform (2) are connected by a motor assembly (5) with a position encoder, and the rotating platform (2) can rotate on the frame (1).

18. The device according to claim 17, characterized in that: The rotating platform (2) is an inverted L-shape; a Y-axis camera (3) and a Z-axis camera (4) are fixed on the two sides of the L-shaped rotating platform (2); a motor assembly (5) with a position encoder is fixed to the inner side of the top edge of the frame (1), and the shaft of the motor assembly (5) is connected to the outer side of the top edge of the inverted L-shaped rotating platform (2) to drive the rotating platform (2) to rotate.

19. The device according to claim 18, characterized in that: The rotating platform (2) is a gantry-type support or an inverted L-shape.

Images