Distance-adaptive crop three-dimensional model construction method, electronic equipment and medium

Abstract

The invention discloses a distance-adaptive crop three-dimensional model construction method, electronic equipment and a medium. The distance-adaptive crop three-dimensional model construction method comprises the following steps: acquiring a plant image training set and training a target recognition model; acquiring an RGB image and a depth image of a plant to be detected through an RGB-D sensor; obtaining the minimum value of the optimal shooting distance of the RGB-D sensor, the maximum value of the optimal shooting distance and the minimum distance Z between the plant to be detected and the RGB-D sensor; inputting an RGB image of a to-be-detected plant into the target recognition model, and when a label output by the target recognition model is complete, the to-be-detected plant is located in the middle area of the RGB image, and the minimum distance Z between the to-be-detected plant and the RGB-D sensor is within the range of the optimal shooting distance minimum value and the optimal shooting distance maximum value of the RGB-D sensor, determining that the to-be-detected plant is in the target recognition model. Image acquisition of the detected plant is completed; and aligning the RGB image and the depth image of the to-be-detected plant to obtain complete point cloud data of the to-be-detected plant.

Description

Technical Field

[0001] The present invention relates to the technical fields of target recognition and three-dimensional reconstruction, and in particular to a method, electronic equipment and medium for constructing a three-dimensional crop model with an adaptive distance. Background Art

[0002] The 3D point cloud model can capture the spatial structure and morphological characteristics of crops. A complete and accurate 3D point cloud model can provide a large amount of high-dimensional crop phenotypic data, such as plant height, stem thickness, internode spacing, etc., which is of great reference value for researchers to carry out crop variety improvement, genetic research and large-scale phenotypic analysis. In the greenhouse facility environment, farmers can provide precise crop management based on the real-time growth status of the crop 3D model, thereby improving the yield and quality of the crop.

[0003] However, due to the fast growth rate of crops in greenhouse facilities, the small gaps between crops and the limited space for equipment movement. Existing 3D reconstruction of greenhouse crops mostly involves manually holding sensors for obtaining 3D information and manually adjusting the straight-line distance and height between the sensor and the crop to obtain 3D models of crops with different shape characteristics, which is both time-consuming and laborious. In particular, the labor cost required in a greenhouse environment with large scale and high density is extremely high, and the accuracy of the constructed model is also greatly reduced.

[0004] Therefore, how to intelligently and automatically adjust the distance and height between the sensor and the crop to obtain a complete three-dimensional model of the crop is an urgent problem to be solved. Summary of the invention

[0005] In view of the deficiencies in the prior art, the embodiments of the present invention provide a method, electronic device, and medium for constructing a three-dimensional crop model with adaptive distance.

[0006] In a first aspect, an embodiment of the present invention provides a method for constructing a three-dimensional crop model with adaptive distance, the method comprising:

[0007] Collect plant images, and annotate the plant images according to "complete" or "missing" to obtain a plant image training set; use the plant image training set to train a target recognition model;

[0008] Collect RGB images and depth images of the plants to be detected through the RGB-D sensor;

[0009] Obtain the minimum and maximum optimal shooting distances of the RGB-D sensor; synthesize a point cloud model based on the RGB image and depth image of the plant to be detected, and obtain the minimum distance Z between the plant to be detected and the RGB-D sensor according to the Z-axis average value of several point clouds closest to the RGB-D sensor;

[0010] The RGB image of the plant to be detected is input into the target recognition model. When the label output by the target recognition model is "complete", the plant to be detected is in the middle area of ​​the RGB image, and the minimum distance Z between the plant to be detected and the RGB-D sensor is within the range of the minimum optimal shooting distance and the maximum optimal shooting distance of the RGB-D sensor, the image acquisition of the plant to be detected is completed;

[0011] Align the RGB image and depth image of the plant to be detected to obtain the complete point cloud data of the plant to be detected.

[0012] In a second aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned adaptive distance crop three-dimensional model construction method.

[0013] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for constructing a three-dimensional crop model with adaptive distance.

[0014] In a fourth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for constructing a three-dimensional crop model with adaptive distance.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention provides a method for constructing a three-dimensional model of crops with an adaptive distance, which improves the quality of the acquired RGB image and depth image by detecting the integrity of the plant to be detected in real time and adjusting the distance between the RGB-D sensor and the plant to be detected according to the result. The present invention can maintain the optimal shooting distance in different scenes by comparing the current distance with the set optimal shooting distance and automatically adjusting the position of the camera according to the result. This adaptive adjustment mechanism ensures the stability and consistency of the shooting results and reduces the dependence on image quality. By adjusting the height of the camera, the target can be accurately adjusted to the middle area of ​​the image, which not only improves the integrity and accuracy of target capture, but also optimizes the subsequent point cloud data processing process. The method of the present invention improves the accuracy of depth information and is particularly suitable for working in scenes with complex environments or high sensor noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0018] Figure 1 is a flow chart of a method for constructing a three-dimensional crop model with adaptive distance provided by an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of the output result of the target recognition model provided by an embodiment of the present invention;

[0020] Figure 3 is a flow chart of moving the position of an RGB-D sensor provided by an embodiment of the present invention;

[0021] Figure 4 is a flow chart of acquiring point cloud data of a plant to be detected provided by an embodiment of the present invention;

[0022] Figure 5 is a schematic diagram of a three-dimensional point cloud reconstruction result of tomatoes at the seedling stage and the flowering and fruiting stage provided by an embodiment of the present invention;

[0023] Figure 6 It is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] It should be noted that, in the absence of conflict, the features in the following embodiments and implementations may be combined with each other.

[0026] like Figure 1 As shown, the present invention provides a method for constructing a crop three-dimensional model with adaptive distance, and the method comprises the following steps:

[0027] Step S1, collect data on plant images based on an RGB-D camera, use the Labelimg tool to annotate the plant images, divide the plant images into two categories: "complete" and "missing", and obtain a plant image training set; use the plant image training set to train the target recognition model.

[0028] Step S2, collecting the RGB image and depth image of the plant to be detected through the RGB-D sensor.

[0029] Step S3, obtaining the minimum value and the maximum value of the optimal shooting distance of the RGB-D sensor; synthesizing a point cloud model based on the RGB image and depth image of the plant to be detected, and obtaining the minimum distance Z between the plant to be detected and the RGB-D sensor according to the Z-axis average value of several point clouds closest to the RGB-D sensor.

[0030] It should be noted that, in this example, the minimum distance Z between the plant to be detected and the RGB-D sensor is obtained according to the Z-axis average value of at least 1000 point clouds closest to the RGB-D sensor; this example does not directly use the minimum value, which can effectively reduce the impact of sensor noise and external environmental interference on the measurement results.

[0031] Step S4, input the RGB image of the plant to be detected into the target recognition model. When the label output by the target recognition model is "complete", the plant to be detected is in the middle area of ​​the RGB image, and the minimum distance Z between the plant to be detected and the RGB-D sensor is within the range of the minimum optimal shooting distance and the maximum optimal shooting distance of the RGB-D sensor, the image acquisition of the detected plant is completed.

[0032] Step S401, input the RGB image of the plant to be detected into the target recognition model to obtain the bounding box coordinates (x_min, y_min, x_max, y_max) of the plant to be detected; wherein x_min represents the minimum value of the horizontal coordinate of the bounding box of the plant to be detected, y_min represents the minimum value of the vertical coordinate of the bounding box of the plant to be detected, x_max represents the maximum value of the horizontal coordinate of the bounding box of the plant to be detected, and y_max represents the maximum value of the vertical coordinate of the bounding box of the plant to be detected;

[0033] Calculate the center position (x_center, y_center) of the plant to be detected, x_center = (x_min + x_max) / 2, and y_center = (y_min + y_max) / 2; where x_center represents the horizontal coordinate center point of the plant to be detected, and y_center represents the vertical coordinate center point of the plant to be detected;

[0034] Step S402, dividing the RGB image into an upper left area, an upper middle area, an upper right area, a left middle area, a central area, a right middle area, a lower left area, a lower middle area, and a lower right area according to the nine-square grid;

[0035] like Figure 2As shown, when x_center ≤ 1 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the upper left area of the RGB image;

[0036] When 1 / 3image_w < x_center ≤ 2 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the middle upper area of the RGB image;

[0037] When x > 2 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the upper right area of the RGB image;

[0038] When x ≤ 1 / 3image_w and 1 / 3image_h < y ≤ 2 / 3image_h, it means that the plant to be detected is in the middle left area of the RGB image;

[0039] When 1 / 3image_w < x ≤ 2 / 3image_w and 1 / 3image_h < y ≤ 2 / 3image_h, it means that the plant to be detected is in the central area of the RGB image;

[0040] When x > 2 / 3image_w and 1 / 3image_h < y ≤ 2 / 3image_h, it means that the plant to be detected is in the middle right area of the RGB image;

[0041] When x ≤ 1 / 3image_w and y > 2 / 3image_h, it means that the plant to be detected is in the lower left area of the RGB image;

[0042] When 1 / 3image_w < x ≤ 2 / 3image_w and y > 2 / 3image_h, it means that the plant to be detected is in the middle lower area of the RGB image;

[0043] When x > 2 / 3image_w and y > 2 / 3image_h, it means that the plant to be detected is in the lower right area of the RGB image;

[0044] Among them, image_w represents the width of the RGB image, and image_h represents the height of the RGB image.

[0045] Step S403, as Figure 3 shown, when the label output by the target recognition model is "complete", and the plant to be detected is in the middle area of the RGB image, and the minimum distance Z between the plant to be detected and the RGB-D sensor is within the range of the minimum value and the maximum value of the best shooting distance of the RGB-D sensor, the image acquisition of the detected plant is completed.

[0046] When the label output by the target recognition model is "complete", and when the minimum distance Z between the plant to be detected and the RGB-D sensor is not within the optimal shooting range of the RGB-D sensor and the minimum distance Z is greater than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax , mark as unable to shoot, control RGB-D sensor to adjust Z=Z btmin , take a photo of the next plant;

[0047] When the minimum distance Z between the plant to be detected and the RGB-D sensor is not within the optimal shooting range of the RGB-D sensor and the minimum distance Z is less than Z btmin , adjust Z = Z btmin , reshoot.

[0048] When the label output by the target recognition model is "missing", and the minimum distance Z between the plant to be detected and the RGB-D sensor is greater than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax When the RGB-D sensor is adjusted, the minimum distance Z between the plant to be detected and the RGB-D sensor is equal to the minimum value of the optimal shooting distance Z of the RGB-D sensor. btmin , take a photo of the next plant;

[0049] When the label output by the target recognition model is "missing", and the minimum distance Z between the plant to be detected and the RGB-D sensor is less than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax When the horizontal movement step is set, the RGB-D sensor is moved backward according to the horizontal movement step and the shooting is performed again;

[0050] The horizontal moving step is set to ΔZ = (Z btmax -Z btmin ) / 10.

[0051] When the plant to be detected is in the upper left area, the upper middle area, or the upper right area of ​​the RGB image, the installation position of the RGB-D sensor is adjusted downward according to the height moving step length;

[0052] When the plant to be detected is in the lower left area, lower middle area, or lower right area of ​​the RGB image, the installation position of the RGB-D sensor is adjusted to move upward according to the height moving step.

[0053] The height moving step size is set to:

[0054] ΔY=(Ymax-Ymin) / 10

[0055] Among them, Ymax represents the maximum installation height of the RGB-D sensor, and Ymin represents the minimum installation height of the RGB-D sensor.

[0056] Step S5, aligning the RGB image and the depth image of the plant to be detected to obtain point cloud data of the plant to be detected.

[0057] Specifically, based on the MapDepthFrameToCameraSpace() function, the RGB image and the depth image of the plant to be detected are aligned to obtain the initial point cloud data of the plant to be detected;

[0058] like Figure 4 As shown, the initial point cloud data of the plant to be detected is filtered by straight-through filtering to remove the background point cloud using coordinate features, color filtering to remove the background point cloud using color features, clustering processing to remove the background point cloud with similar color and distance to the target plant, and outliers are removed to remove noise, so as to obtain the final complete point cloud data of the plant to be detected.

[0059] When the crop is a tomato plant, the constructed 3D model is as follows Figure 5 As shown, it includes three-dimensional point cloud models of two periods: seedling stage and flowering and fruiting stage.

[0060] Among them, because the plant point cloud is mainly green, yellow (mainly flowers and withered leaves) and red (fruit ripening period), color filtering can also be used to remove background information. The color filtering method is similar to the straight-through filtering method. The background information is filtered out by setting the range on the three channels of r, g, and b according to the color of the point cloud.

[0061] It should be noted that the present invention combines multiple filtering techniques such as straight-through filtering, color filtering, Euclidean clustering extraction, and radius outlier removal to effectively remove background information and noise. Compared with a single filtering method, this multiple filtering strategy can clean up the data more comprehensively, making the final crop three-dimensional point cloud model cleaner and more reliable. At the same time, this processing is of great significance for subsequent analysis and processing (such as size measurement and disease detection).

[0062] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method for constructing a three-dimensional crop model with adaptive distance. Figure 6 As shown, it is a hardware structure diagram of any device with data processing capability where the method for constructing a crop three-dimensional model with adaptive distance provided by an embodiment of the present invention is located. Figure 6 In addition to the processor, memory and network interface shown, any device with data processing capability in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capability, which will not be described in detail.

[0063] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the above-mentioned method for constructing a crop three-dimensional model with adaptive distance is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.

[0064] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A method for constructing a three-dimensional crop model with adaptive distance, characterized in that: The method includes: Performing image acquisition on the plants, and annotating the plant images as "complete" or "missing" to obtain a plant image training set; training an object recognition model using the plant image training set; Collecting the RGB image and depth image of the plant to be detected through an RGB-D sensor; Obtaining the minimum value and maximum value of the optimal shooting distance of the RGB-D sensor; synthesizing a point cloud model based on the RGB image and depth image of the plant to be detected, and obtaining the minimum distance Z between the plant to be detected and the RGB-D sensor according to the average value of the Z axes of several point clouds closest to the RGB-D sensor; Inputting the RGB image of the plant to be detected into the object recognition model. When the label output by the object recognition model is "complete", and the plant to be detected is in the middle area of the RGB image, and the minimum distance Z between the plant to be detected and the RGB-D sensor is within the range of the minimum value and maximum value of the optimal shooting distance of the RGB-D sensor, the image acquisition of the plant to be detected is completed; Aligning the RGB image and depth image of the plant to be detected to obtain the complete point cloud data of the plant to be detected.

2. The method for constructing a crop three-dimensional model with adaptive distance according to claim 1, characterized in that: Inputting the RGB image of the plant to be detected into the object recognition model to obtain the bounding box coordinates (x_min, y_min, x_max, y_max) of the plant to be detected; where, x_min represents the minimum value of the abscissa of the bounding box of the plant to be detected, y_min represents the minimum value of the ordinate of the bounding box of the plant to be detected, x_max represents the maximum value of the abscissa of the bounding box of the plant to be detected, and y_max represents the maximum value of the ordinate of the bounding box of the plant to be detected; Calculating the center position (x_center, y_center) of the plant to be detected, x_center = (x_min + x_max) / 2, and y_center = (y_min + y_max) / 2; where, x_center represents the abscissa center point of the plant to be detected, and y_center represents the ordinate center point of the plant to be detected; Dividing the RGB image into a top-left area, a top-middle area, a top-right area, a left-middle area, a center area, a right-middle area, a bottom-left area, a bottom-middle area, and a bottom-right area according to a nine-grid; When x_center ≤ 1 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the top-left area of the RGB image; When 1 / 3image_w < x_center ≤ 2 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the top-middle area of the RGB image; When x > 2 / 3image_w and y ≤ 1 / 3image_h, it means that the plant to be detected is in the top-right area of the RGB image; When x ≤ 1 / 3image_w and 1 / 3image_h < y ≤ 2 / 3image_h, it means that the plant to be detected is in the left-middle area of the RGB image; When 1 / 3 image_w < x ≤ 2 / 3 image_w and 1 / 3 image_h < y ≤ 2 / 3 image_h, it indicates that the plant to be detected is in the central region of the RGB image; When x > 2 / 3 image_w and 1 / 3 image_h < y ≤ 2 / 3 image_h, it indicates that the plant to be detected is in the upper right central region of the RGB image; When x ≤ 1 / 3 image_w and y > 2 / 3 image_h, it indicates that the plant to be detected is in the lower left region of the RGB image; When 1 / 3 image_w < x ≤ 2 / 3 image_w and y > 2 / 3 image_h, it indicates that the plant to be detected is in the lower central region of the RGB image; When x > 2 / 3 image_w and y > 2 / 3 image_h, it indicates that the plant to be detected is in the lower right region of the RGB image; Wherein, image_w represents the width of the RGB image, and image_h represents the height of the RGB image.

3. The method for constructing a crop three-dimensional model with adaptive distance according to claim 2, characterized in that: Set the height movement step size; When the plant to be detected is in the upper left region, upper central region, or upper right region of the RGB image, adjust the installation position of the RGB-D sensor to move downward according to the height movement step size; When the plant to be detected is in the lower left region, lower central region, or lower right region of the RGB image, adjust the installation position of the RGB-D sensor to move upward according to the height movement step size.

4. The method for constructing a three-dimensional crop model with adaptive distance according to claim 3, characterized in that: The height movement step size is set as: ΔY = (Ymax - Ymin) / 10 Wherein, Ymax represents the highest installation height of the RGB-D sensor, and Ymin represents the lowest installation height of the RGB-D sensor.

5. The method for constructing a three-dimensional crop model with adaptive distance according to claim 1, characterized in that: The RGB image of the plant to be detected is input into the target recognition model. When the label output by the target recognition model is "missing", and the minimum distance Z between the plant to be detected and the RGB-D sensor is greater than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax When the RGB-D sensor is adjusted, the minimum distance Z between the plant to be detected and the RGB-D sensor is equal to the minimum value of the optimal shooting distance Z of the RGB-D sensor. btmin , take a photo of the next plant; When the label output by the target recognition model is "missing", and the minimum distance Z between the plant to be detected and the RGB-D sensor is less than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax When the horizontal movement step is set, the RGB-D sensor is moved backward according to the horizontal movement step and the shooting is performed again; The horizontal moving step size is set to ΔZ = (Z btmax -Z btmin ) / 10.

6. The method for constructing a three-dimensional crop model with adaptive distance according to claim 1, characterized in that: The RGB image of the plant to be detected is input into the target recognition model. When the label output by the target recognition model is "complete", and when the minimum distance Z between the plant to be detected and the RGB-D sensor is not within the optimal shooting range of the RGB-D sensor and the minimum distance Z is greater than the maximum value Z of the optimal shooting distance of the RGB-D sensor btmax , mark as unable to shoot, control RGB-D sensor to adjust Z=Z btmin , take a photo of the next plant; When the minimum distance Z between the plant to be detected and the RGB-D sensor is not within the optimal shooting range of the RGB-D sensor and the minimum distance Z is less than Z btmin , adjust Z = Z btmin , reshoot.

7. The method for constructing a three-dimensional crop model with adaptive distance according to claim 1, characterized in that: The process of aligning the RGB image and the depth image of the plant to be detected to obtain the point cloud data of the plant to be detected includes: Align the RGB image and the depth image of the plant to be detected to obtain the initial point cloud data of the plant to be detected; Perform直通滤波 (it should be "pass-through filtering" in English), color filtering, clustering processing, and remove outliers on the initial point cloud data of the plant to be detected to obtain the final complete point cloud data of the plant to be detected.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein, the memory is used to store program data, and the processor is used to execute the program data to implement the adaptive distance crop three-dimensional model construction method described in any one of claims 1-7 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, it implements the adaptive distance crop three-dimensional model construction method described in any one of claims 1-7.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by the processor, it implements the adaptive distance crop three-dimensional model construction method described in any one of claims 1-7.

Images