High-throughput potted plant 3D reconstruction method and system based on three-dimensional calibration

Abstract

The invention discloses a three-dimensional calibration-based high-throughput potted plant 3D reconstruction method and system, and the method comprises the following steps: making a three-dimensional calibration box, placing potted plants in the three-dimensional calibration box, and carrying out the image collection of the potted plants through a camera, and obtaining an image sequence of each plant rotating a circle; obtaining an internal reference matrix of the camera, and performing calibration point identification on the image sequence to obtain an external reference matrix of each image; carrying out segmentation processing and color rendering on a plant region in the image sequence to obtain a plant binary image; based on the plant binary image, the camera internal reference matrix and the external reference matrix of each image, performing 3D reconstruction by adopting a voxel engraving method to obtain a plant model; and performing back projection coloring on the plant model to finally obtain a colored plant model. According to the method, high-flux full-automatic three-dimensional reconstruction of the whole growth period of the potted plant is realized, and the reconstructed point cloud still keeps the original plant size, so that the subsequent phenotype extraction and other work can be conveniently carried out.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural automation, and in particular relates to a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. Background Art

[0002] In the past decade, various new technologies have attracted widespread attention in the agricultural field due to the rapid development of new sensors and automation technologies. Two-dimensional imaging technology is widely used for structural trait estimation, growth analysis, and yield estimation of crops at different growth stages. Although many hardware systems and corresponding image processing and analysis algorithms have been developed, there are still some unavoidable defects, such as the change in the distance between the camera viewpoint and the crop causing different plant sizes, parameter estimation without plant volume information, and occlusion problems caused by the complex structure of the crop.

[0003] The prior art proposes to use laser scanners to obtain three-dimensional information of plants and canopies. However, the hardware equipment is expensive and cumbersome to operate, and data redundancy makes post-processing difficult. If the traditional SfM reconstruction method is used, it has poor robustness, requires a large acquisition space, and takes a long time to reconstruct. It is not suitable for dynamic detection of plants throughout the growth period, and its point cloud model lacks practical physical meaning. The deep learning method is used to reconstruct plants, which depends on the background or the characteristics of the plants themselves. The reconstruction effect is poor for plants with unclear characteristics such as rice and wheat, and its point cloud model lacks practical physical meaning. The spatial carving method is used to obtain three-dimensional models of plants such as soybeans and corn, but most of them infer image external parameters through the shooting time interval and the rotation speed of the rotating table. The system requirements are strict and the operation is complicated. If the plant is not placed at the center of the rotating table during acquisition, it will affect the accuracy of the external parameter calibration.

[0004] In view of the problems in the existing technology, it is urgent to propose a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. Summary of the invention

[0005] In order to solve the above technical problems, the present invention proposes a high-throughput potted plant 3D reconstruction method and system based on stereo calibration to solve the problems existing in the above-mentioned prior art.

[0006] To achieve the above object, the present invention provides a high-throughput 3D reconstruction method for potted plants based on stereo calibration, comprising the following steps:

[0007] A three-dimensional calibration box is made, potted plants are placed in the three-dimensional calibration box, and a camera is used to collect images to obtain an image sequence of each plant after one rotation;

[0008] Obtaining the camera's intrinsic parameter matrix, and performing calibration point recognition on the image sequence to obtain the extrinsic parameter matrix of each image;

[0009] Segment and color render the plant area in the image sequence to obtain a plant binary image;

[0010] Based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image, a voxel carving method is used to perform 3D reconstruction to obtain a plant model;

[0011] The plant model is back-projected and colored to finally obtain a colored plant model.

[0012] Optionally, the process of making a three-dimensional calibration box, placing the potted plant in the three-dimensional calibration box, and using a camera to collect images includes:

[0013] A three-dimensional calibration box is built based on four planar checkerboard calibration plates with the same height as the potted plants, and marking points are set at the intersection vertices of two adjacent planes; the potted plants are placed in the center of the three-dimensional calibration box, and images are collected by fixing the camera and rotating the potted plants at a uniform speed.

[0014] Optionally, the process of performing calibration point recognition on the image sequence to obtain an extrinsic parameter matrix of each image includes:

[0015] Based on the calibration point recognition method, the extrinsic reference region of interest of the image sequence is extracted, and the corner point detection is performed on the extracted extrinsic reference region of interest to obtain the pixel coordinates of the corner points in each image; based on the change of the coordinates of the centroid points of the extrinsic reference regions of interest of adjacent images, the number of occurrences of each plane is obtained, and the corner point detection order is preset. Based on the number of occurrences of each plane and the corner point detection order, the world coordinates of each corner point are obtained; based on the pixel coordinates and world coordinates of each corner point, the extrinsic parameter matrix of each image is obtained.

[0016] Optionally, the process of extracting the extrinsic reference region of interest of the image sequence based on the calibration point recognition method includes:

[0017] Based on the markers on the stereo calibration box, several contours of each image are obtained;

[0018] Several contours of each image are ranked based on their areas;

[0019] Counting the number of contours of each image, and discarding the corresponding image when the number of contours is less than a preset number;

[0020] When the number of contours is equal to a preset number, the vertex coordinates of a rectangular area surrounding the plant target are calculated based on the centroid points of the four contours, and the external reference region of interest of the image is extracted based on the vertex coordinates of the rectangular area;

[0021] When the number of contours is greater than a preset number, for all sorted contours, the centroid points of the first three contours and the centroid points of the remaining contours are calculated, and the contour centroid point with the smallest column coordinate distance from the third contour centroid point is selected from the centroid points of the remaining contours. The vertex coordinates of the rectangular area surrounding the plant target are calculated together with the centroid points of the first three contours, and the external reference region of interest of the image is extracted based on the vertex coordinates of the rectangular area.

[0022] Optionally, the process of performing corner point detection on the extracted extrinsic reference region of interest includes:

[0023] Perform corner point detection on the extracted extrinsic reference region of interest to obtain the pixel coordinates of the corner points;

[0024] Divide the corner points along the length direction of the chessboard into a group, and the number of groups is the same as the number of corner points along the height direction of the chessboard;

[0025] Sort the groups according to the ascending order of the ordinate of the first corner point in each group.

[0026] Optionally, the corner point detection sequence is: starting from the upper left corner point, and following the sequence from left to right and from top to bottom.

[0027] Optionally, the process of obtaining the number of occurrences of each plane based on the change in the coordinates of the centroid of the region of interest of the adjacent image extrinsic reference includes:

[0028] Preset rotation direction;

[0029] Get the centroid point of the region of interest of each image extrinsic reference;

[0030] When the centroid rotates clockwise, the image sequence belonging to the same plane is identified by comparing the changes in the centroid column coordinates of the image, and the number of images in the sequence is output when the centroid column coordinates begin to increase, and then the next group of images is processed until all images are processed;

[0031] When the centroid rotates counterclockwise, the image sequence belonging to the same plane is identified by comparing the changes in the centroid column coordinates of the image, and the number of images in the sequence is output when the centroid column coordinates begin to decrease, and then the next group of images is processed until all images are processed.

[0032] Optionally, based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image, a voxel carving method is used to perform 3D reconstruction to obtain a plant model, which includes:

[0033] The plant binary image is converted into one-dimensional data and encoded into decimal numbers for storage; a cube containing the sample to be reconstructed is initialized and the cube is divided into a number of voxels; each voxel point is projected onto each image in turn, and the number of foreground points is counted to determine whether the voxel belongs to the plant, and the voxels that do not belong to the plant are discarded; finally, all the plant binary images are processed using CUDA parallel computing technology to complete the three-dimensional reconstruction of the plant.

[0034] Optionally, the process of performing back-projection coloring on the plant model to obtain a colored plant model includes:

[0035] Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, back-projection ray tracing is performed on the pixels of each image to determine the shell points corresponding to the plant area, and the pixel colors are assigned to the corresponding shell points. All colored shell points are merged, and the overlapping points are removed by calculating the color mean to finally obtain a colored plant model.

[0036] The present invention also provides a high-throughput potted plant 3D reconstruction system based on stereo calibration, which is used to implement a high-throughput potted plant 3D reconstruction method based on stereo calibration, comprising:

[0037] An image acquisition module is used to make a three-dimensional calibration box, place potted plants in the three-dimensional calibration box, and use a camera to acquire images to obtain an image sequence of each plant after one rotation;

[0038] A camera calibration module, used to obtain a camera intrinsic parameter matrix, and to identify calibration points on the image sequence to obtain an extrinsic parameter matrix for each image;

[0039] An image segmentation module is used to perform segmentation processing and color rendering on the plant area in the image sequence to obtain a plant binary image;

[0040] A voxel carving reconstruction module is used to perform 3D reconstruction using a voxel carving method based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image to obtain a plant model;

[0041] The back-projection coloring module is used to perform back-projection coloring on the plant model to finally obtain a colored plant model.

[0042] Compared with the prior art, the present invention has the following advantages and technical effects:

[0043] The present invention proposes a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. In the image acquisition part, a three-dimensional reconstruction method based on a stereo calibration box is adopted to realize the process of extrinsic parameter calibration for each image. Compared with the extrinsic parameters calculated by feature point matching, the extrinsic parameters obtained based on the stereo calibration box are more accurate, which provides a basis for subsequent reconstruction; and during the acquisition process, there is no need to place each plant at the rotation center, which is more flexible.

[0044] In the camera calibration section, users can customize parameters according to actual conditions, which is more in line with the user's usage scenarios and enhances applicability; customized corner point detection order avoids errors and can be applied to calibration boxes of any size.

[0045] In the voxel carving algorithm part, only one cycle is required for reconstruction, which improves the reconstruction efficiency. At the same time, users can adaptively adjust the reconstruction threshold according to the image acquisition quality, and the reconstruction quality can be selected independently.

[0046] In the point cloud coloring part, the algorithm optimizes the process, reduces the demand for computer performance, realizes automatic image selection and coloring, and greatly shortens the time. The entire process not only enables users to customize the main parameters to enhance applicability, but also realizes automation and can batch process multiple plant samples at different periods. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0048] Figure 1 Schematic diagram of the process of a high-throughput 3D reconstruction method of potted plants based on stereo calibration according to an embodiment of the present invention;

[0049] Figure 2 Schematic diagram of the structure of a high-throughput potted plant 3D reconstruction system based on stereo calibration according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0051] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] At present, there are the following technical problems in the 3D reconstruction methods: first, how to realize the automation of the entire process of plant 3D reconstruction; second, the collected images often have cluttered backgrounds and plants with different screen proportions; third, in terms of camera calibration, the existing methods lack external parameter calibration methods for each view; finally, the traditional 3D reconstruction algorithm is inefficient, and the reconstruction method based on deep learning not only requires a large amount of complex data annotation, but also does not work well for plants with unclear features such as rice and wheat, and the point cloud model usually cannot be directly mapped to the real physical size, and still requires additional unit conversion steps.

[0053] In response to the above-mentioned problems in the prior art, this embodiment proposes a high-throughput potted plant 3D reconstruction method and system based on stereo calibration, which realizes high-throughput and fully automatic three-dimensional reconstruction of potted plants throughout the entire growth period. The reconstructed point cloud still maintains the original plant scale, facilitating subsequent phenotypic extraction and other work.

[0054] like Figure 1 As shown, the high-throughput potted plant 3D reconstruction method based on stereo calibration provided in this embodiment includes image acquisition, camera calibration, image segmentation, voxel carving reconstruction and back-projection coloring; wherein:

[0055] Image acquisition, including using a plane chessboard calibration plate to acquire internal reference images, making a three-dimensional calibration box composed of four plane chessboard calibration plates for external reference calibration of plants, placing potted plants in the center of the calibration box, and acquiring images by fixing the camera and rotating the plants at a constant speed to obtain an image sequence of each plant after one rotation;

[0056] Camera calibration, including obtaining the camera intrinsic parameter matrix and camera distortion coefficient by using Zhang Zhengyou calibration method for the intrinsic reference image, extracting the main plane in the image, i.e., the extrinsic reference region of interest (ROI), by identifying the marker points for the plant image, performing corner point detection on the ROI to obtain the pixel coordinates of each corner point in each image, automatically obtaining the number of occurrences of each plane according to the change in the coordinates of the centroid points of the ROI of adjacent images, determining the world coordinates of each corner point based on the corner point detection order and the number of occurrences of each plane, and calculating the extrinsic parameter matrix of each image through the pixel coordinates and world coordinates of each corner point;

[0057] Image segmentation: use color threshold to roughly segment the plant area, perform connected domain analysis on the rough segmentation result to remove noise points in small areas to obtain fine segmentation results, and perform color rendering on the fine segmentation result to obtain a binary image of the plant area;

[0058] Voxel carving reconstruction: input the plant binary image, the external parameters corresponding to each view and the camera internal parameters, reconstruct based on the voxel carving method, and use CUDA parallel computing to improve the reconstruction efficiency to obtain a colorless point cloud model, that is, a colorless plant model;

[0059] Back-projection coloring, considering that only the outermost points in the point cloud model have color, according to the camera intrinsic parameters and the extrinsic parameter matrix of each image, the three-dimensional points corresponding to the plant area, namely the shell points, are determined by back-projection ray tracing for the pixels of each image, and the pixel color is assigned to the corresponding shell point. All colored shell points are merged, and the overlapping points are removed by calculating their color mean, and finally a point cloud model with color information is obtained, that is, a colored plant model.

[0060] As a specific embodiment, image acquisition is specifically achieved in the following manner: a plane chessboard calibration plate is placed in front of the lens, and several internal reference images at different angles are obtained by continuously changing the position and angle; the three-dimensional calibration box is designed similar to a fish tank, and the side is composed of 4 plane calibration plates, the height of which is the same as the height of the pot, and the two vertices where two adjacent planes intersect are affixed with red semicircular marking points of the same size, the inside of the calibration box is hollow and a 5 cm high circular bayonet is provided in the center of the bottom; the origin of the world coordinate system is defined as the center of the upper surface of the calibration box, and the Z axis is placed along the height direction. The potted plant is placed in the circular bayonet inside the calibration box to achieve the coincidence of the center of the potted plant with the center of the calibration box; the camera is fixed and the plant rotates at a uniform speed for image acquisition, and an image sequence of each plant rotating one circle is obtained.

[0061] As a specific embodiment, when performing camera calibration, two checkerboard planes may appear in one image during the rotation of the calibration box, and it is necessary to segment the main plane area with a larger area, that is, the ROI of the external reference image;

[0062] Among them, the camera external reference image ROI extraction is specifically implemented through the following steps:

[0063] Step 3-1, find the red marked area in the image, hereinafter referred to as the contour;

[0064] Step 3-2, sort all contours according to their area from large to small;

[0065] Step 3-3, count the number of contours, and perform the following judgments and operations:

[0066] 3-3-1, determine whether it is less than 4, if it is, directly proceed to settlement step 3-5-1;

[0067] 3-3-2, determine whether it is equal to 4, if it is, directly proceed to settlement step 3-5-2;

[0068] 3-3-3, determine whether it is greater than 4, if so, go to step 3-4;

[0069] Step 3-4, for all the sorted contours, calculate the centroids of the first three contours, then calculate the centroids of the remaining contours, find the contour centroid with the smallest column coordinate distance from the third contour centroid, and enter the settlement step 3-5-2;

[0070] Step 3-5, settlement steps:

[0071] 3-5-1, if the number of contours is less than 4, the image is discarded;

[0072] 3-5-2, if the number of contour centroid points is equal to 4, the vertex coordinates of the rectangular area surrounding the target are calculated based on the four centroid points, and the external reference image ROI is extracted on the original image based on the above bounding box coordinate information, and the area outside the ROI is set to black.

[0073] Furthermore, during camera calibration, in order to avoid confusion in the order of corner point detection in different image sequences, the pixel coordinates of each corner point in each image are reordered and saved to ensure that the starting corner point is always the first one in the upper left corner, and the detection order is from left to right and from top to bottom;

[0074] Furthermore, the corner point detection for the ROI of each image is implemented by the following steps:

[0075] Step 4-1, input the ROI image to perform corner point detection and obtain the pixel coordinates of the corner points;

[0076] Step 4-2, group all corner points into groups of patternSize_x, with a total of patternSize_y groups, where patternSize_x and patternSize_y are the number of corner points along the length direction and height direction of the chessboard, respectively;

[0077] Step 4-3, sort the patternSize_y groups according to the y coordinate of the first corner point in each group from small to large.

[0078] Furthermore, when performing camera calibration, since the stereo calibration box performs rotational motion, it is necessary to automatically obtain the number of occurrences of each plane according to the coordinate changes of the centroid points of the ROI of the adjacent images;

[0079] The number of occurrences of each plane is obtained by the following steps:

[0080] Step 5-1, manually input the rotation direction;

[0081] Step 5-2, calculate the centroid of each image ROI in turn;

[0082] Step 5-3, according to the rotation direction, make the following judgments and operations:

[0083] 5-3-1 Determine whether it rotates clockwise, if yes, proceed to step 5-4;

[0084] 5-3-2 Determine whether it rotates counterclockwise, if yes, proceed to step 5-5;

[0085] Step 5-4, rotate clockwise:

[0086] 5-4-1, sequentially compare the changes in the centroid column coordinates of two adjacent images;

[0087] 5-4-2, if the centroid point column coordinates gradually decrease, it means that the ROIs of these images belong to the same plane, and this process is called the same round of movement;

[0088] 5-4-3, until the centroid column coordinates of the current image are greater than the previous centroid, output the number of images accumulated in the previous round of movement, excluding the current image;

[0089] 5-4-4, use the current image as the starting image for the next plane movement and return to step 5-4-1;

[0090] 5-4-5, determine whether the current image is the last one. If so, output the number of images accumulated during the movement and end the algorithm;

[0091] Step 5-5, rotate counterclockwise:

[0092] 5-5-1, sequentially compare the changes in the centroid column coordinates of two adjacent images;

[0093] 5-5-2, if the centroid point column coordinates gradually increase, it means that the ROIs of these images belong to the same plane, and this process is called the same round of movement;

[0094] 5-5-3, until the column coordinates of the centroid of the current image are smaller than the previous centroid, output the number of images accumulated in the previous round of movement, excluding the current image;

[0095] 5-5-4, use the current image as the starting image for the next plane movement and return to step 5-5-1;

[0096] 5-5-5, determine whether the current image is the last one. If so, output the number of images accumulated during the movement and end the algorithm.

[0097] Furthermore, when performing camera calibration, the calculation of the extrinsic parameter matrix of each view is specifically implemented in the following way: the origin of the world coordinate system is defined as the center of the upper surface of the cubic calibration box, and the Z axis is along the growth direction of the plant; according to the customized world coordinate system, based on the detection order of the corner point pixel coordinates and the number of occurrences of the above-mentioned planes, the world coordinates of each corner point in each image are obtained; the extrinsic parameter matrix of each view is calculated by the iterative method in solvePnP.

[0098] As a specific embodiment, when performing voxel carving, by inputting the segmented plant binary image, the external parameters corresponding to each view and the camera internal parameters, reconstruction is performed based on the voxel carving method to obtain a colorless point cloud model;

[0099] Among them, voxel carving is specifically achieved through the following steps:

[0100] Step 7-1, for the plant binary image, define the foreground point as 1 and the background point as 0; merge the image into one-dimensional data, the size of which is the number of pixels; record each group of 8 data as a binary number, convert it into a decimal number for storage, and the range of the decimal number is 0-255;

[0101] Step 7-2, initialize a cube whose size can completely contain the sample to be reconstructed; divide the cube evenly into a number of small cubes, namely voxels;

[0102] Step 7-3, using the centroid of each cube to represent each voxel, i.e., voxel point, for each voxel point, setting a number m of reconstructed images, with an initial value of 0;

[0103] Step 7-4, project each voxel point onto each image in turn, and determine whether it belongs to the foreground or background in the current image. If it belongs to the foreground point, then m+1;

[0104] Step 7-5, using CUDA parallel computing, so that each thread corresponds to a voxel one by one, and the corresponding relationship is represented by a thread index until all views are traversed;

[0105] Step 7-6, set a reconstruction image threshold n, and judge m. If m≥n, the voxel point is considered to be a plant point and is retained; otherwise, it is a background point and is discarded.

[0106] As a specific embodiment, when performing back-projection coloring, according to the camera intrinsic parameters and the extrinsic parameter matrix of each image, the three-dimensional point corresponding to the plant area, i.e., the shell point, is determined by back-projection ray tracing for the pixel points of each image, the color of the pixel point is assigned to the corresponding shell point, all the colored shell points are merged, and the overlapping points are removed by calculating their color mean, and finally a point cloud model with color information is obtained;

[0107] Among them, the back projection shading is specifically implemented through the following steps:

[0108] Step 8-1, read the original image, binary image and corresponding camera parameters for color rendering. Starting from N views, use the undistort function to correct the image distortion and save it;

[0109] Step 8-2, traverse the foreground points (two-dimensional pixel points) of the plant area in the current view, use GPU optimization to optimize these foreground points, and use each kernel function to perform back-projection calculation on each foreground point to determine its corresponding voxel point; obtain the shell model corresponding to each view, a total of N;

[0110] Step 8-3, for the retained shell points, obtain the RGB information of the corresponding pixel points in each image in turn, and assign it to the corresponding voxel point, and finally output the shell model containing color information;

[0111] Step 8-4, for the colored shell model, point cloud merging is performed, and overlapping points are removed by calculating their color mean, and finally a complete reconstructed point cloud model is obtained.

[0112] like Figure 2 As shown, this embodiment also provides a high-throughput potted plant 3D reconstruction system based on stereo calibration, which is used to implement a high-throughput potted plant 3D reconstruction method based on stereo calibration. The system includes the following modules:

[0113] The image acquisition module includes using a plane chessboard calibration plate to acquire internal reference images, making a three-dimensional calibration box composed of four plane chessboard calibration plates for external reference calibration of plants, placing potted plants in the center of the calibration box, and acquiring images by fixing the camera and rotating the plants at a constant speed to obtain an image sequence of each plant after one rotation;

[0114] The camera calibration module includes obtaining the camera intrinsic parameter matrix and camera distortion coefficient by using Zhang Zhengyou calibration method for the intrinsic reference image, extracting the main plane in the image, i.e., the external reference ROI, by identifying the marker points for the plant image, performing corner point detection on the ROI to obtain the pixel coordinates of each corner point in each image, automatically obtaining the number of occurrences of each plane according to the change of the coordinates of the centroid points of the ROI of adjacent images, determining the world coordinates of each corner point based on the corner point detection sequence and the number of occurrences of each plane, and calculating the external reference matrix of each image through the pixel coordinates and world coordinates of each corner point;

[0115] The image segmentation module uses a color threshold to roughly segment the plant area, performs a connected domain analysis on the rough segmentation result to remove noise points in small areas to obtain a fine segmentation result, and performs color rendering on the fine segmentation result to obtain a binary image of the plant area;

[0116] The voxel carving reconstruction module inputs the plant binary image, the external parameters corresponding to each view and the camera internal parameters, reconstructs based on the voxel carving method, and uses CUDA parallel computing to improve the reconstruction efficiency and obtain a colorless point cloud model;

[0117] The back-projection coloring module takes into account that only the outermost points in the point cloud model have color. According to the camera intrinsic parameters and the extrinsic parameter matrix of each image, the three-dimensional points corresponding to the plant area, namely the shell points, are determined by back-projection ray tracing for the pixels of each image. The pixel color is assigned to the corresponding shell point, all the colored shell points are merged, and the overlapping points are removed by calculating their color mean, finally obtaining a point cloud model with color information.

[0118] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A high-throughput 3D reconstruction method for potted plants based on stereo calibration, characterized in that: The following steps are involved: A three-dimensional calibration box is made, potted plants are placed in the three-dimensional calibration box, and a camera is used to collect images to obtain an image sequence of each plant after one rotation; Obtaining the camera's intrinsic parameter matrix, and performing calibration point recognition on the image sequence to obtain the extrinsic parameter matrix of each image; Segment and color render the plant area in the image sequence to obtain a plant binary image; Based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image, a voxel carving method is used to perform 3D reconstruction to obtain a plant model; The plant model is back-projected and colored to finally obtain a colored plant model.

2. The method according to claim 1, characterized in that The process of making a stereoscopic calibration box, placing a potted plant in the stereoscopic calibration box, and using a camera to collect images includes: A three-dimensional calibration box is built based on four planar checkerboard calibration plates with the same height as the potted plants, and marking points are set at the intersection vertices of two adjacent planes; the potted plants are placed in the center of the three-dimensional calibration box, and images are collected by fixing the camera and rotating the potted plants at a uniform speed.

3. The method according to claim 1, characterized in that The process of identifying the calibration points of the image sequence and obtaining the external parameter matrix of each image includes: Based on the calibration point recognition method, the extrinsic reference region of interest of the image sequence is extracted, and the corner point detection is performed on the extracted extrinsic reference region of interest to obtain the pixel coordinates of the corner points in each image; based on the change of the coordinates of the centroid points of the extrinsic reference regions of interest of adjacent images, the number of occurrences of each plane is obtained, and the corner point detection order is preset. Based on the number of occurrences of each plane and the corner point detection order, the world coordinates of each corner point are obtained; based on the pixel coordinates and world coordinates of each corner point, the extrinsic parameter matrix of each image is obtained.

4. The method according to claim 3, characterized in that The process of extracting the extrinsic reference region of interest of the image sequence based on the calibration point recognition method includes: Based on the markers on the stereo calibration box, several contours of each image are obtained; Several contours of each image are ranked based on their areas; Counting the number of contours of each image, and discarding the corresponding image when the number of contours is less than a preset number; When the number of contours is equal to a preset number, the vertex coordinates of a rectangular area surrounding the plant target are calculated based on the centroid points of the four contours, and the external reference region of interest of the image is extracted based on the vertex coordinates of the rectangular area; When the number of contours is greater than a preset number, for all sorted contours, the centroid points of the first three contours and the centroid points of the remaining contours are calculated, and the contour centroid point with the smallest column coordinate distance from the third contour centroid point is selected from the centroid points of the remaining contours. The vertex coordinates of the rectangular area surrounding the plant target are calculated together with the centroid points of the first three contours, and the external reference region of interest of the image is extracted based on the vertex coordinates of the rectangular area.

5. The method according to claim 3, characterized in that: The process of detecting corner points on the extracted extrinsic region of interest includes: Perform corner point detection on the extracted extrinsic reference region of interest to obtain the pixel coordinates of the corner points; Divide the corner points along the length direction of the chessboard into a group, and the number of groups is the same as the number of corner points along the height direction of the chessboard; Sort the groups according to the ascending order of the ordinate of the first corner point in each group.

6. The method according to claim 3, characterized in that The corner point detection sequence is: starting from the upper left corner point, and following the sequence from left to right and from top to bottom.

7. The method according to claim 3, characterized in that Based on the coordinate changes of the centroid points of the extrinsic interest regions of adjacent images, the process of obtaining the number of occurrences of each plane includes: Preset rotation direction; Get the centroid point of the region of interest of each image extrinsic reference; When the centroid rotates clockwise, the image sequence belonging to the same plane is identified by comparing the changes in the centroid column coordinates of the image, and the number of images in the sequence is output when the centroid column coordinates begin to increase, and then the next group of images is processed until all images are processed; When the centroid rotates counterclockwise, the image sequence belonging to the same plane is identified by comparing the changes in the centroid column coordinates of the image, and the number of images in the sequence is output when the centroid column coordinates begin to decrease, and then the next group of images is processed until all images are processed.

8. The method according to claim 1, characterized in that Based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image, a voxel carving method is used to perform 3D reconstruction, and the process of obtaining a plant model includes: The plant binary image is converted into one-dimensional data and encoded into decimal numbers for storage; a cube containing the sample to be reconstructed is initialized and the cube is divided into a number of voxels; each voxel point is projected onto each image in turn, and the number of foreground points is counted to determine whether the voxel belongs to the plant, and the voxels that do not belong to the plant are discarded; finally, all the plant binary images are processed using CUDA parallel computing technology to complete the three-dimensional reconstruction of the plant.

9. The method according to claim 1, characterized in that: The process of performing back-projection coloring on the plant model to obtain a colored plant model includes: Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, back-projection ray tracing is performed on the pixels of each image to determine the shell points corresponding to the plant area, and the pixel colors are assigned to the corresponding shell points. All colored shell points are merged, and the overlapping points are removed by calculating the color mean to finally obtain a colored plant model.

10. A high-throughput 3D reconstruction system for potted plants based on stereo calibration, characterized in that: The method for realizing the high-throughput 3D reconstruction of potted plants based on stereo calibration according to any one of claims 1 to 9 comprises: An image acquisition module is used to make a three-dimensional calibration box, place potted plants in the three-dimensional calibration box, and use a camera to acquire images to obtain an image sequence of each plant after one rotation; A camera calibration module, used to obtain a camera intrinsic parameter matrix, and to identify calibration points on the image sequence to obtain an extrinsic parameter matrix for each image; An image segmentation module is used to perform segmentation processing and color rendering on the plant area in the image sequence to obtain a plant binary image; A voxel carving reconstruction module is used to perform 3D reconstruction using a voxel carving method based on the plant binary image, the camera intrinsic parameter matrix and the extrinsic parameter matrix of each image to obtain a plant model; The back-projection coloring module is used to perform back-projection coloring on the plant model to finally obtain a colored plant model.

Images