A High-Throughput 3D Reconstruction Method and System for Potted Plants Based on Stereo Calibration
The high-throughput 3D reconstruction method for potted plants using stereo calibration solves the problems of expensive equipment, complex operation, and long reconstruction time in existing technologies. It achieves efficient and automated 3D reconstruction of plants throughout their entire growth period, is applicable to calibration boxes of different sizes and custom parameters, and supports batch processing of multiple plant samples.
Patent Information
- Application Number
- CN202411818932.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-11-14
- Estimated Expiration
- 2044-12-11
AI Technical Summary
Existing technologies for 3D reconstruction of plants suffer from problems such as expensive equipment, complex operation, long reconstruction time, poor robustness, and lack of physical meaning in point cloud models. In particular, they are not effective for reconstructing plants with indistinct features.
A high-throughput 3D reconstruction method for potted plants based on stereo calibration is adopted. By creating a stereo calibration box, the image sequence is obtained by fixing the plant with a camera and rotating it at a constant speed. The plant region is segmented and color rendered by combining the camera intrinsic and extrinsic parameter matrices. The voxel sculpting method is used for 3D reconstruction, and the colored plant model is obtained by back projection coloring.
It achieves high-throughput fully automated 3D reconstruction of plants throughout their entire growth period. The reconstructed point cloud maintains the original plant scale, has strong applicability, high reconstruction efficiency, reduces the demand on computer performance, is suitable for calibration boxes of different sizes and custom parameters, and supports batch processing of multiple plant samples.
Smart Images
Figure CN119741426B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of agricultural automation technology, and in particular relates to a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. Background Technology
[0002] Over the past decade, the rapid development of new sensors and automation technologies has brought various new technologies to the agricultural field, attracting widespread attention. Two-dimensional imaging technology is commonly used for estimating structural traits, growth analysis, and yield estimation at different growth stages of crops. Although many hardware systems and corresponding image processing and analysis algorithms have been developed, some unavoidable shortcomings remain. These include variations in plant size caused by changes in the camera viewpoint and distance between the crop and the plant, parameter estimation lacking information on plant volume, and occlusion problems caused by the complex structure of the crop.
[0003] Existing technologies propose using laser scanners to acquire 3D information of plants and canopies; however, the hardware is expensive and the operation is cumbersome, and data redundancy makes post-processing difficult. Traditional SfM reconstruction methods have poor robustness, require a large acquisition space, and have long reconstruction times, making them unsuitable for dynamic detection of plants throughout their entire growth period, and their point cloud models lack practical physical meaning. Deep learning methods for plant reconstruction rely on background or plant features, resulting in poor reconstruction effects for plants with indistinct features, such as rice and wheat, and their point cloud models lack practical physical meaning. Spatial sculpting methods are used to acquire 3D models of plants such as soybeans and corn, but most of these methods deduce image extrinsic parameters by reversing the shooting time interval and the rotation speed of the rotary table. The system requirements are strict and the operation is complex. If the plant is not placed in the center of the rotary table during acquisition, it will affect the accuracy of extrinsic parameter calibration.
[0004] To address the problems in existing technologies, there is an urgent need to propose a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention proposes a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration, thereby resolving the issues present in the prior art.
[0006] To achieve the above objectives, this invention provides a high-throughput 3D reconstruction method for potted plants based on stereo calibration, comprising the following steps:
[0007] A stereo calibration box is made, and the potted plants are placed in the stereo calibration box. Images are acquired using a camera to obtain an image sequence of each plant rotating one revolution.
[0008] Obtain the camera intrinsic parameter matrix and perform calibration point identification on the image sequence to obtain the extrinsic parameter matrix for each image;
[0009] The plant regions in the image sequence are segmented and color-rendered to obtain a binary image of the plant.
[0010] Based on the plant binary image, camera intrinsic parameter matrix and extrinsic parameter matrix of each image, a voxel sculpting method is used to perform 3D reconstruction to obtain the plant model.
[0011] The plant model is then back-projected and colored to obtain a colored plant model.
[0012] Optionally, a stereo calibration box is fabricated, and the process of placing the potted plant in the stereo calibration box and acquiring images using a camera includes:
[0013] A three-dimensional calibration box is constructed based on four planar checkerboard calibration boards of the same height as the potted plant, and marker points are set at the intersection vertices of two adjacent planes. The potted plant is placed in the center of the three-dimensional calibration box, and images are acquired by fixing the camera and rotating the potted plant at a constant speed.
[0014] Optionally, the process of performing calibration point identification on the image sequence to obtain the extrinsic parameter matrix of each image includes:
[0015] The extrinsic region of interest (ROI) of the image sequence is extracted based on the calibration point recognition method. Corner detection is performed on the extracted ROI to obtain the pixel coordinates of the corners in each image. Based on the change of centroid coordinates of the ROI of adjacent images, the occurrence frequency of each plane is obtained. The corner detection order is preset. Based on the occurrence frequency of each plane and the corner detection order, the world coordinates of each corner are obtained. Based on the pixel coordinates and world coordinates of each corner, the extrinsic parameter matrix of each image is obtained.
[0016] Optionally, the process of extracting the extrinsic region of interest of the image sequence based on the calibration point recognition method includes:
[0017] Based on the marker points on the stereo calibration box, several contours of each image are obtained;
[0018] Based on the contour area, several contours in each image are sorted.
[0019] Count the number of contours in each image; if the number of contours is less than a preset number, discard the corresponding image.
[0020] When the number of contours is equal to the preset number, the vertex coordinates of the rectangular region surrounding the plant target are calculated based on the centroid points of the four contours, and the extrinsic region of interest of the image is extracted based on the vertex coordinates of the rectangular region.
[0021] When the number of contours is greater than a preset number, for all sorted contours, the centroids of the first three contours and the remaining contours are calculated. From the centroids of the remaining contours, the contour centroid with the smallest column coordinate distance to the centroid of the third contour is selected. Together with the centroids of the first three contours, the vertex coordinates of the rectangular region surrounding the plant target are calculated. Based on the vertex coordinates of the rectangular region, the extrinsic region of interest of the image is extracted.
[0022] Optionally, the process of corner detection on the extracted extrinsic region of interest includes:
[0023] Corner detection is performed on the extracted extrinsic region of interest to obtain the pixel coordinates of the corners;
[0024] Divide the corner points along the length of the chessboard into groups, with the number of groups being the same as the number of corner points along the height of the chessboard.
[0025] Sort the groups according to the y-coordinate of the first corner point in each group from smallest to largest.
[0026] Optionally, the corner detection order is as follows: starting from the top left corner, proceeding sequentially from left to right and from top to bottom.
[0027] Optionally, the process of obtaining the occurrence count of each plane based on the change in the centroid coordinates of the regions of interest in adjacent image extrinsic parameters includes:
[0028] Preset rotation direction;
[0029] Obtain the centroid of the region of interest for each image's extrinsic parameters;
[0030] As the centroid rotates clockwise, by comparing the changes in the centroid column coordinates of the images, the image sequences belonging to the same plane are identified, and the number of images in the sequence is output when the centroid column coordinates begin to increase. Then, the next set of images is processed until all images have been processed.
[0031] As the centroid rotates counterclockwise, the changes in the centroid coordinates of the image are compared to identify the image sequence belonging to the same plane. When the centroid coordinates begin to decrease, the number of images in that sequence is output, and then the next set of images is processed until all images have been processed.
[0032] Optionally, based on the plant binary image, camera intrinsic parameter matrix, and extrinsic parameter matrix of each image, a voxel sculpting method is used for 3D reconstruction to obtain the plant model. The process includes:
[0033] The binary images of the plants are converted into one-dimensional data and encoded into decimal numbers for storage. A cube containing the samples to be reconstructed is initialized and divided into several voxels. Each voxel is projected onto each image in sequence, and the number of foreground points is counted to determine whether a voxel belongs to a plant. Voxels that do not belong to a plant are discarded. Finally, CUDA parallel computing technology is used to process all the binary images of the plants to complete the three-dimensional reconstruction of the plants.
[0034] Optionally, the process of performing back-projection coloring on the plant model to obtain a colored plant model includes:
[0035] Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, back-projection ray tracing is performed on the pixels of each image to determine the shell points corresponding to the plant region. The pixel color is assigned to the corresponding shell point, all colored shell points are merged, and overlapping points are removed by calculating the color mean, finally obtaining the colored plant model.
[0036] This invention also provides a high-throughput 3D reconstruction system for potted plants based on stereo calibration, used to realize a method for high-throughput 3D reconstruction of potted plants based on stereo calibration, comprising:
[0037] The image acquisition module is used to create a stereo calibration box. The potted plants are placed in the stereo calibration box and images are acquired by a camera to obtain an image sequence of each plant rotating one revolution.
[0038] The camera calibration module is used to obtain the camera intrinsic parameter matrix and to perform calibration point identification on the image sequence to obtain the extrinsic parameter matrix of each image.
[0039] The image segmentation module is used to segment and color render the plant region in the image sequence to obtain a binary image of the plant.
[0040] The voxel sculpting and reconstruction module is used to perform 3D reconstruction based on the plant binary image, camera intrinsic parameter matrix and extrinsic parameter matrix of each image, using the voxel sculpting method to obtain the plant model.
[0041] The back-projection coloring module is used to perform back-projection coloring on the plant model, and finally obtain a colored plant model.
[0042] Compared with the prior art, the present invention has the following advantages and technical effects:
[0043] This invention proposes a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. In the image acquisition part, a three-dimensional reconstruction method based on stereo calibration box is adopted to realize the process of extrinsic parameter calibration for each image. Compared with the extrinsic parameters calculated by feature point matching, the extrinsic parameters obtained based on stereo calibration box are more accurate, providing a foundation for subsequent reconstruction. Moreover, it is not necessary to place each plant at the center of rotation during the acquisition process, which makes it more flexible.
[0044] In the camera calibration section, users can customize parameters according to actual conditions, which better suits the user's usage scenarios and enhances applicability; the customizable corner detection order avoids errors and is applicable to calibration boxes of any size.
[0045] In the voxel sculpting algorithm, reconstruction only requires one loop, which improves reconstruction efficiency. At the same time, users can adaptively adjust the reconstruction threshold according to the image acquisition quality, and the reconstruction quality can be selected independently.
[0046] In the point cloud coloring process, the algorithm optimization reduces the demand on computer performance, automates image selection and coloring, and significantly shortens the time. The entire process not only allows users to customize key parameters, enhancing applicability, but also automates the process, enabling batch processing of multiple plant samples from different stages. Attached Figure Description
[0047] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:
[0048] Figure 1 This is a flowchart illustrating the high-throughput 3D reconstruction method for potted plants based on stereo calibration, according to an embodiment of the present invention.
[0049] Figure 2 This is a schematic diagram of the structure of a high-throughput 3D reconstruction system for potted plants based on stereo calibration, according to an embodiment of the present invention. Detailed Implementation
[0050] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.
[0051] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.
[0052] Currently, 3D reconstruction methods face the following technical challenges: First, how to automate the entire process of 3D reconstruction of plants; second, the acquired images often have cluttered backgrounds and inconsistent plant sizes; third, in terms of camera calibration, existing methods lack extrinsic parameter calibration methods for each view; and finally, traditional 3D reconstruction algorithms are inefficient, while deep learning-based reconstruction methods not only require a large amount of complex data annotation but also do not perform well for plants with indistinct features, such as rice and wheat. Furthermore, point cloud models usually cannot be directly mapped to real physical dimensions, requiring additional unit conversion steps.
[0053] To address the aforementioned problems in the prior art, this embodiment proposes a high-throughput 3D reconstruction method and system for potted plants based on stereo calibration. This method achieves high-throughput, fully automated 3D reconstruction of potted plants throughout their entire growth period, and the reconstructed point cloud retains the original plant scale, facilitating subsequent phenotypic extraction and other tasks.
[0054] like Figure 1 As shown, the high-throughput 3D reconstruction method for potted plants based on stereo calibration provided in this embodiment includes image acquisition, camera calibration, image segmentation, voxel sculpting reconstruction, and back-projection coloring; wherein:
[0055] Image acquisition includes acquiring internal reference images using a planar checkerboard calibration plate, creating a three-dimensional calibration box consisting of four planar checkerboard calibration plates for external reference calibration of the plant, placing the potted plant in the center of the calibration box, and acquiring images by fixing the camera and rotating the plant at a constant speed to obtain an image sequence of each plant rotating one revolution.
[0056] Camera calibration includes obtaining the camera intrinsic parameter matrix and camera distortion coefficients using Zhang Zhengyou's calibration method for intrinsic parameter images; extracting the principal plane, i.e. the extrinsic region of interest (ROI), in plant images through marker point recognition; obtaining the pixel coordinates of each corner point in each image by performing corner point detection for the ROI; automatically obtaining the occurrence frequency of each plane based on the change of the centroid coordinates of the ROI in adjacent images; determining the world coordinates of each corner point based on the corner point detection order and the occurrence frequency of each plane; and calculating the extrinsic parameter matrix of each image using the pixel coordinates and world coordinates of each corner point.
[0057] Image segmentation is performed by using a color threshold to coarsely segment the plant region. Connectivity analysis is then performed on the coarse segmentation results to remove noise points in small regions and obtain fine segmentation results. Finally, color rendering is applied to the fine segmentation results to obtain a binary image of the plant region.
[0058] Voxel sculpting reconstruction: Input the binary image of the plant, the extrinsic parameters of each view and the intrinsic parameters of the camera, and reconstruct it based on the voxel sculpting method. CUDA parallel computing is used to improve the reconstruction efficiency and obtain a colorless point cloud model, i.e. a colorless plant model.
[0059] Back-projection coloring considers that only the outermost points in the point cloud model have color. Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, the three-dimensional points corresponding to the plant area, i.e., the shell points, are determined for each image pixel by back-projection ray tracing. The color of the pixel is assigned to its corresponding shell point. All colored shell points are merged, and overlapping points are removed by calculating their color mean. Finally, a point cloud model with color information is obtained, i.e., a colored plant model.
[0060] As a specific implementation, image acquisition is achieved as follows: a planar checkerboard calibration plate is placed in front of the lens, and several internal reference images are obtained from different angles by continuously changing its position and angle; the three-dimensional calibration box is designed like a fish tank, with four planar calibration plates on the side, the height of which is the same as the height of the pot, and two red semi-circular markers of the same size are affixed to the two vertices where two adjacent planes intersect; the calibration box is hollow inside and has a circular slot with a height of 5cm at the bottom center; the origin of the world coordinate system is defined as the center of the upper surface of the calibration box, and the Z-axis is along the height direction; the potted plant is placed in the circular slot inside the calibration box to make the center of the potted plant coincide with the center of the calibration box; image acquisition is performed by fixing the camera and rotating the plant at a uniform speed to obtain an image sequence of each plant rotating one revolution.
[0061] As a specific example, during camera calibration, during the rotation of the calibration box, an image may contain two checkerboard planes. It is necessary to segment out the main plane region with a larger area, which is the ROI of the extrinsic image.
[0062] The extraction of the ROI from the camera extrinsic image is achieved through the following steps:
[0063] Step 3-1: Locate the red-marked areas in the image, hereinafter referred to as the outline;
[0064] Step 3-2: Sort all contours from largest to smallest area.
[0065] Step 3-3: Count the number of contours and perform the following judgments and operations:
[0066] 3-3-1, Determine if it is less than 4; if so, proceed directly to settlement step 3-5-1.
[0067] 3-3-2, check if it equals 4; if so, proceed directly to settlement step 3-5-2.
[0068] 3-3-3, determine if it is greater than 4; if so, proceed to step 3-4.
[0069] Step 3-4: For all sorted contours, calculate the centroids of the first three contours, then calculate the centroids of the remaining contours, find the centroid of the contour with the smallest column coordinate distance to the centroid of the third contour, and proceed to the settlement step 3-5-2.
[0070] Steps 3-5, Settlement Steps:
[0071] 3-5-1, If the number of contours is less than 4, then discard the image;
[0072] 3-5-2, If the number of centroid points of the contour is equal to 4, then calculate the vertex coordinates of the rectangular region surrounding the target based on the four centroid points. Based on the above bounding box coordinate information, perform ROI extraction on the original image and set the region outside the ROI to black.
[0073] Furthermore, during camera calibration, in order to avoid the disorder of different image sequences when performing corner detection, the pixel coordinates of each corner point in each image are reordered and saved to ensure that the starting corner point is always the first one in the upper left corner, and the detection order is from left to right and from top to bottom.
[0074] Furthermore, corner detection for each ROI is achieved through the following steps:
[0075] Step 4-1: Input the ROI image and perform corner detection to obtain the pixel coordinates of the corners;
[0076] Step 4-2: Group all corner points into groups of patternSize_x, for a total of patternSize_y groups, where patternSize_x and patternSize_y are the number of corner points along the length and height of the chessboard, respectively.
[0077] Step 4-3: Sort the above patternSize_y groups according to the y-coordinate of the first corner point in each group from smallest to largest.
[0078] Furthermore, during camera calibration, since the stereo calibration box rotates, it is necessary to automatically obtain the number of times each plane appears based on the changes in the centroid coordinates of the ROIs in adjacent images.
[0079] The specific steps to obtain the occurrence count of each plane are as follows:
[0080] Step 5-1: Manually input the rotation direction;
[0081] Step 5-2: Calculate the centroid of the ROI for each image in sequence;
[0082] Step 5-3: Based on the direction of rotation, perform the following judgments and operations:
[0083] 5-3-1 Determine if it rotates clockwise; if so, proceed to step 5-4.
[0084] 5-3-2 Determine if it rotates counterclockwise; if so, proceed to step 5-5.
[0085] Step 5-4, rotate clockwise:
[0086] 5-4-1, sequentially compare the changes in the column coordinates of the centroid points in two adjacent images;
[0087] 5-4-2, If the coordinates of the centroid points gradually decrease, it indicates that the ROIs of these images belong to the same plane, and this process is called the same round of movement;
[0088] 5-4-3, until the column coordinates of the centroid of the current image are greater than the previous centroid, output the number of images accumulated in the previous round of movement, excluding the current image;
[0089] 5-4-4, Use the current image as the starting image for the next plane movement, and return to step 5-4-1;
[0090] 5-4-5, Determine if the current image is the last one. If so, output the number of images accumulated during the movement and end the algorithm.
[0091] Step 5-5, rotate counterclockwise:
[0092] 5-5-1, sequentially compare the changes in the column coordinates of the centroid points in two adjacent images;
[0093] 5-5-2, If the coordinates of the centroid points gradually increase, it indicates that the ROIs of these images belong to the same plane, and this process is called the same round of movement;
[0094] 5-5-3, until the column coordinates of the centroid of the current image are less than the previous centroid, output the number of images accumulated in the previous round of movement, excluding the current image;
[0095] 5-5-4, Use the current image as the starting image for the next plane movement, and return to step 5-5-1;
[0096] 5-5-5, determine if the current image is the last one. If so, output the number of images accumulated during the movement and end the algorithm.
[0097] Furthermore, during camera calibration, the extrinsic matrix of each view is calculated as follows: the origin of the world coordinate system is defined as the center of the upper surface of the cube calibration box, and the Z-axis is along the plant growth direction; based on the custom world coordinate system, the world coordinates of each corner point in each image are obtained according to the detection order of the corner pixel coordinates and the occurrence frequency of each plane mentioned above; the extrinsic matrix of each view is calculated using the iterative method in solvePnP.
[0098] As a specific implementation, when performing voxel sculpting, the segmented binary image of the plant, the extrinsic parameters of each view and the intrinsic parameters of the camera are input, and the reconstruction is performed based on the voxel sculpting method to obtain a colorless point cloud model.
[0099] Voxel sculpting is achieved through the following steps:
[0100] Step 7-1: For the binary image of the plant, define the foreground point as 1 and the background point as 0; merge the image into one-dimensional data, the size of which is the number of pixels; group every 8 data points into a binary number, convert it into a decimal number for storage, and the range of this decimal number is 0-255;
[0101] Step 7-2: Initialize a cube whose size can completely contain the sample to be reconstructed; divide the cube into several smaller cubes, i.e., voxels;
[0102] Step 7-3: Use the centroid of each cube to represent each voxel, i.e., voxel point. For each voxel point, set a number of reconstructed images m, with an initial value of 0.
[0103] Step 7-4: Project each voxel point onto each image in sequence, and determine whether it belongs to the foreground or the background in the current image. If it belongs to the foreground, then m+1.
[0104] Step 7-5: Use CUDA parallel computing to make each thread correspond one-to-one with a voxel. The correspondence is represented by the thread index until all views have been traversed.
[0105] Step 7-6: Set a reconstructed image threshold n, and judge m. If m≥n, the voxel point is considered a plant point and is retained; otherwise, it is a background point and is discarded.
[0106] As a specific implementation, when performing back-projection coloring, based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, the three-dimensional points corresponding to the plant area, namely the shell points, are determined for each image pixel by back-projection ray tracing. The pixel color is assigned to its corresponding shell point, all colored shell points are merged, and overlapping points are removed by calculating their color mean, finally obtaining a point cloud model with color information.
[0107] The back-projection coloring is specifically implemented through the following steps:
[0108] Step 8-1: Read the original image, binary image, and corresponding camera parameters used for color rendering. Starting from N views, use the `undistort` function to correct distortion in the images and save them.
[0109] Step 8-2: Traverse the foreground points (2D pixels) of the plant area in the current view, and use GPU optimization for these foreground points. Each kernel function performs back projection calculation on each foreground point to determine its corresponding voxel point; obtain the shell model corresponding to each view, a total of N;
[0110] Step 8-3: For the retained shell points, sequentially obtain the RGB information of the corresponding pixel points in each image and assign it to the corresponding voxel points, and finally output the shell model containing color information.
[0111] Step 8-4: For the colored shell model, perform point cloud merging, and remove overlapping points by calculating their color mean, finally obtaining the complete reconstructed point cloud model.
[0112] like Figure 2 As shown in the figure, this embodiment also provides a high-throughput 3D reconstruction system for potted plants based on stereo calibration, used to realize a high-throughput 3D reconstruction method for potted plants based on stereo calibration. The system includes the following modules.
[0113] The image acquisition module includes internal parameter image acquisition using a planar checkerboard calibration plate, and the fabrication of a three-dimensional calibration box consisting of four planar checkerboard calibration plates for external parameter calibration of the plant. The potted plant is placed in the center of the calibration box, and the image is acquired by fixing the camera and rotating the plant at a uniform speed, obtaining an image sequence of each plant rotating one revolution.
[0114] The camera calibration module includes obtaining the camera intrinsic parameter matrix and camera distortion coefficients using Zhang Zhengyou's calibration method for intrinsic parameter images; extracting the principal plane (ROI) in the image (i.e., extrinsic parameters) for plant images through marker point recognition; performing corner detection on the ROI to obtain the pixel coordinates of each corner point in each image; automatically obtaining the occurrence frequency of each plane based on the change in the centroid coordinates of the ROI in adjacent images; determining the world coordinates of each corner point based on the corner detection order and the occurrence frequency of each plane; and calculating the extrinsic parameter matrix of each image using the pixel coordinates and world coordinates of each corner point.
[0115] The image segmentation module uses a color threshold to coarsely segment the plant region, performs connected component analysis on the coarse segmentation result to remove noise points in small regions and obtain a fine segmentation result, and performs color rendering on the fine segmentation result to obtain a binary image of the plant region.
[0116] The voxel sculpting and reconstruction module takes a binary image of the plant as input, along with the extrinsic parameters and camera intrinsic parameters corresponding to each view. It then performs reconstruction based on the voxel sculpting method and utilizes CUDA parallel computing to improve reconstruction efficiency, resulting in a colorless point cloud model.
[0117] The back-projection coloring module considers that only the outermost points in the point cloud model have color. Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, the module determines the 3D points (shell points) corresponding to the plant area for each pixel in the image through back-projection ray tracing. The pixel color is assigned to its corresponding shell point. All colored shell points are merged, and overlapping points are removed by calculating their color mean, finally obtaining a point cloud model with color information.
[0118] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A high-throughput 3D reconstruction method for potted plants based on stereo calibration, characterized in that, Includes the following steps: A stereo calibration box is made, and the potted plants are placed in the stereo calibration box. Images are acquired using a camera to obtain an image sequence of each plant rotating one revolution. Obtain the camera intrinsic parameter matrix and perform calibration point identification on the image sequence to obtain the extrinsic parameter matrix for each image; The plant regions in the image sequence are segmented and color-rendered to obtain a binary image of the plant. Based on the plant binary image, camera intrinsic parameter matrix and extrinsic parameter matrix of each image, a voxel sculpting method is used to perform 3D reconstruction to obtain the plant model. The plant model is then back-projected and colored to obtain a colored plant model. The process of identifying calibration points in the image sequence to obtain the extrinsic parameter matrix for each image includes: The extrinsic regions of interest (ROIs) of the image sequence are extracted using a calibration point recognition method. Corner detection is then performed on the extracted ROIs to obtain the pixel coordinates of the corners in each image. Based on the change in the centroid coordinates of the ROIs of adjacent images, the occurrence frequency of each plane is obtained. A preset corner detection order is established, and the world coordinates of each corner are obtained based on the occurrence frequency of each plane and the corner detection order. Finally, the extrinsic parameter matrix of each image is obtained based on the pixel coordinates and world coordinates of each corner. The process of extracting the extrinsic region of interest from the image sequence based on the calibration point recognition method includes: Based on the marker points on the stereo calibration box, several contours of each image are obtained; Based on the contour area, several contours in each image are sorted. Count the number of contours in each image; if the number of contours is less than a preset number, discard the corresponding image. When the number of contours is equal to the preset number, the vertex coordinates of the rectangular region surrounding the plant target are calculated based on the centroid points of the four contours, and the extrinsic region of interest of the image is extracted based on the vertex coordinates of the rectangular region. When the number of contours is greater than the preset number, for all sorted contours, calculate the centroids of the first three contours and the centroids of the remaining contours. Select the centroid of the contour with the smallest column coordinate distance from the centroid of the third contour from the centroids of the remaining contours. Calculate the vertex coordinates of the rectangular region surrounding the plant target together with the centroids of the first three contours. Extract the extrinsic region of interest of the image based on the vertex coordinates of the rectangular region. The process of corner detection in the extracted extrinsic region of interest includes: Corner detection is performed on the extracted extrinsic region of interest to obtain the pixel coordinates of the corners; Divide the corner points along the length of the chessboard into groups, with the number of groups being the same as the number of corner points along the height of the chessboard. Sort the groups according to the y-coordinate of the first corner point in each group from smallest to largest; The corner detection sequence is as follows: starting from the top left corner, proceeding sequentially from left to right and from top to bottom; The process of obtaining the frequency of occurrence of each plane based on the change in the centroid coordinates of the regions of interest in adjacent image extrinsic parameters includes: Preset rotation direction; Obtain the centroid of the region of interest for each image's extrinsic parameters; As the centroid rotates clockwise, by comparing the changes in the centroid column coordinates of the images, the image sequences belonging to the same plane are identified, and the number of images in the sequence is output when the centroid column coordinates begin to increase. Then, the next set of images is processed until all images have been processed. As the centroid rotates counterclockwise, the changes in the centroid coordinates of the image are compared to identify the image sequence belonging to the same plane. When the centroid coordinates begin to decrease, the number of images in that sequence is output, and then the next set of images is processed until all images have been processed.
2. The method according to claim 1, characterized in that, The process of constructing a stereo calibration box, placing the potted plant inside the stereo calibration box, and acquiring images using a camera includes: A three-dimensional calibration box is constructed based on four planar checkerboard calibration boards of the same height as the potted plant, and marker points are set at the intersection vertices of two adjacent planes. The potted plant is placed in the center of the three-dimensional calibration box, and images are acquired by fixing the camera and rotating the potted plant at a constant speed.
3. The method according to claim 1, characterized in that, Based on the binary image of the plant, the camera intrinsic parameter matrix, and the extrinsic parameter matrix of each image, 3D reconstruction is performed using the voxel sculpting method to obtain the plant model. The process includes: The binary images of the plants are converted into one-dimensional data and encoded into decimal numbers for storage. A cube containing the samples to be reconstructed is initialized and divided into several voxels. Each voxel is projected onto each image in sequence, and the number of foreground points is counted to determine whether a voxel belongs to a plant. Voxels that do not belong to a plant are discarded. Finally, CUDA parallel computing technology is used to process all the binary images of the plants to complete the three-dimensional reconstruction of the plants.
4. The method according to claim 1, characterized in that, The process of performing back-projection coloring on the plant model to obtain a colored plant model includes: Based on the camera intrinsic parameters and the extrinsic parameter matrix of each image, back-projection ray tracing is performed on the pixels of each image to determine the shell points corresponding to the plant region. The pixel color is assigned to the corresponding shell point, all colored shell points are merged, and overlapping points are removed by calculating the color mean, finally obtaining the colored plant model.
5. A high-throughput 3D reconstruction system for potted plants based on stereo calibration, characterized in that, The method for implementing the high-throughput 3D reconstruction of potted plants based on stereo calibration as described in any one of claims 1-4 includes: The image acquisition module is used to create a stereo calibration box. The potted plants are placed in the stereo calibration box and images are acquired by a camera to obtain an image sequence of each plant rotating one revolution. The camera calibration module is used to obtain the camera intrinsic parameter matrix and to perform calibration point identification on the image sequence to obtain the extrinsic parameter matrix of each image. The image segmentation module is used to segment and color render the plant region in the image sequence to obtain a binary image of the plant. The voxel sculpting and reconstruction module is used to perform 3D reconstruction based on the plant binary image, camera intrinsic parameter matrix and extrinsic parameter matrix of each image, using the voxel sculpting method to obtain the plant model. The back-projection coloring module is used to perform back-projection coloring on the plant model, and finally obtain a colored plant model.