A plant three-dimensional reconstruction method and related device

Abstract

This application discloses a method and related apparatus for 3D reconstruction of plants, relating to the field of computer vision technology. The method performs multiple iterations to update the Gaussian ellipsoid set. In each iteration, when rendering a color image from each viewpoint, the depth values ​​of each key point are obtained based on the depth image, and the depth weight of the target Gaussian ellipsoid is obtained based on the depth values. The depth weight is inversely correlated with the depth value, that is, it increases the contribution of Gaussian ellipsoids with smaller depth values ​​to color rendering. By introducing depth weights, the depth information distinction of subtle regions is enhanced, and the contribution of Gaussian ellipsoids at different depth levels to color rendering is precisely controlled, thereby improving the geometric accuracy and sense of hierarchy of the 3D reconstruction of plants, and further improving the restoration accuracy of the 3D reconstruction model of plants.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and related apparatus for three-dimensional reconstruction of plants. Background Technology

[0002] Plant 3D modeling transforms the morphology, structure, and growth characteristics of plants into a 3D reconstruction model for analysis, monitoring, and management. Accurate plant 3D modeling can optimize planting management, improve production efficiency, and thus promote the development of smart agriculture, achieving automation and precision in agricultural management.

[0003] 3D Gaussian sputtering technology has powerful scene expression capabilities and supports fast rendering. Therefore, it can be used to model plants in 3D. However, when using traditional 3D Gaussian sputtering technology to model plants in 3D, there is a general drawback of poor rendering effect, resulting in a low degree of restoration of the plant in the 3D reconstruction model. Summary of the Invention

[0004] In view of the above problems, this application provides a method and related apparatus for three-dimensional reconstruction of plants, so as to improve the accuracy of the three-dimensional reconstruction model in reproducing the plant. The specific solution is as follows:

[0005] The first aspect of this application provides a method for three-dimensional reconstruction of plants.

[0006] Three-dimensional point cloud data is obtained based on the target color image sequence of the target plant; wherein, the target color image sequence includes multiple color images from different perspectives arranged in chronological order, and the three-dimensional point cloud data includes the three-dimensional coordinate values ​​and color values ​​of multiple key points;

[0007] The Gaussian ellipsoids of each key point are initialized based on the three-dimensional point cloud data to obtain a set of Gaussian ellipsoids. The set of Gaussian ellipsoids includes the Gaussian ellipsoids of each key point. The Gaussian parameters of the Gaussian ellipsoids include the mean vector, covariance matrix, color value, and opacity.

[0008] The iteration is performed to update the Gaussian ellipsoid set until a preset iteration termination condition is met, resulting in the updated Gaussian ellipsoid set.

[0009] Based on the updated Gaussian ellipsoid set, a three-dimensional reconstruction model of the target plant is constructed;

[0010] The iteration includes:

[0011] One color image is selected from the target color image sequence as the training color image for the iteration;

[0012] For each pixel in the training color image, the depth value of each target Gaussian ellipsoid is obtained based on the depth image corresponding to each target Gaussian ellipsoid of the pixel; the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids;

[0013] The depth weights of each of the target Gaussian ellipsoids are calculated based on at least the depth values; wherein the depth weights are inversely correlated with the depth values.

[0014] For each pixel, the rendering color value of the pixel is calculated based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid;

[0015] Based on the rendering color values ​​of each pixel in the training color image, a rendered image of the target color image is obtained.

[0016] The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

[0017] In one possible implementation, three-dimensional point cloud data is obtained based on a target color image sequence of the target plant, including:

[0018] The original color image sequence of the target plant is acquired by a depth camera, and the original color image sequence is filtered based on a preset image quality index to obtain the target color image sequence. The image quality index includes the blur detection probability.

[0019] For each color image in the target color image sequence, the scale-invariant feature transform matching algorithm is used to extract the key points of the color image and the descriptors of the key points;

[0020] Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs for every two color images;

[0021] Based on multiple key point pairs from every two color images, three-dimensional point cloud data of the target plant is generated.

[0022] In one possible implementation, generating the 3D point cloud data of the target plant based on multiple pairs of keypoints from every two color images includes:

[0023] Starting with a time sequence n of 2, traverse the color images in the target color image sequence from smallest to largest:

[0024] When n=2, the fundamental matrix is ​​calculated using the eight-point algorithm based on the pixel coordinates of the key point pairs of the second color image and the first color image. The fundamental matrix is ​​then decomposed to obtain the extrinsic matrix of the second color image, which includes a rotation matrix and a translation vector.

[0025] Using triangulation, the three-dimensional coordinates of each key point in the second color image are calculated based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters, and the extrinsic parameter matrix of the second color image.

[0026] When n > 2, the external parameter matrix of the nth color image is calculated using the perspective n-point PnP algorithm based on the key point pairs of the nth color image and the (n-1)th color image.

[0027] Using triangulation, the three-dimensional coordinates of each key point in the nth color image are calculated based on the pixel coordinates of each key point pair in the nth color image and the (n-1)th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the (n-1)th color image, and the extrinsic parameter matrix of the nth color image.

[0028] The three-dimensional point cloud data is generated based on the three-dimensional coordinates and color values ​​of each key point in each color image.

[0029] In one possible implementation, the depth weights of each of the target Gaussian ellipsoids are calculated based at least on the depth values, including:

[0030] The depth intervals are divided into multiple depth intervals according to the minimum and maximum depth values, resulting in a depth interval sequence. The depth interval sequence includes multiple depth intervals arranged in ascending order of depth value.

[0031] For each depth interval, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order of depth values ​​to obtain the ellipsoid subsequence corresponding to the depth interval.

[0032] For each depth interval, the depth center of the depth interval is calculated based on the position of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value.

[0033] For each target Gaussian ellipsoid within the ellipsoidal subsequence of each depth interval, the depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0034] In one possible implementation, for each pixel, the rendered color value of the pixel is calculated based on the depth weights and Gaussian parameters of each of the target Gaussian ellipsoids, including:

[0035] Project each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtain the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each target Gaussian ellipsoid based on the Gaussian parameters of the Gaussian ellipsoid.

[0036] The opacity of the two-dimensional Gaussian ellipse is calculated based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse.

[0037] For each depth interval, the rendering color value of the depth interval is calculated based on the color value of the target Gaussian ellipsoid, the depth weight, and the opacity of the two-dimensional Gaussian ellipse within the depth interval.

[0038] The rendering color value of the pixel is calculated based on the rendering color values ​​of all depth ranges of the pixel.

[0039] In one possible implementation, the rendered color value for the target depth interval is calculated based on the color value, depth weight, and opacity of the target Gaussian ellipsoid within the target depth interval. The target depth interval can be any depth interval, including:

[0040] For a target Gaussian ellipsoid within the target depth interval, the weighted parameters of each preceding Gaussian ellipsoid of the target Gaussian ellipsoid are multiplied to obtain a cumulative weighted parameter. The preceding Gaussian ellipsoid is a Gaussian ellipsoid whose rank in the subsequence of ellipsoids in the target depth interval is lower than that of the target Gaussian ellipsoid. The cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the preceding Gaussian ellipsoid. The weighted two-dimensional opacity is equal to the depth weight of the preceding Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse.

[0041] For a target Gaussian ellipsoid within the target depth range, calculate the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain the opacity parameter;

[0042] For a target Gaussian ellipsoid within the target depth range, calculate the product of the color value and the opacity parameter of the target Gaussian ellipsoid, and use it as the color to be rendered;

[0043] Using the depth weight of each of the target Gaussian ellipsoids as the weighting coefficient, the colors to be rendered for each of the target Gaussian ellipsoids are weighted and summed to obtain the rendering color value of the target depth range.

[0044] A second aspect of this application provides a three-dimensional reconstruction device for plants, comprising:

[0045] A point cloud data construction unit is used to acquire three-dimensional point cloud data based on a target color image sequence of a target plant; wherein, the target color image sequence includes multiple color images from different perspectives arranged in chronological order, and the three-dimensional point cloud data includes the three-dimensional coordinate values ​​and color values ​​of multiple key points.

[0046] A Gaussian initialization unit is used to initialize the Gaussian ellipsoids of each key point based on the three-dimensional point cloud data to obtain a set of Gaussian ellipsoids. The set of Gaussian ellipsoids includes the Gaussian ellipsoids of each key point. The Gaussian parameters of the Gaussian ellipsoids include the mean vector, covariance matrix, color value, and opacity.

[0047] A Gaussian iterative update unit is used to construct a three-dimensional reconstruction model of the target plant based on the updated Gaussian ellipsoid set.

[0048] A 3D modeling unit is used to update the Gaussian ellipsoid set based on the loss values ​​of the training color image and the rendered image.

[0049] Specifically, the Gaussian update unit is used to perform the iteration as follows:

[0050] One color image is selected from the target color image sequence as the training color image for the iteration;

[0051] For each pixel in the training color image, the depth value of each target Gaussian ellipsoid is obtained based on the depth image corresponding to each target Gaussian ellipsoid of the pixel; the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids;

[0052] The depth weights of each of the target Gaussian ellipsoids are calculated based on at least the depth values; wherein the depth weights are inversely correlated with the depth values.

[0053] For each pixel, the rendering color value of the pixel is calculated based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid;

[0054] Based on the rendering color values ​​of each pixel in the training color image, a rendered image of the target color image is obtained.

[0055] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the plant three-dimensional reconstruction method described in the first aspect or any implementation thereof.

[0056] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0057] The memory is used to store computer programs;

[0058] The processor is used to execute the computer program so that the electronic device can implement the plant three-dimensional reconstruction method of the first aspect or any implementation thereof.

[0059] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the plant three-dimensional reconstruction method described in the first aspect or any implementation thereof.

[0060] Using the above technical solution, the plant 3D reconstruction method and related apparatus provided in this application perform multiple iterations to update the Gaussian ellipsoid set. In each iteration, when rendering the color image under each viewpoint, the depth value of each key point is obtained based on the depth image, and the depth weight of the target Gaussian ellipsoid is obtained based on the depth value. The depth weight is inversely correlated with the depth value, that is, the contribution of the Gaussian ellipsoid with a smaller depth value to color rendering is increased. By introducing the depth weight, the depth information distinction of the fine area is enhanced, and the contribution of Gaussian ellipsoids at different depth levels to the rendering color is precisely controlled, thereby improving the geometric accuracy and sense of hierarchy of the plant 3D reconstruction, and further improving the restoration degree of the plant by the 3D reconstruction model. Attached Figure Description

[0061] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0062] Figure 1 A flowchart illustrating a method for three-dimensional reconstruction of a plant provided in an embodiment of this application;

[0063] Figure 2 A flowchart illustrating the specific implementation of a plant three-dimensional reconstruction method provided in this application embodiment;

[0064] Figure 3 A schematic diagram of the structure of a plant three-dimensional reconstruction device provided in an embodiment of this application;

[0065] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0067] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0068] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0069] This application can be applied to the field of computer vision technology, specifically to the field of plant research and management. It involves creating 3D digital models of agricultural or economic crops in a production environment to monitor their morphology, structure, and growth characteristics at various stages of their growth cycle. Taking tomato plants as an example, a tomato plant refers to the entire plant, including roots, stems, and leaves. As a common agricultural vegetable, tomatoes are highly nutritious, have high yields, and are adaptable to various environments, making them an indispensable vegetable in the daily lives of Chinese people. The 3D modeling method for tomato plants provided in this application can be specifically applied to 3D modeling of tomato plants. By improving the rendering effect when using 3D Gaussian sputtering for 3D modeling of tomato plants, the accuracy of the 3D reconstruction model of the tomato plant is improved. Furthermore, this provides accurate data for precise monitoring and management of tomato plant growth, thereby improving the level of agricultural automation.

[0070] The plant 3D reconstruction method provided in this application embodiment can be applied to a 3D reconstruction system. The 3D reconstruction system includes a 3D reconstruction device and a depth camera. The depth camera can be an Intel RealSense D415 depth camera, which can be a handheld depth camera or installed in the plant's growth environment. The installation location and method can be determined according to the actual environment. The 3D reconstruction device includes a terminal device and / or a server with 3D reconstruction capabilities. The terminal device can be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc. The server can be a single-point server or a server cluster, which is not limited in this embodiment.

[0071] In the rendering process of 3D Gaussian sputtering technology, Gaussian ellipsoids at multiple key points may contribute color information to the same two-dimensional pixel. If depth information is not considered, this may lead to blurring or ghosting in the rendering results. Since depth images from depth cameras can provide high-precision depth information, this application introduces depth weighting to enhance the differentiation of depth information in subtle regions. Even when depth values ​​are similar, the contribution of Gaussian ellipsoids at different depth levels can be more precisely controlled, improving the geometric accuracy and sense of hierarchy in the 3D reconstruction of the plant, thereby improving the fidelity of the 3D reconstruction model to the plant.

[0072] The method for three-dimensional reconstruction of plants according to embodiments of this application will be described in detail below with reference to the accompanying drawings. Figure 1 A flowchart illustrating a method for three-dimensional reconstruction of a plant provided in an embodiment of this application is shown below. Figure 1 As shown, this method includes:

[0073] S11. Obtain three-dimensional point cloud data based on the target plant's target color image sequence.

[0074] In this embodiment, the three-dimensional point cloud data includes the three-dimensional coordinates and color values ​​of multiple key points, wherein the key points are extracted from the color image.

[0075] In this embodiment, the target color image sequence includes multiple color images from different perspectives arranged in chronological order.

[0076] In one optional embodiment, color images and depth images of the target plant are acquired from multiple consecutive viewpoints using a depth camera to obtain raw image data. After preprocessing the raw image data, such as image enhancement, denoising, and sampling, a target color image sequence is obtained.

[0077] S12. Initialize the Gaussian ellipsoids of each key point based on the 3D point cloud data to obtain the Gaussian ellipsoid set.

[0078] In this embodiment, the Gaussian ellipsoid set includes Gaussian ellipsoids of each key point, and the Gaussian parameters of the Gaussian ellipsoids include mean vector, covariance matrix, color value and opacity.

[0079] In this embodiment, the initialized Gaussian ellipsoid set includes the initial values ​​of each Gaussian parameter of the Gaussian ellipsoid at each key point.

[0080] S13. Perform iterations to update the Gaussian ellipsoid set until the preset iteration termination condition is met, and obtain the updated Gaussian ellipsoid set.

[0081] In this embodiment, the iteration termination condition includes reaching a preset iteration number threshold. For example, N is the preset iteration number threshold. When the iteration number n=N, the iteration stops and the updated Gaussian ellipsoid set is output.

[0082] In this embodiment, any iteration includes:

[0083] S131. Select one color image from the target color image sequence as the training color image for iteration.

[0084] In this embodiment, each iteration randomly selects a color image from the target color image sequence or selects it in a preset order as the training color image for the iteration, and N is configured to be a value much larger than the number of color images.

[0085] S132. For each pixel in the training color image, obtain the depth value of each target Gaussian ellipsoid based on the depth image corresponding to each target Gaussian ellipsoid of the pixel.

[0086] In this embodiment, the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids, that is, the Gaussian ellipsoid contained under the view frustum of the pixel.

[0087] S133. Calculate the depth weights of each target Gaussian ellipsoid based at least on the depth values.

[0088] In this embodiment, the depth weight is inversely correlated with the depth value; that is, the closer the depth value, the greater the depth weight.

[0089] S134. For each pixel, calculate the rendering color value of the pixel based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid.

[0090] S135. Based on the rendering color values ​​of each pixel in the training color image, render the target color image to obtain the rendered image.

[0091] S136. Update the Gaussian ellipsoid set based on the loss values ​​of the training color image and the rendered image.

[0092] S14. Based on the updated Gaussian ellipsoid set, construct a three-dimensional reconstruction model of the target plant.

[0093] As can be seen from the above technical solution, the plant 3D reconstruction method provided in this application performs multiple iterations to update the Gaussian ellipsoid set. In each iteration, when rendering the color image under each viewpoint, the depth value of each key point is obtained based on the depth image, and the depth weight of the target Gaussian ellipsoid is obtained based on the depth value. The depth weight is inversely correlated with the depth value, that is, it increases the contribution of the Gaussian ellipsoid with a smaller depth value to the color rendering. By introducing the depth weight, the depth information distinction of the fine area is enhanced, and the contribution of the Gaussian ellipsoid at different depth levels to the rendering color is precisely controlled, thereby improving the geometric accuracy and sense of hierarchy of the plant 3D reconstruction, and further improving the restoration degree of the plant by the 3D reconstruction model.

[0094] See Figure 2 , Figure 2 A flowchart illustrating the specific implementation of a plant three-dimensional reconstruction method provided in this application embodiment is shown below. Figure 2 As shown, this method specifically includes steps S201 to 217, as follows:

[0095] S201. Obtain the original image sequence of the target plant using a depth camera, and filter the original image sequence based on preset image quality indicators to obtain the target image sequence.

[0096] In this embodiment, the original image sequence includes an original color image sequence and an original depth image sequence, and the target image sequence includes a target color image sequence and a target depth image sequence.

[0097] In this embodiment, the original color image sequence includes color images arranged in chronological order, and the original depth image sequence includes depth images arranged in chronological order. Here, "chronological order" refers to the sequence of shooting times.

[0098] Specifically, when the time for 3D reconstruction is reached, the depth camera is controlled to capture video streams around the plant at a preset resolution, frame rate, shooting duration, and movement trajectory. Image frames are then collected from the video stream at a preset sampling frequency to obtain the original color image sequence and the original depth image sequence.

[0099] Taking a tomato plant as the target plant as an example, with a resolution of 1280 × 720, a frame rate of 30, a shooting duration of 1 minute, a 360-degree surround motion trajectory, and a sampling frequency of 5 frames, when the preset reconstruction cycle start time is reached, the Intel Realsense D415 depth camera is controlled to shoot around the plant for 1 minute at a resolution of 1280 × 720 and a frame rate of 30, acquiring a video stream containing 1800 image frames. Using a sampling frequency of selecting 1 frame every 5 frames, 360 depth images and color images are sampled from the video stream. The depth images are arranged in time sequence to obtain the original depth image sequence, and the color images are arranged in time sequence to obtain the original color image sequence.

[0100] It should be noted that the original depth images and original color images in the original image set are ordered according to the chronological order of the shooting time. Based on the continuity of image frames, the calculation speed of subsequent image matching can be accelerated.

[0101] In this embodiment, the image quality index includes the Probability of Blur Detection (CPBD) value. The CPBD value is an image quality index that conforms to human visual characteristics; a higher value reflects clearer image details. Specifically, the CPBD value of each color image in the original color image sequence is calculated. Color images with a CPBD value lower than a preset threshold are removed, and color images with a CPBD value not lower than the threshold are retained to obtain the target color image sequence. Further, the depth images corresponding to the color images with a CPBD value lower than the threshold in the original depth image sequence are deleted to obtain the target depth image sequence.

[0102] Continuing from the previous example, calculate the CPBD values ​​of 360 color images of the tomato plant, retain 250 color images with CPBD values ​​not lower than the CPBD threshold and their corresponding depth images, thus obtaining a target color image sequence including color images from 250 viewpoints and a target depth image sequence including depth images from 250 viewpoints.

[0103] This step improves the image quality of the plant color images by filtering the original image sequence based on CPBD values, thereby helping to improve the 3D reconstruction effect.

[0104] S202. For each color image in the target color image sequence, use the scale-invariant feature transform matching algorithm to extract the key points of the color images and the descriptors of the key points.

[0105] In this embodiment, the SIFT (Scale Invariant Feature Transform) algorithm is used to extract key points from each color image, and a SIFT descriptor is generated for each key point. The SIFT descriptor describes the features of the key point. Specifically, the SIFT descriptor can be a feature vector.

[0106] S203. Based on the descriptor, perform keypoint matching on every two color images in the target color image sequence to obtain multiple keypoint pairs for every two color images.

[0107] In this embodiment, the keypoint pair of the first color image and the second color image includes keypoints of the first target color image and the second target color image that satisfy preset matching conditions. The matching conditions include that the Euclidean distance between SIFT descriptors is less than a preset distance threshold. The first target color image and the second target color image can be any two color images.

[0108] S204. Based on multiple key point pairs between every two color images, generate three-dimensional point cloud data of the target plant.

[0109] In this embodiment, the nth color image is denoted as the color image with the order n in the target color image sequence, where n is greater than 1.

[0110] Optional methods for generating 3D point cloud data of the target plant include A1~A3:

[0111] A1. When n=2, based on the pixel coordinates of the keypoint pairs of the second color image and the first color image, the fundamental matrix is ​​calculated using the eight-point algorithm, and the fundamental matrix is ​​decomposed to obtain the extrinsic matrix of the second color image.

[0112] In this embodiment, the first color image and the second color image are the color images with positions 1 and 2 in the target color image sequence, respectively. It should be noted that the fundamental matrix describes the epipolar geometric relationship between the first and second color images. The extrinsic parameter matrix is ​​a combination of rotation matrices and translation vectors, used to transform points in the world coordinate system to the camera coordinate system. The rotation matrix is ​​the rotation component of the camera pose, used to describe the camera's orientation. The translation vector is the translation component of the camera pose, used to describe the camera's position.

[0113] Specifically, let p1 be the pixel coordinates of keypoint Q1 in the first color image I1, and p2 be the pixel coordinates of keypoint Q2 in the second color image I2 that matches keypoint Q1.

[0114] Based on the epipolar geometry relationship between I1 and I2, at least 8 keypoint pairs are used. The pixel coordinates of each keypoint pair are substituted into the first function to calculate the fundamental matrix. The first function is as shown in formula (1):

[0115] (1);

[0116] It should be noted that the fundamental matrix F has 7 degrees of freedom and requires at least 8 keypoint pairs that match features. If there are 8 keypoint pairs, a unique solution is obtained. If there are more than 8 keypoint pairs, a least squares solution is obtained. An approximate solution is found by minimizing the sum of squared errors.

[0117] Furthermore, the fundamental matrix is ​​decomposed to obtain the extrinsic matrix, which includes the rotation matrix R and the translation vector t. The decomposition method of the extrinsic matrix is ​​shown in formula (2):

[0118] (2).

[0119] A2. Using triangulation, calculate the three-dimensional coordinates of each key point in the second color image based on the pixel coordinates, camera intrinsic and extrinsic parameter matrices, and other key points in the second and first color images.

[0120] In this embodiment, the extrinsic parameter matrix includes a rotation matrix and a translation vector.

[0121] Taking point P as an example, in an optional embodiment, the three-dimensional coordinates of key point P in the second color image are calculated using a linear method as shown in formulas (3) and (4):

[0122] (3);

[0123] (4);

[0124] In this context, p1, p2, K, R, and t are all known. p1 and p2 are the pixel (homogeneous) coordinates of the keypoints in the keypoint pair, K is the camera intrinsic parameter, and R and t are the rotation matrix and translation vector of the camera viewpoint of the second image, respectively. P is the three-dimensional (homogeneous) coordinate in the world coordinate system.

[0125] A3. When n > 2, traverse the color images in time order, and use the PnP algorithm to calculate the extrinsic parameter matrix of the nth color image based on the key point pairs of the nth color image and the (n-1)th color image.

[0126] In this embodiment, the PnP (Perspective-n-Point) algorithm is a method for solving the motion of 3D to 2D point pairs. Specifically, based on the coordinate data of keypoint pairs in the nth color image and the (n-1)th color image, the camera pose corresponding to the nth color image, i.e., the extrinsic parameter matrix, is estimated.

[0127] In this embodiment, the coordinate data of keypoint pairs in the nth color image and the (n-1)th color image are obtained. The coordinate data of the keypoint pairs include the three-dimensional coordinate values ​​of the keypoints in the (n-1)th color image and the pixel coordinate values ​​of the nth color image. The PnP algorithm is used to solve the camera pose corresponding to the nth color image.

[0128] It should be noted that epipolar geometry methods require at least 8 keypoint pairs and have issues with initialization, pure rotation, and scaling. The PnP algorithm does not require epipolar constraints and only needs 3 keypoint pairs to achieve camera pose estimation.

[0129] A4. Using triangulation, based on the pixel coordinates of each keypoint pair in the nth color image and the (n-1)th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the (n-1)th color image, and the extrinsic parameter matrix of the nth color image, calculate the three-dimensional coordinates of each keypoint in the nth color image.

[0130] In this embodiment, the method for calculating the three-dimensional coordinates of the key point P using triangulation is described in formulas (5) and (6):

[0131] p1 = K[R(n-1) t(n-1)]P (5);

[0132] p2 = K[Rn tn]P (6;

[0133] In formulas (5) and (6), n is greater than 2, and R(n-1) and t(n-1) are the rotation matrix and translation vector of the camera viewpoint of the (n-1)th image. Rn and tn are the rotation matrix and translation vector of the camera viewpoint of the nth color image in the extrinsic parameter matrix.

[0134] It should be noted that by sequentially traversing all the color images, the three-dimensional coordinates of each key point in each color image and the extrinsic parameter matrix under the corresponding viewpoint are obtained.

[0135] A5. Generate 3D point cloud data based on the 3D coordinates of each key point in each color image.

[0136] In this embodiment, the 3D point cloud data includes the 3D coordinates and color values ​​of each key point.

[0137] In summary, starting with a time sequence of n=2, the color images in the target color image sequence are traversed from smallest to largest: When n=2, based on the pixel coordinates of the keypoint pairs in the second and first color images, the fundamental matrix is ​​calculated using the eight-point algorithm. The fundamental matrix is ​​then decomposed to obtain the extrinsic parameter matrix of the second color image, which includes rotation and translation vectors. Using triangulation, the 3D coordinates of each keypoint in the second color image are calculated based on the pixel coordinates of each keypoint pair in the second and first color images, camera intrinsics, and the extrinsic parameter matrix of the second color image. When n>2, based on the keypoint pairs in the nth and (n-1th)th color images, the extrinsic parameter matrix of the nth color image is calculated using the perspective n-point PnP algorithm. Using triangulation, the 3D coordinates of each keypoint in the nth color image are calculated based on the pixel coordinates of each keypoint pair in the nth and (n-1th)th color images, camera intrinsics, the extrinsic parameter matrix of the (n-1th)th color image, and the extrinsic parameter matrix of the nth color image. Based on the 3D coordinates and color values ​​of each keypoint in each color image, 3D point cloud data is generated.

[0138] S205. Initialize the Gaussian ellipsoids of each key point based on the 3D point cloud data to obtain the initial Gaussian ellipsoid set.

[0139] In this embodiment, the Gaussian ellipsoid set includes Gaussian ellipsoids for each key point. A Gaussian ellipsoid is a Gaussian-parameterized 3D object used to describe the positional and shape information of the key points.

[0140] Specifically, the Gaussian parameters of the Gaussian ellipsoid include the mean vector, covariance matrix, color value, and opacity. The color value of the 3D Gaussian ellipsoid is represented using a continuous spherical harmonic function, and the opacity is represented using a learnable and optimizeable point cloud opacity α. The position and shape of the Gaussian ellipsoid are represented by the mean vector μ and the covariance matrix Σ. The 3D spatial position of the Gaussian ellipsoid is described by the mean vector, and for each Gaussian ellipsoid, its mean represents the coordinates of the center of the Gaussian ellipsoid in 3D space, while the shape is described by the covariance matrix. The mean of the Gaussian ellipsoid for keypoints is the 3D coordinates of the keypoint, and the covariance of the Gaussian ellipsoid is calculated using the n nearest neighbors of the keypoint.

[0141] Alternatively, a 3D Gaussian ellipsoid can be represented using a 3D Gaussian function G(x) that includes position and shape information, as shown in Equation (7):

[0142] (7);

[0143] The location information is x=(x1, x2, x3), which is the three-dimensional coordinate of a point in the three-dimensional point cloud.

[0144] Based on formula (7), the 3D Gaussian function is decomposed into two components: the rotation matrix R and the scaling matrix S, as shown in formula (8):

[0145] (8);

[0146] Equation (8) expresses the geometric meaning of first rotating the 3D Gaussian ellipsoid to be aligned with the ellipsoid world, then shrinking it along the axis, and then rotating it back. This decomposition method can both guarantee the positive semidefinite property of the covariance matrix and reduce the difficulty of matrix operations.

[0147] It should be noted that S202~S205 are specific methods for initializing the Gaussian ellipsoid set based on SFM (Structure from Motion) technology, which involves four processes: feature extraction and matching, epipolar constraint solution for camera initial pose, PnP estimation for camera pose, and triangulation to solve for the three-dimensional coordinates of key points.

[0148] Furthermore, using the Gaussian ellipsoid set obtained in this step as the initial Gaussian ellipsoid set, iteratively execute S206~S219 to obtain the updated Gaussian ellipsoid set, which is also the target Gaussian ellipsoid set.

[0149] S206. Randomly select a training color image from the target color image sequence. For each pixel in the training color image, obtain multiple target Gaussian ellipsoids for the pixel from the Gaussian ellipsoid set.

[0150] In this embodiment, for a single pixel, the Gaussian ellipsoid covering that pixel is used as the target Gaussian ellipsoid. For example, for a target pixel in a training color image, M target Gaussian ellipsoids for the target pixel are obtained from the Gaussian ellipsoid set. The specific projection process in this embodiment includes:

[0151] Given the camera pose in one frame 3D points in the world coordinate system Transform to the camera coordinate system, and use affine transformation to convert the 3D Gaussian coordinates. Projected onto a 2D image plane to form a 2D Gaussian The formulas (9) and (10) are as follows:

[0152] (9);

[0153] (10);

[0154] in, The approximate Jacobian matrix representing the projective transformation. This represents the rotation matrix from the world coordinate system to the camera coordinate system.

[0155] S207. For each target Gaussian ellipsoid, obtain the depth value of the target Gaussian ellipsoid based on the target depth image.

[0156] In this embodiment, the target depth image is the depth image of the key points represented by the target Gaussian ellipsoid. The target depth image is obtained from the target depth image sequence, and the depth information of the key points represented by the target Gaussian ellipsoid is obtained based on the target depth image. The depth information includes depth values. The target Gaussian ellipsoids are sorted according to their depth values ​​to obtain a target Gaussian ellipsoid sequence. The target Gaussian ellipsoid sequence is denoted as {G1~GM}. It should be noted that, according to the basic principles of imaging, objects closer to the camera will cause occlusion of objects farther from the camera. S208: Multiple depth intervals are divided according to the minimum and maximum depth values ​​to obtain a depth interval sequence.

[0157] In this embodiment, the depth interval sequence includes depth intervals arranged in ascending order of boundary values, with the minimum depth value being the minimum depth value of the target Gaussian ellipsoid and the maximum depth value being the maximum depth value of the target Gaussian ellipsoid.

[0158] In this embodiment, depth intervals can be divided according to a preset interval length or a preset number of intervals. For example, the minimum depth value of the M target Gaussian ellipsoids of the target pixel is dmin, and the maximum depth value is dmax. K depth intervals are obtained by uniformly dividing the range [dmin, dmax]. The k-th (k∈[1,K]) depth interval is denoted as [dk1, dk2), that is, dk1 is the minimum boundary value of the k-th depth interval, dk2 is the maximum boundary value of the k-th depth interval, d11 is dmin, and dk2 is dmax. The K depth intervals are sorted in ascending order of minimum boundary value to obtain the depth interval sequence {[dk1, dk2)}.

[0159] S209. For each depth interval, sort the target Gaussian ellipsoids with depth values ​​within the depth interval in ascending order of depth value to obtain the ellipsoid subsequence corresponding to the depth interval.

[0160] In this embodiment, the number of target Gaussian ellipsoids in the ellipsoidal subsequence Sk corresponding to the kth depth interval is denoted as I, where the i-th (i∈[1,I]) target Gaussian ellipsoid is the target Gaussian ellipsoid with the order i.

[0161] S210. For each depth interval, calculate the depth center of the depth interval based on its position in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value.

[0162] In this embodiment, the difference between the maximum depth value and the minimum depth value is calculated to obtain the total depth difference. The total depth difference is divided by the number of depth intervals to obtain the interval range value of the depth interval. The interval range value is multiplied by the product of the order of the depth interval in the depth interval sequence, and then the minimum depth value is added to obtain the depth center of the depth interval.

[0163] Taking the calculation of the depth center of the k-th depth interval as an example, the depth center of the k-th depth interval... The calculation formula is formula (11):

[0164] (11).

[0165] S211. For the target Gaussian ellipsoid within the ellipsoidal subsequence of each depth interval, calculate the depth weight of the target Gaussian ellipsoid based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0166] In this embodiment, the depth weight of the target Gaussian ellipsoid is inversely correlated with the depth value.

[0167] In this embodiment, the depth weight of the i-th target Gaussian ellipsoid within the ellipsoidal subsequence of the k-th depth interval is used to calculate the depth weight. See formula (12) for the formula:

[0168] (12);

[0169] In the formula, This represents the preset weight adjustment parameter corresponding to the k-th depth interval, used to control the weight distribution in that interval. This represents the depth value of the i-th target Gaussian ellipsoid within the ellipsoidal subsequence of the k-th depth interval.

[0170] S212. Project each target Gaussian ellipsoid onto the two-dimensional plane of the training image to obtain the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each target Gaussian ellipsoid.

[0171] In this embodiment, the Gaussian function of the target Gaussian ellipsoid of the key point is expressed as:

[0172] .

[0173] Correspondingly, the mean vector and covariance matrix are represented by Gaussian functions of a two-dimensional Gaussian ellipse as follows:

[0174] ;

[0175] Where μ and Σ represent the mean vector and covariance matrix of the Gaussian ellipsoid, respectively. and Let represent the mean vector and covariance matrix of the two-dimensional Gaussian ellipse after projection onto the Gaussian ellipsoid, respectively.

[0176] S213. For each depth interval, calculate the rendering color value of the depth interval based on the depth weight, color value, opacity of the target Gaussian ellipsoid within the ellipsoidal subsequence of the depth interval, as well as the mean vector and covariance matrix of the two-dimensional Gaussian ellipsoid.

[0177] In this embodiment, for a target Gaussian ellipsoid within the target depth range, the weighted parameters of each preceding Gaussian ellipsoid of the target Gaussian ellipsoid are multiplied to obtain the cumulative weighted parameter. The preceding Gaussian ellipsoid is the Gaussian ellipsoid whose order in the subsequence of ellipsoids within the target depth range is less than that of the target Gaussian ellipsoid. The cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the preceding Gaussian ellipsoid. The weighted two-dimensional opacity is equal to the depth weight of the preceding Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse.

[0178] For a target Gaussian ellipsoid within the target depth range, calculate the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain the opacity parameter.

[0179] For a target Gaussian ellipsoid within the target depth range, calculate the product of the target Gaussian ellipsoid's color value and opacity parameter, and use it as the color to be rendered.

[0180] Using the depth weights of each target Gaussian ellipsoid as weighting coefficients, the colors to be rendered for each target Gaussian ellipsoid are weighted and summed to obtain the rendered color values ​​for the target depth interval. In this embodiment, taking the k-th depth interval as an example, the rendered color values ​​for the k-th depth interval are... The calculation formula is shown in formula (13):

[0181] (13);

[0182] In the formula, The first pixel in the target Gaussian ellipsoid sequence represents the first pixel. A target Gaussian ellipsoid G color value, The target Gaussian ellipsoid G represents Depth weights The target Gaussian ellipsoid G represents The opacity (also representing the opacity of the two-dimensional Gaussian ellipse after projection). This represents the target opacity. The calculation method for target opacity is shown in formula (14). The depth weights of the target Gaussian ellipsoid Gj (j ≤ i) are represented. This represents the opacity of the target Gaussian ellipsoid Gj.

[0183] (14);

[0184] in, The Gaussian function representing a two-dimensional Gaussian ellipse.

[0185] It should be noted that for a Gaussian ellipsoid or a two-dimensional Gaussian ellipse, the closer to the center, the greater the opacity; the farther away from the center, the smaller the opacity. S214. For each pixel, calculate the pixel's rendered color value based on the rendered color values ​​for each depth interval.

[0186] S215. Based on the rendering color values ​​of each pixel in the nth color image, render the rendered image of the training color image.

[0187] S216. Based on the loss values ​​of the rendered image and the color image, update the Gaussian ellipsoid set using the backpropagation algorithm.

[0188] S217. Determine whether the preset iteration termination condition has been met. If yes, use the updated Gaussian ellipsoid set as the target Gaussian ellipsoid set. If not, execute the next iteration.

[0189] In this embodiment, the iteration termination condition includes the iteration count reaching a preset iteration count threshold. For example, N=30000. When the iteration count reaches 30000, the iteration stops. Otherwise, the iteration count is incremented by 1, and the next iteration is executed according to S206~217.

[0190] S218. Based on the target Gaussian ellipsoid set, generate a three-dimensional reconstruction model of the target plant.

[0191] As can be seen from the above technical solution, unlike the traditional method of rendering according to depth value, this solution introduces the depth weight of the Gaussian ellipsoid calculated from the depth image on the basis of traditional 3D Gaussian sputtering technology. The depth weight is inversely correlated with the depth value of the Gaussian ellipsoid, so that key points closer to the depth camera (i.e., key points with smaller depth values) have a greater influence on the rendering color of pixels. Introducing the depth weight into the rendering formula effectively reflects the true distribution of point cloud in three-dimensional space, enhances the differentiation of depth information in fine areas, more accurately controls the contribution of Gaussian points at different depth levels, and ensures that the foreground details are better preserved during rendering, thereby improving the geometric accuracy and sense of hierarchy of the 3D plant reconstruction model and improving the restoration accuracy of the 3D reconstruction model of the plant.

[0192] Furthermore, this solution divides the depth value of the target Gaussian ellipsoid into depth intervals and renders the color of pixels in units of depth intervals. This not only preserves the global depth sorting characteristics but also ensures the contribution of the target Gaussian ellipsoid within each depth interval. In addition, it more accurately reflects the relative position of the target Gaussian ellipsoid in three-dimensional space. By dividing the depth intervals, it better handles regions with similar depths but complex geometric features, ultimately improving the geometric accuracy and visual consistency of the rendering results.

[0193] Furthermore, this solution is based on 3D Gaussian splashing differentiable rendering technology suitable for fusion data of depth and color images. By combining the image information of depth and color images, it enhances the usability of 3D Gaussian splashing technology and its application scenarios in the three-dimensional reconstruction of agricultural plants.

[0194] It should be noted that, Figure 2 The above is only a specific implementation process of a plant three-dimensional reconstruction method provided in this application embodiment. This application can also be implemented by other optional specific implementation processes. For example, S211~S214 is only an optional method for calculating the depth weight of the target Gaussian ellipsoid. In other optional embodiments, the depth weight of the target Gaussian ellipsoid is inversely proportional to the depth value of the target Gaussian ellipsoid, and the quotient of 1 divided by the depth value of the target Gaussian ellipsoid is used as the depth weight of the target Gaussian ellipsoid.

[0195] For example, in an alternative embodiment, a system of equations for the three-dimensional coordinates of the key point P in the second color image can be calculated using a nonlinear method, as follows:

[0196] Define the energy functions as in formulas (15) and (16):

[0197] E = d(p1,K[I,0]P)+d(p2,K[R2,t2]P) (15);

[0198] P = argminE (16)

[0199] Where d(p1,K[I,0]P) represents the distance between the pixel of the 3D point with world coordinates P projected onto the image where p1 is located and p1, and d(p2,K[R2,t2]P) represents the distance between the pixel of the 3D point with world coordinates P projected onto the image where p2 is located and p2. The 3D coordinates of P are solved using Newton's method or the Levenberg-Marquardt method.

[0200] For example, S215~S216 is only one optional method for obtaining the rendering color of a pixel. In other optional embodiments, the Gaussian sputtering rendering results of each target Gaussian ellipsoid are weighted and summed using the depth weight of the target Gaussian ellipsoid as a weighting coefficient to obtain the rendering color of the pixel. Taking M target Gaussian ellipsoids corresponding to a target pixel as an example, the rendering color of the target pixel is calculated. For specific methods, please refer to formula (17):

[0201] (17);

[0202] in, The first Gaussian ellipsoid in the target pixel sequence represents the first... A target Gaussian ellipsoid G color value, The target Gaussian ellipsoid G represents Depth weights The target Gaussian ellipsoid G represents Opacity This represents the opacity of the target Gaussian ellipsoid Gr, where r is no greater than m.

[0203] For example, S202~S205 is an optional method for initializing a Gaussian ellipsoid set based on a set of color images of the target plant. In one optional embodiment, a multi-view point cloud stitching method or a SLAM algorithm can be used to obtain three-dimensional point cloud data. In another optional embodiment, algorithms such as ORB or SuperPoint can be used to extract key points and key point descriptors from the color images.

[0204] For example, in the rendering process of 3D Gaussian sputtering, high-quality details and edge sharpening are crucial to the rendering effect. High-quality details highlight details, making textures, colors, and shapes more discernible, enhancing the realism and three-dimensionality of objects, while edge sharpening strengthens the outlines of objects, making the boundaries between different objects more distinct, thereby reducing blurriness. High-quality images not only enhance visual appeal but also optimize rendering performance. Therefore, in one alternative embodiment, a rendering optimization step is added before 3D Gaussian sputtering rendering, utilizing a high-resolution reconstruction network to improve image quality. Specifically, the high-resolution reconstruction network HRNet is a generative adversarial network model for image super-resolution. Image super-resolution technology can effectively reduce noise in images, thereby reducing point cloud noise generated by the reconstruction system. By combining a perceptual optimization strategy with generative adversarial networks (GANs), basic residual units are replaced with dense residual units to enable the network model to output original resolution results. Upsampling units and batch normalization units are removed, allowing the model to provide sharper edges and more visually appealing images. This enables the radiation field to more effectively extract the structural features of the tomato plant, thus achieving a more accurate 3D reconstruction of the tomato plant. Therefore, performing this rendering optimization step before the initial SFM point cloud reconstruction not only improves the quality of the reconstructed point cloud but also enhances the rendering effect during Gaussian sputtering. Without changing the image resolution, it improves the detail quality and edge sharpness of the image. This high-quality image not only enhances visual appeal but also effectively improves the rendering effect of the 3D plant reconstruction model generated by 3D Gaussian sputtering technology.

[0205] The above describes a plant three-dimensional reconstruction method provided by the embodiments of this application. The following will describe the apparatus for performing the above-described plant three-dimensional reconstruction method.

[0206] Please see Figure 3 , Figure 3 This is a schematic diagram of a plant three-dimensional reconstruction device provided in an embodiment of this application. The plant three-dimensional reconstruction device is configured in the client, such as... Figure 3 As shown, the plant three-dimensional reconstruction device 300 includes:

[0207] The point cloud data construction unit 301 is used to acquire three-dimensional point cloud data based on the target color image sequence of the target plant; wherein, the target color image sequence includes multiple color images from different perspectives arranged in time sequence, and the three-dimensional point cloud data includes the three-dimensional coordinate values ​​and color values ​​of multiple key points.

[0208] Gaussian initialization unit 302 is used to initialize the Gaussian ellipsoids of each key point based on the three-dimensional point cloud data to obtain a set of Gaussian ellipsoids. The set of Gaussian ellipsoids includes the Gaussian ellipsoids of each key point. The Gaussian parameters of the Gaussian ellipsoids include the mean vector, covariance matrix, color value and opacity.

[0209] The Gaussian iterative update unit 303 is used to perform iterations to update the Gaussian ellipsoid set until a preset iteration termination condition is reached, thereby obtaining the updated Gaussian ellipsoid set.

[0210] The three-dimensional modeling unit 304 is used to construct a three-dimensional reconstruction model of the target plant based on the updated Gaussian ellipsoid set;

[0211] Specifically, the Gaussian update unit is used to perform the iteration as follows:

[0212] One color image is selected from the target color image sequence as the training color image for the iteration;

[0213] For each pixel in the training color image, the depth value of each target Gaussian ellipsoid is obtained based on the depth image corresponding to each target Gaussian ellipsoid of the pixel; the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids;

[0214] The depth weights of each of the target Gaussian ellipsoids are calculated based on at least the depth values; wherein the depth weights are inversely correlated with the depth values.

[0215] For each pixel, the rendering color value of the pixel is calculated based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid;

[0216] Based on the rendering color values ​​of each pixel in the training color image, a rendered image of the target color image is obtained.

[0217] The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

[0218] In one possible implementation, the point cloud data construction unit, when acquiring 3D point cloud data based on a target color image sequence of the target plant, is specifically used for:

[0219] The original color image sequence of the target plant is acquired by a depth camera, and the original color image sequence is filtered based on a preset image quality index to obtain the target color image sequence. The image quality index includes the blur detection probability.

[0220] For each color image in the target color image sequence, the scale-invariant feature transform matching algorithm is used to extract the key points of the color image and the descriptors of the key points;

[0221] Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs for every two color images;

[0222] Based on multiple key point pairs from every two color images, three-dimensional point cloud data of the target plant is generated.

[0223] In one possible implementation, the point cloud data construction unit, when generating the three-dimensional point cloud data of the target plant based on multiple pairs of key points from every two color images, is specifically used for:

[0224] Starting with a time sequence n of 2, traverse the color images in the target color image sequence from smallest to largest:

[0225] When n=2, the fundamental matrix is ​​calculated using the eight-point algorithm based on the pixel coordinates of the key point pairs of the second color image and the first color image. The fundamental matrix is ​​then decomposed to obtain the extrinsic matrix of the second color image, which includes a rotation matrix and a translation vector.

[0226] Using triangulation, the three-dimensional coordinates of each key point in the second color image are calculated based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters, and the extrinsic parameter matrix of the second color image.

[0227] When n > 2, the external parameter matrix of the nth color image is calculated using the perspective n-point PnP algorithm based on the key point pairs of the nth color image and the (n-1)th color image.

[0228] Using triangulation, the three-dimensional coordinates of each key point in the nth color image are calculated based on the pixel coordinates of each key point pair in the nth color image and the (n-1)th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the (n-1)th color image, and the extrinsic parameter matrix of the nth color image.

[0229] The three-dimensional point cloud data is generated based on the three-dimensional coordinates and color values ​​of each key point in each color image.

[0230] In one possible implementation, the Gaussian iterative update unit, when calculating the depth weights of each of the target Gaussian ellipsoids based at least on the depth values, is specifically used for:

[0231] The depth intervals are divided into multiple depth intervals according to the minimum and maximum depth values, resulting in a depth interval sequence. The depth interval sequence includes multiple depth intervals arranged in ascending order of depth value.

[0232] For each depth interval, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order of depth values ​​to obtain the ellipsoid subsequence corresponding to the depth interval.

[0233] For each depth interval, the depth center of the depth interval is calculated based on the position of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value.

[0234] For each target Gaussian ellipsoid within the ellipsoidal subsequence of each depth interval, the depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0235] In one possible implementation, the Gaussian iterative update unit is used to calculate the rendered color value of each pixel based on the depth weights and Gaussian parameters of each target Gaussian ellipsoid, specifically for:

[0236] Project each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtain the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each target Gaussian ellipsoid based on the Gaussian parameters of the Gaussian ellipsoid.

[0237] The opacity of the two-dimensional Gaussian ellipse is calculated based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse.

[0238] For each depth interval, the rendering color value of the depth interval is calculated based on the color value of the target Gaussian ellipsoid, the depth weight, and the opacity of the two-dimensional Gaussian ellipse within the depth interval.

[0239] The rendering color value of the pixel is calculated based on the rendering color values ​​of all depth ranges of the pixel.

[0240] In one possible implementation, the Gaussian iterative update unit is used to calculate the rendering color value of the target depth interval based on the color value of the target Gaussian ellipsoid, the depth weight, and the opacity of the two-dimensional Gaussian ellipse within the target depth interval. Specifically, when the target depth interval is any one of the depth intervals, it is used for:

[0241] For a target Gaussian ellipsoid within the target depth interval, the weighted parameters of each preceding Gaussian ellipsoid of the target Gaussian ellipsoid are multiplied to obtain a cumulative weighted parameter. The preceding Gaussian ellipsoid is a Gaussian ellipsoid whose rank in the subsequence of ellipsoids in the target depth interval is lower than that of the target Gaussian ellipsoid. The cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the preceding Gaussian ellipsoid. The weighted two-dimensional opacity is equal to the depth weight of the preceding Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse.

[0242] For a target Gaussian ellipsoid within the target depth range, calculate the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain the opacity parameter;

[0243] For a target Gaussian ellipsoid within the target depth range, calculate the product of the color value and the opacity parameter of the target Gaussian ellipsoid, and use it as the color to be rendered;

[0244] Using the depth weight of each of the target Gaussian ellipsoids as the weighting coefficient, the colors to be rendered for each of the target Gaussian ellipsoids are weighted and summed to obtain the rendering color value of the target depth range.

[0245] It should be noted that the specific structure and function of the plant three-dimensional reconstruction device can be found in the above embodiments.

[0246] This application also provides an electronic device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0247] like Figure 4 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the electronic device is powered on, the RAM 403 also stores various programs and data required for the operation of the electronic device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0248] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0249] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the plant three-dimensional reconstruction methods provided in this application.

[0250] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the plant three-dimensional reconstruction methods provided in this application.

[0251] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0252] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0253] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0254] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for three-dimensional reconstruction of a plant, characterized in that, include: Three-dimensional point cloud data is obtained based on the target color image sequence of the target plant; wherein, the target color image sequence includes multiple color images from different perspectives arranged in chronological order, and the three-dimensional point cloud data includes the three-dimensional coordinate values ​​and color values ​​of multiple key points; The Gaussian ellipsoids of each key point are initialized based on the three-dimensional point cloud data to obtain a set of Gaussian ellipsoids. The set of Gaussian ellipsoids includes the Gaussian ellipsoids of each key point. The Gaussian parameters of the Gaussian ellipsoids include the mean vector, covariance matrix, color value, and opacity. The iteration is performed to update the Gaussian ellipsoid set until a preset iteration termination condition is met, resulting in the updated Gaussian ellipsoid set. Based on the updated Gaussian ellipsoid set, a three-dimensional reconstruction model of the target plant is constructed; The iteration includes: One color image is selected from the target color image sequence as the training color image for the iteration; For each pixel in the training color image, the depth value of each target Gaussian ellipsoid is obtained based on the depth image corresponding to each target Gaussian ellipsoid of the pixel; the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids; The depth weights of each of the target Gaussian ellipsoids are calculated based on at least the depth values; wherein the depth weights are inversely correlated with the depth values. For each pixel, the rendering color value of the pixel is calculated based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid; Based on the rendering color values ​​of each pixel in the training color image, a rendered image of the target color image is obtained. The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

2. The method for three-dimensional reconstruction of plants according to claim 1, characterized in that, The acquisition of three-dimensional point cloud data based on the target plant's target color image sequence includes: The original color image sequence of the target plant is acquired by a depth camera, and the original color image sequence is filtered based on a preset image quality index to obtain the target color image sequence. The image quality index includes the blur detection probability. For each color image in the target color image sequence, the scale-invariant feature transform matching algorithm is used to extract the key points of the color image and the descriptors of the key points; Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs for every two color images; Based on multiple key point pairs from every two color images, three-dimensional point cloud data of the target plant is generated.

3. The method for three-dimensional reconstruction of plants according to claim 2, characterized in that, The process of generating the three-dimensional point cloud data of the target plant based on multiple pairs of key points from every two color images includes: Starting with a time sequence n of 2, traverse the color images in the target color image sequence from smallest to largest: When n=2, the fundamental matrix is ​​calculated using the eight-point algorithm based on the pixel coordinates of the key point pairs of the second color image and the first color image. The fundamental matrix is ​​then decomposed to obtain the extrinsic matrix of the second color image, which includes a rotation matrix and a translation vector. Using triangulation, the three-dimensional coordinates of each key point in the second color image are calculated based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters, and the extrinsic parameter matrix of the second color image. When n > 2, the external parameter matrix of the nth color image is calculated using the perspective n-point PnP algorithm based on the key point pairs of the nth color image and the (n-1)th color image. Using triangulation, the three-dimensional coordinates of each key point in the nth color image are calculated based on the pixel coordinates of each key point pair in the nth color image and the (n-1)th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the (n-1)th color image, and the extrinsic parameter matrix of the nth color image. The three-dimensional point cloud data is generated based on the three-dimensional coordinates and color values ​​of each key point in each color image.

4. The method for three-dimensional reconstruction of plants according to claim 1, characterized in that, The calculation of depth weights for each of the target Gaussian ellipsoids, based at least on depth values, includes: The depth intervals are divided into multiple depth intervals according to the minimum and maximum depth values, resulting in a depth interval sequence. The depth interval sequence includes multiple depth intervals arranged in ascending order of depth value. For each depth interval, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order of depth values ​​to obtain the ellipsoid subsequence corresponding to the depth interval. For each depth interval, the depth center of the depth interval is calculated based on the position of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value. For each target Gaussian ellipsoid within the ellipsoidal subsequence of each depth interval, the depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

5. The method for three-dimensional reconstruction of plants according to claim 4, characterized in that, For each pixel, the rendering color value of the pixel is calculated based on the depth weights and Gaussian parameters of each target Gaussian ellipsoid, including: Project each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtain the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each target Gaussian ellipsoid based on the Gaussian parameters of the Gaussian ellipsoid. The opacity of the two-dimensional Gaussian ellipse is calculated based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse. For each depth interval, the rendering color value of the depth interval is calculated based on the color value of the target Gaussian ellipsoid, the depth weight, and the opacity of the two-dimensional Gaussian ellipse within the depth interval. The rendering color value of the pixel is calculated based on the rendering color values ​​of all depth ranges of the pixel.

6. The method for three-dimensional reconstruction of plants according to claim 5, characterized in that, Based on the color value, depth weight, and opacity of the two-dimensional Gaussian ellipse within the target depth interval, the rendered color value for the target depth interval is calculated. The target depth interval can be any depth interval, including: For a target Gaussian ellipsoid within the target depth interval, the weighted parameters of each preceding Gaussian ellipsoid of the target Gaussian ellipsoid are multiplied to obtain a cumulative weighted parameter. The preceding Gaussian ellipsoid is a Gaussian ellipsoid whose rank in the subsequence of ellipsoids in the target depth interval is lower than that of the target Gaussian ellipsoid. The cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the preceding Gaussian ellipsoid. The weighted two-dimensional opacity is equal to the depth weight of the preceding Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse. For a target Gaussian ellipsoid within the target depth range, calculate the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain the opacity parameter; For a target Gaussian ellipsoid within the target depth range, calculate the product of the color value and the opacity parameter of the target Gaussian ellipsoid, and use it as the color to be rendered; Using the depth weight of each of the target Gaussian ellipsoids as the weighting coefficient, the colors to be rendered for each of the target Gaussian ellipsoids are weighted and summed to obtain the rendering color value of the target depth range.

7. A three-dimensional reconstruction device for plants, characterized in that, include: A point cloud data construction unit is used to acquire three-dimensional point cloud data based on a target color image sequence of a target plant; wherein, the target color image sequence includes multiple color images from different perspectives arranged in chronological order, and the three-dimensional point cloud data includes the three-dimensional coordinate values ​​and color values ​​of multiple key points. A Gaussian initialization unit is used to initialize the Gaussian ellipsoids of each key point based on the three-dimensional point cloud data to obtain a set of Gaussian ellipsoids. The set of Gaussian ellipsoids includes the Gaussian ellipsoids of each key point. The Gaussian parameters of the Gaussian ellipsoids include the mean vector, covariance matrix, color value, and opacity. A Gaussian iterative update unit is used to perform iterations to update the Gaussian ellipsoid set until a preset iteration termination condition is met, thereby obtaining the updated Gaussian ellipsoid set. A 3D modeling unit is used to construct a 3D reconstruction model of the target plant based on the updated Gaussian ellipsoid set. Specifically, the Gaussian iterative update unit is used to perform the iteration, specifically for: One color image is selected from the target color image sequence as the training color image for the iteration; For each pixel in the training color image, the depth value of each target Gaussian ellipsoid is obtained based on the depth image corresponding to each target Gaussian ellipsoid of the pixel; the target Gaussian ellipsoid of the pixel is the Gaussian ellipsoid that covers the pixel in the set of Gaussian ellipsoids; The depth weights of each of the target Gaussian ellipsoids are calculated based on at least the depth values; wherein the depth weights are inversely correlated with the depth values. For each pixel, the rendering color value of the pixel is calculated based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid; Based on the rendering color values ​​of each pixel in the training color image, a rendered image of the target color image is obtained. The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

8. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the plant three-dimensional reconstruction method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the plant three-dimensional reconstruction method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the plant three-dimensional reconstruction method as described in any one of claims 1 to 6.

Images