Plant three-dimensional reconstruction method and related device

Abstract

The invention discloses a plant three-dimensional reconstruction method and a related device, and relates to the technical field of computer vision, the method executes multiple iterations to update a Gaussian ellipsoid set, when a color image under each view angle is rendered in each iteration, the depth value of each key point is obtained based on a depth image, and the color image at each view angle is rendered. The depth weight of the target Gaussian ellipsoid is obtained based on the depth value, the depth weight is inversely correlated with the depth value, that is, the contribution of the Gaussian ellipsoid with the small depth value to color rendering is improved, depth information distinguishing of a fine region is enhanced by introducing the depth weight, the contribution of Gaussian ellipsoids of different depth levels to rendering dyeing is accurately controlled, and the color rendering efficiency is improved. Therefore, the geometric accuracy and layering sense of three-dimensional reconstruction of the plant are improved, and the reduction degree of the three-dimensional reconstruction model to the plant is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a three-dimensional plant reconstruction method and related devices. Background Art

[0002] Plant three-dimensional modeling is the process of converting the plant's morphology, structure and growth characteristics into a three-dimensional reconstruction model for analysis, monitoring and management. Accurate plant three-dimensional modeling can optimize planting management and improve production efficiency, thereby promoting the development of smart agriculture and realizing the automation and refinement of agricultural management.

[0003] Three-dimensional Gaussian sputtering technology has powerful scene expression capabilities and supports fast rendering. Therefore, three-dimensional Gaussian sputtering can be used to perform three-dimensional modeling of plants. However, when using traditional three-dimensional Gaussian sputtering technology to perform three-dimensional modeling of plants, there is a general disadvantage of poor rendering effect, resulting in a low degree of restoration of the plant's three-dimensional reconstruction model. Summary of the invention

[0004] In view of the above problems, the present application provides a plant three-dimensional reconstruction method and related devices to achieve the purpose of improving the restoration degree of the plant by the plant three-dimensional reconstruction model. The specific scheme is as follows:

[0005] The first aspect of the present application provides a method for three-dimensional reconstruction of a plant.

[0006] Acquire three-dimensional point cloud data based on a target color image sequence of a target plant; wherein the target color image sequence includes a plurality of color images at different viewing angles arranged in time sequence, and the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of a plurality of key points;

[0007] Initializing the Gaussian ellipsoid of each of the key points according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set, wherein the Gaussian ellipsoid set includes the Gaussian ellipsoid of each of the key points, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity;

[0008] Performing iteration to update the Gaussian ellipsoid set until a preset iteration termination condition is reached to obtain the updated Gaussian ellipsoid set;

[0009] Based on the updated Gaussian ellipsoid set, constructing a three-dimensional reconstruction model of the target plant;

[0010] The iteration includes:

[0011] A color image selected from the target color image sequence as the iterative training color image;

[0012] For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, the depth value of each target Gaussian ellipsoid is obtained; the target Gaussian ellipsoid of the pixel point is a Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point;

[0013] Calculating the depth weight of each of the target Gaussian ellipsoids based at least on the depth value; wherein the depth weight is inversely correlated with the depth value;

[0014] For each of the pixel points, based on the depth weight and Gaussian parameters of each of the target Gaussian ellipsoids, a rendering color value of the pixel point is calculated;

[0015] Rendering to obtain a rendered image of the target color image based on the rendered color value of each pixel of the training color image;

[0016] The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

[0017] In a possible implementation, obtaining three-dimensional point cloud data based on a target color image sequence of a target plant includes:

[0018] Acquire the original color image sequence of the target plant through a depth camera, and filter the original color image sequence based on a preset image quality index to obtain the target color image sequence, wherein the image quality index includes a blur detection probability;

[0019] For each color image in the target color image sequence, using a scale-invariant feature transformation matching algorithm to extract key points of the color image and descriptors of the key points;

[0020] Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs of every two color images;

[0021] Three-dimensional point cloud data of the target plant is generated based on multiple pairs of key points of every two color images.

[0022] In a possible implementation, generating the three-dimensional point cloud data of the target plant according to the multiple pairs of key points of each two color images includes:

[0023] Starting from time sequence n=2, traverse the color images in the target color image sequence from small to large:

[0024] When n=2, based on the pixel coordinates of the key point pairs of the second color image and the first color image, an eight-point algorithm is used to calculate a basic matrix, and the basic matrix is ​​decomposed to obtain an extrinsic parameter matrix of the second color image, where the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0025] Calculate the three-dimensional coordinates of each key point in the second color image based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters and the extrinsic parameter matrix of the second color image using a triangulation method;

[0026] When n>2, based on the key point pairs of the nth color image and the n-1th color image, the perspective n-point PnP algorithm is used to calculate the extrinsic parameter matrix of the nth color image;

[0027] Calculate the three-dimensional coordinates of each key point in the nth color image based on the pixel coordinates of each key point pair in the nth color image and the n-1th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the n-1th color image, and the extrinsic parameter matrix of the nth color image using a triangulation method;

[0028] The three-dimensional point cloud data is generated according to the three-dimensional coordinate value and color value of each key point in each color image.

[0029] In a possible implementation, calculating the depth weight of each target Gaussian ellipsoid at least based on the depth value includes:

[0030] Divide a plurality of depth intervals according to a minimum depth value and a maximum depth value to obtain a depth interval sequence, wherein the depth interval sequence includes a plurality of depth intervals arranged from small to large according to the depth value;

[0031] For each of the depth intervals, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order according to the depth values ​​to obtain an ellipsoid subsequence corresponding to the depth interval;

[0032] For each of the depth intervals, based on the order of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value, calculate the depth center of the depth interval;

[0033] For each target Gaussian ellipsoid in the ellipsoid subsequence of each depth interval, a depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0034] In a possible implementation, for each pixel point, based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid, calculating the rendering color value of the pixel point includes:

[0035] Projecting each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtaining the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each of the target Gaussian ellipsoids based on the Gaussian parameters of the Gaussian ellipsoid;

[0036] Calculate the opacity of the two-dimensional Gaussian ellipse based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse;

[0037] For each of the depth intervals, based on the color value of the target Gaussian ellipsoid within the depth interval, the depth weight, and the opacity of the two-dimensional Gaussian ellipse, a rendering color value of the depth interval is calculated;

[0038] The rendering color value of the pixel is calculated based on the rendering color values ​​of all the depth intervals of the pixel.

[0039] In a possible implementation, based on the color value of the target Gaussian ellipsoid in the target depth interval, the depth weight, and the opacity of the two-dimensional Gaussian ellipse, the rendering color value of the target depth interval is calculated, and the target depth interval is any one of the depth intervals, including:

[0040] For a target Gaussian ellipsoid within the target depth interval, weighted parameters of each previous Gaussian ellipsoid of the target Gaussian ellipsoid are cumulatively multiplied to obtain a cumulative weighted parameter, wherein the previous Gaussian ellipsoid is a Gaussian ellipsoid with a smaller rank than the target Gaussian ellipsoid in the ellipsoid subsequence within the target depth interval, and the cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the previous Gaussian ellipsoid, and the weighted two-dimensional opacity is equal to the depth weight of the previous Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse;

[0041] For a target Gaussian ellipsoid within the target depth interval, calculating the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain an opacity parameter;

[0042] For a target Gaussian ellipsoid within the target depth interval, calculating the product of the color value and the opacity parameter of the target Gaussian ellipsoid as the color to be rendered;

[0043] The depth weight of each target Gaussian ellipsoid is used as a weight coefficient, and weighted addition is performed on the colors to be rendered of each target Gaussian ellipsoid to obtain a rendering color value of the target depth interval.

[0044] A second aspect of the present application provides a plant three-dimensional reconstruction device, comprising:

[0045] A point cloud data construction unit, used to obtain three-dimensional point cloud data based on a target color image sequence of a target plant; wherein the target color image sequence includes a plurality of color images under different viewing angles arranged in time sequence, and the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of a plurality of key points;

[0046] A Gaussian initialization unit, used for initializing the Gaussian ellipsoid of each of the key points according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set, wherein the Gaussian ellipsoid set includes the Gaussian ellipsoids of each of the key points, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity;

[0047] A Gaussian iterative updating unit, used for constructing a three-dimensional reconstruction model of the target plant based on the updated Gaussian ellipsoid set;

[0048] The three-dimensional modeling unit is used to update the Gaussian ellipsoid set based on the loss values ​​of the training color image and the rendered image.

[0049] Wherein, when the Gaussian update unit is used to perform the iteration, it is specifically used to:

[0050] A color image selected from the target color image sequence as the iterative training color image;

[0051] For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, the depth value of each target Gaussian ellipsoid is obtained; the target Gaussian ellipsoid of the pixel point is a Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point;

[0052] Calculating the depth weight of each of the target Gaussian ellipsoids based at least on the depth value; wherein the depth weight is inversely correlated with the depth value;

[0053] For each of the pixel points, based on the depth weight and Gaussian parameters of each of the target Gaussian ellipsoids, a rendering color value of the pixel point is calculated;

[0054] Based on the rendered color value of each pixel point of the training color image, a rendered image of the target color image is rendered.

[0055] A third aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the plant three-dimensional reconstruction method of the first aspect or any implementation of the first aspect.

[0056] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0057] The memory is used to store computer programs;

[0058] The processor is used to execute the computer program so that the electronic device can implement the plant three-dimensional reconstruction method of the first aspect or any implementation manner of the first aspect.

[0059] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the plant three-dimensional reconstruction method of the above-mentioned first aspect or any implementation of the first aspect.

[0060] By means of the above-mentioned technical scheme, the present application provides a three-dimensional reconstruction method and related devices for plants, which perform multiple iterations to update a set of Gaussian ellipsoids. When rendering a color image at each perspective in each iteration, the depth value of each key point is obtained based on the depth image, and the depth weight of the target Gaussian ellipsoid is obtained based on the depth value. The depth weight is inversely correlated with the depth value, that is, the contribution of the Gaussian ellipsoid with a smaller depth value to color rendering is increased, and the depth information distinction of subtle areas is enhanced by introducing the depth weight, and the contribution of the Gaussian ellipsoids at different depth levels to rendering and coloring is accurately controlled, thereby improving the geometric accuracy and layering of the three-dimensional reconstruction of the plant, and further improving the degree of restoration of the three-dimensional reconstruction model to the plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0062] Figure 1 A schematic diagram of a process for three-dimensional reconstruction of a plant provided in an embodiment of the present application;

[0063] Figure 2 A flowchart of a specific implementation of a three-dimensional plant reconstruction method provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of the structure of a plant three-dimensional reconstruction device provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0067] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0068] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0069] The present application can be applied to the field of computer vision technology, and specifically to the field of plant research and management, to perform three-dimensional modeling of crops or cash crops in a production environment, and to generate a three-dimensional digital model of the plant, so as to monitor the morphology, structure and growth characteristics of the plant at various stages in the growth cycle. Taking a tomato plant as an example, a tomato plant refers to the plant body of a tomato, including roots, stems, leaves and other parts. As a kind of conventional agricultural vegetable, tomato is rich in nutritional value and has a large tomato yield and strong ability to adapt to the environment when planted. It is an important vegetable that is indispensable in the daily life of the Chinese people. The three-dimensional modeling method of a plant provided in an embodiment of the present application can be specifically applied to the three-dimensional modeling of a tomato plant, by improving the rendering effect of the three-dimensional modeling of a tomato plant based on three-dimensional Gaussian sputtering, thereby improving the reduction of the three-dimensional reconstruction model of the tomato plant to the tomato plant, and further, providing an accurate basis for accurately monitoring and managing the growth of tomato plants, and improving the level of agricultural automation.

[0070] A three-dimensional plant reconstruction method provided in an embodiment of the present application can be applied to a three-dimensional reconstruction system, wherein the three-dimensional reconstruction system includes a three-dimensional reconstruction device and a depth camera, wherein the depth camera can be an Intel Realsense D415 depth camera, the depth camera can be a handheld depth camera, or it can be installed in the plant growth environment, and the installation position and installation method can be determined according to the actual environment. The three-dimensional reconstruction device includes a terminal device and / or a server with a three-dimensional reconstruction function, wherein the terminal device can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the server includes a single-point server or a server cluster, which is not limited in this embodiment.

[0071] In the rendering process of 3D Gaussian sputtering technology, Gaussian ellipsoids of multiple key points may contribute color information to the same two-dimensional pixel. If the depth information is not considered, the rendering result may be blurred or ghosted. Since the depth image of the depth camera can provide high-precision depth information, based on this, the embodiment of the present application introduces depth weights to enhance the depth information distinction of subtle areas. Even when the depth values ​​are close, the contribution of Gaussian ellipsoids at different depth levels can be more accurately controlled, thereby improving the geometric accuracy and layering of the three-dimensional reconstruction of the plant, thereby improving the restoration degree of the three-dimensional reconstruction model to the plant.

[0072] The following is a detailed introduction to the plant three-dimensional reconstruction method of the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 A schematic diagram of a process of a three-dimensional plant reconstruction method provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0073] S11. Acquire three-dimensional point cloud data based on a target color image sequence of a target plant.

[0074] In this embodiment, the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of multiple key points, wherein the key points are extracted from the color image.

[0075] In this embodiment, the target color image sequence includes a plurality of color images at different viewing angles arranged in time sequence.

[0076] In an optional embodiment, a depth camera is used to acquire color images and depth images of a target plant at multiple continuous viewing angles to obtain raw image data, and the raw image data is preprocessed by image enhancement, denoising, sampling, etc. to obtain a target color image sequence.

[0077] S12. Initialize the Gaussian ellipsoid of each key point according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set.

[0078] In this embodiment, the Gaussian ellipsoid set includes the Gaussian ellipsoid of each key point, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity.

[0079] In this embodiment, the initialized Gaussian ellipsoid set includes initial values ​​of various Gaussian parameters of the Gaussian ellipsoids of various key points.

[0080] S13, executing iteration to update the Gaussian ellipsoid set until a preset iteration termination condition is reached to obtain an updated Gaussian ellipsoid set.

[0081] In this embodiment, the iteration termination condition includes reaching a preset iteration number threshold, for example, N is a preset iteration number threshold, and when the iteration number n=N, the iteration is stopped and an updated Gaussian ellipsoid set is output.

[0082] In this embodiment, any iteration includes:

[0083] S131. Select a color image from the target color image sequence as an iterative training color image.

[0084] In this embodiment, in each iteration, a color image is randomly selected from the target color image sequence or in a preset order as an iterative training color image, and N is configured to be a value much larger than the number of color images.

[0085] S132 . For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, obtain the depth value of each target Gaussian ellipsoid.

[0086] In this embodiment, the target Gaussian ellipsoid of the pixel point is the Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point, that is, the Gaussian ellipsoid contained in the viewing cone of the pixel point.

[0087] S133. Calculate the depth weight of each target Gaussian ellipsoid based at least on the depth value.

[0088] In this embodiment, the depth weight is inversely correlated with the depth value, that is, the closer the depth value is, the greater the depth weight is.

[0089] S134 . For each pixel point, based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid, calculate the rendering color value of the pixel point.

[0090] S135 . Rendering to obtain a rendered image of the target color image based on the rendered color value of each pixel of the training color image.

[0091] S136. Update the Gaussian ellipsoid set based on the loss values ​​of the training color image and the rendered image.

[0092] S14. Based on the updated Gaussian ellipsoid set, a three-dimensional reconstruction model of the target plant is constructed.

[0093] It can be seen from the above technical scheme that a three-dimensional reconstruction method of a plant provided in an embodiment of the present application performs multiple iterations to update a set of Gaussian ellipsoids. When rendering a color image at each perspective in each iteration, the depth value of each key point is obtained based on the depth image, and the depth weight of the target Gaussian ellipsoid is obtained based on the depth value. The depth weight is inversely correlated with the depth value, that is, the contribution of the Gaussian ellipsoid with a smaller depth value to the color rendering is increased, and the depth information distinction of subtle areas is enhanced by introducing the depth weight, and the contribution of Gaussian ellipsoids at different depth levels to the rendering and coloring is accurately controlled, thereby improving the geometric accuracy and layering of the three-dimensional reconstruction of the plant, and further improving the restoration degree of the three-dimensional reconstruction model to the plant.

[0094] See also Figure 2 , Figure 2 A specific implementation flow chart of a plant 3D reconstruction method provided in an embodiment of the present application is as follows: Figure 2 As shown, the method specifically includes S201-217, as follows:

[0095] S201, obtaining an original image sequence of a target plant through a depth camera, and screening the original image sequence based on a preset image quality index to obtain a target image sequence.

[0096] In this embodiment, the original image sequence includes an original color image sequence and an original depth image sequence, and the target image sequence includes a target color image sequence and a target depth image sequence.

[0097] In this embodiment, the original color image sequence includes color images arranged in time sequence, and the original depth image sequence includes depth images arranged in time sequence, wherein the time sequence refers to the order of shooting time.

[0098] Specifically, when the time for three-dimensional reconstruction is reached, the depth camera is controlled to shoot around the plant with a preset resolution, frame rate, shooting duration, and moving trajectory to obtain a video stream, and image frames are collected from the video stream at a preset sampling frequency to obtain an original color image sequence and an original depth image sequence.

[0099] Taking the target plant as a tomato plant as an example, the resolution is 1280 × 720, the frame rate is 30, the shooting time is 1 minute, the moving trajectory is 360 degrees, and the sampling frequency is 5 frames. Then, when the start time of the preset reconstruction cycle is reached, the Intel Realsense D415 depth camera is controlled to shoot around the plant for 1 minute at a resolution of 1280 × 720 and a frame rate of 30, and a video stream including 1800 image frames is collected. The sampling frequency of selecting 1 frame every 5 frames is adopted, and 360 depth images and color images are sampled from the video stream. The depth images are arranged in time sequence to obtain the original depth image sequence, and the color images are arranged in time sequence to obtain the original color image sequence.

[0100] It should be noted that the original depth images and original color images in the original image set are sorted in the order of shooting time. Based on the continuity of the image frames, the calculation speed of subsequent image matching can be accelerated.

[0101] In this embodiment, the image quality index includes a blur detection probability CPBD value. The CPBD value is an image quality index that conforms to human visual characteristics. The larger the value, the clearer the image details. Specifically, the CPBD value of each color image in the original color image sequence is calculated, and the color images below the preset CPBD threshold are removed and deleted, and the color images not below the CPBD threshold are retained to obtain the target color image sequence. Furthermore, the depth images corresponding to the color images below the CPBD threshold in the original depth image sequence are deleted to obtain the target depth image sequence.

[0102] Continuing with the previous example, the CPBD values ​​of 360 color images of tomato plants are calculated, and 250 color images and corresponding depth images whose CPBD values ​​are not less than the CPBD threshold are retained, thereby obtaining a target color image sequence including color images under 250 viewing angles and a target depth image sequence including depth images under 250 viewing angles.

[0103] This step improves the image quality of the plant color image by screening the original image sequence based on the CPBD value, thereby helping to improve the three-dimensional reconstruction effect.

[0104] S202 : For each color image in the target color image sequence, use a scale-invariant feature transformation matching algorithm to extract key points of the color image and descriptors of the key points.

[0105] In this embodiment, a SIFT (Scale Invariant Feature Transform) algorithm is used to extract key points of each color image, and a SIFT descriptor is generated for each key point. The SIFT descriptor describes the features of the key point. Specifically, the SIFT descriptor can be a feature vector.

[0106] S203 . Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple key point pairs of every two color images.

[0107] In this embodiment, the key point pairs of the first color image and the second color image include key points of the first target color image and key points of the second target color image that meet a preset matching condition, and the matching condition includes that the Euclidean distance between SIFT descriptors is less than a preset distance threshold. Wherein, the first target color image and the second target color image are any two color images.

[0108] S204, generating three-dimensional point cloud data of the target plant based on the multiple pairs of key points of every two color images.

[0109] In this embodiment, the nth color image is the color image with a sequence number n in the target color image sequence, where n is greater than 1.

[0110] An optional method for generating three-dimensional point cloud data of a target plant includes A1 to A3:

[0111] A1. When n=2, based on the pixel coordinates of the key point pairs of the second color image and the first color image, the eight-point algorithm is used to calculate the basic matrix, and the basic matrix is ​​decomposed to obtain the external parameter matrix of the second color image.

[0112] In this embodiment, the first color image and the second color image are color images with order 1 and 2 in the target color image sequence, respectively. It should be noted that the basic matrix describes the epipolar geometric relationship between the first color image and the second color image. The extrinsic matrix is ​​a combination of a rotation matrix and a translation vector, which is used to transform a point in the world coordinate system to the camera coordinate system. The rotation matrix is ​​the rotation part in the camera pose, which is used to describe the orientation of the camera. The translation vector is the translation part in the camera pose, which is used to describe the position of the camera.

[0113] Specifically, let p1 be the pixel coordinates of the key point Q1 in the first color image I1, and p2 be the pixel coordinates of the key point Q2 matching the key point Q1 in the second color image I2.

[0114] Based on the epipolar geometric relationship between I1 and I2, at least 8 key point pairs are used, and the pixel coordinates of each key point pair are substituted into the first function to calculate the basic matrix. The first function is as shown in formula (1):

[0115] (1);

[0116] It should be noted that the degree of freedom of the basic matrix F is 7, and there must be at least 8 key point pairs for feature matching. If the number of key point pairs is 8, a unique solution is obtained. If the number of key point pairs is more than 8, a least squares solution is obtained, and an approximate solution is found by minimizing the sum of squares of the errors.

[0117] Furthermore, the basic matrix is ​​decomposed to obtain the extrinsic parameter matrix, which includes the rotation matrix R and the translation vector t. The decomposition method of the extrinsic parameter matrix is ​​shown in formula (2):

[0118] (2).

[0119] A2. Using triangulation, based on the pixel coordinates of each key point in the second color image and the first color image, the camera intrinsic parameter and extrinsic parameter matrix, calculate the three-dimensional coordinates of each key point in the second color image.

[0120] In this embodiment, the extrinsic parameter matrix includes a rotation matrix and a translation vector.

[0121] Taking point P as an example, in an optional embodiment, a linear method is used to calculate the three-dimensional coordinates of the key point P in the second color image, such as formula (3) and formula (4):

[0122] (3);

[0123] (4);

[0124] Among them, p1, p2, K, R, t are all known, p1 and p2 are the pixel (homogeneous) coordinates of the key points in the key point pair, K is the camera intrinsic parameter, R and t are the rotation matrix and translation vector of the camera view of the second image, respectively. P is the three-dimensional (homogeneous) coordinate in the world coordinate system.

[0125] A3. When n>2, the color images are traversed in time sequence, and the extrinsic parameter matrix of the nth color image is calculated using the PnP algorithm based on the key point pairs of the nth color image and the n-1th color image.

[0126] In this embodiment, the PnP (Perspective-n-Point) algorithm is a method for solving the motion of 3D to 2D point pairs. Specifically, based on the coordinate data of the key point pairs of the nth color image and the n-1th color image, the camera pose corresponding to the nth color image, that is, the extrinsic parameter matrix, is estimated.

[0127] In this embodiment, the coordinate data of the key point pairs of the nth color image and the n-1th color image are obtained, the coordinate data of the key point pairs include the three-dimensional coordinate values ​​of the key points of the n-1th color image and the pixel coordinate values ​​of the nth color image, and the PnP algorithm is used to solve the camera pose corresponding to the nth color image.

[0128] It should be noted that the epipolar geometry method requires at least 8 key point pairs, and there are problems with initialization, pure rotation and scale. The PnP algorithm does not require the use of epipolar constraints and only requires 3 pairs of key point pairs to achieve camera pose estimation.

[0129] A4. Using triangulation, the three-dimensional coordinates of each key point in the nth color image are calculated based on the pixel coordinates of each key point pair in the nth color image and the n-1th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the n-1th color image, and the extrinsic parameter matrix of the nth color image.

[0130] In this embodiment, the method of using triangulation to calculate the three-dimensional coordinates of the key point P is shown in formula (5) and formula (6):

[0131] p1 = K[R(n-1) t(n-1)]P (5);

[0132] p2 = K[Rn tn]P (6);

[0133] In formulas (5) and (6), n is greater than 2, R(n-1) and t(n-1) are the rotation matrix and translation vector of the camera view of the n-1th image. Rn and tn are the rotation matrix and translation vector of the camera view in the extrinsic matrix of the nth color image.

[0134] It should be noted that all color images are traversed in sequence to obtain the three-dimensional coordinate value of each key point in each color image and the extrinsic parameter matrix under the corresponding viewing angle.

[0135] A5. Generate three-dimensional point cloud data based on the three-dimensional coordinate values ​​of each key point in each color image.

[0136] In this embodiment, the three-dimensional point cloud data includes the three-dimensional coordinate value and color value of each key point.

[0137] In summary, starting from the time sequence n of 2, the color images in the target color image sequence are traversed from small to large: when n=2, based on the pixel coordinates of the key point pairs of the second color image and the first color image, the eight-point algorithm is used to calculate the basic matrix, and the basic matrix is ​​decomposed to obtain the external parameter matrix of the second color image, which includes the rotation matrix and the translation vector. Using the triangulation method, the three-dimensional coordinates of each key point in the second color image are calculated based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters and the external parameter matrix of the second color image. When n>2, based on the key point pairs of the nth color image and the n-1th color image, the perspective n-point PnP algorithm is used to calculate the external parameter matrix of the nth color image. Using the triangulation method, based on the pixel coordinates of each key point pair in the nth color image and the n-1th color image, the camera intrinsic parameters, the external parameter matrix of the n-1th color image and the external parameter matrix of the nth color image, the three-dimensional coordinates of each key point in the nth color image are calculated. According to the three-dimensional coordinate values ​​and color values ​​of each key point in each color image, three-dimensional point cloud data is generated.

[0138] S205 , initializing the Gaussian ellipsoid of each key point based on the three-dimensional point cloud data to obtain an initial Gaussian ellipsoid set.

[0139] In this embodiment, the Gaussian ellipsoid set includes the Gaussian ellipsoid of each key point. The Gaussian ellipsoid is a Gaussian parameterized three-dimensional object used to describe the position information and shape information of the key point.

[0140] Specifically, the Gaussian parameters of the Gaussian ellipsoid include the mean vector, covariance matrix, color value and opacity, where the color value of the 3D Gaussian ellipsoid is represented by a continuous spherical harmonic function, and the opacity of the 3D Gaussian ellipsoid is represented by the point cloud opacity α that can be learned and optimized. The Gaussian ellipsoid uses the mean vector μ and the covariance matrix Σ to represent the position and shape. The Gaussian ellipsoid is described by the mean vector in three-dimensional space. For each Gaussian ellipsoid, its mean represents the coordinates of the center of the Gaussian ellipsoid in three-dimensional space, and the shape is described by the covariance matrix. The mean of the Gaussian ellipsoid of the key point is the three-dimensional coordinate of the key point, and the covariance of the Gaussian ellipsoid is calculated using the n nearest neighbors of the key point.

[0141] Optionally, a 3D Gaussian function G(x) containing position information and shape information is used to represent the 3D Gaussian ellipsoid, as shown in formula (7):

[0142] (7);

[0143] Among them, the position information is x=(x1, x2, x3), which is the three-dimensional coordinates of the point in the three-dimensional point cloud.

[0144] Based on formula (7), the 3D Gaussian function is decomposed into two components: the rotation matrix R and the scaling matrix S, as shown in formula (8):

[0145] (8);

[0146] The geometric meaning expressed by formula (8) is to first rotate the 3D Gaussian ellipsoid to be flush with the ellipsoid world, then shrink it along the axis, and then rotate it back. This decomposition method can not only ensure the semi-positive definite property of the covariance matrix, but also reduce the difficulty of operations between matrices.

[0147] It should be noted that S202~S205 are specific methods for initializing the Gaussian ellipsoid set based on the SFM (Structure from Motion) technology, which uses four processes: feature extraction and matching, solving the initial camera pose by epipolar constraints, estimating the camera pose by PnP, and solving the three-dimensional coordinates of key points by triangulation.

[0148] Furthermore, the Gaussian ellipsoid set obtained in this step is used as the initial Gaussian ellipsoid set, and S206 to S219 are iteratively executed to obtain an updated Gaussian ellipsoid set, that is, a target Gaussian ellipsoid set.

[0149] S206 , randomly selecting a training color image from the target color image sequence, and obtaining multiple target Gaussian ellipsoids of each pixel point in the training color image from a Gaussian ellipsoid set.

[0150] In this embodiment, for a pixel point, the Gaussian ellipsoid covering the pixel point is used as the target Gaussian ellipsoid. For example, for a target pixel point in a training color image, M target Gaussian ellipsoids of the target pixel point are obtained from the Gaussian ellipsoid set. In this embodiment, the specific process of projection includes:

[0151] Given a frame of camera pose , the 3D point in the world coordinate system Transform to the camera coordinate system and transform the 3D Gaussian Projected onto the 2D image plane to form a 2D Gaussian , expressed as formula (9) and formula (10) as follows:

[0152] (9);

[0153] (10);

[0154] in, represents the approximate Jacobian matrix of the projective transformation, Represents the rotation matrix from the world coordinate system to the camera coordinate system.

[0155] S207 . For each target Gaussian ellipsoid, obtain a depth value of the target Gaussian ellipsoid based on the target depth image.

[0156] In this embodiment, the target depth image is a depth image where the key points represented by the target Gaussian ellipsoid are located. The target depth image is obtained from the target depth image sequence, and the depth information of the key points represented by the target Gaussian ellipsoid is obtained based on the target depth image, and the depth information includes the depth value. The target Gaussian ellipsoids are sorted according to the depth values ​​to obtain a target Gaussian ellipsoid sequence. The target Gaussian ellipsoid sequence is recorded as {G1~GM}. It should be noted that according to the basic principle of imaging, objects close to the camera will cause occlusion to objects far away from the camera. S208, divide multiple depth intervals according to the minimum depth value and the maximum depth value to obtain a depth interval sequence.

[0157] In this embodiment, the depth interval sequence includes depth intervals arranged in ascending order according to boundary values, the minimum depth value is the minimum depth value of the target Gaussian ellipsoid, and the maximum depth value is the maximum depth value of the target Gaussian ellipsoid.

[0158] In this embodiment, the depth interval can be divided according to a preset interval length, and the depth interval can be evenly divided according to a preset number of intervals. For example, the minimum depth value of the M target Gaussian ellipsoids of the target pixel point is dmin, and the maximum depth value is dmax. K depth intervals are evenly divided in the range [dmin, dmax]. The kth (k∈[1,K]) depth interval is recorded as [dk1, dk2), that is, dk1 is the minimum boundary value of the kth depth interval, dk2 is the maximum boundary value of the kth depth interval, d11 is dmin, and dK2 is dmax. Sort the K depth intervals from small to large according to the minimum boundary value, and the depth interval sequence obtained is {[dk1, dk2)}.

[0159] S209 . For each depth interval, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted from small to large according to the depth values ​​to obtain an ellipsoid subsequence corresponding to the depth interval.

[0160] In this embodiment, the number of target Gaussian ellipsoids in the ellipsoid subsequence Sk corresponding to the k-th depth interval is denoted as I, wherein the i-th (i∈[1,I]) target Gaussian ellipsoid is the target Gaussian ellipsoid with the order i.

[0161] S210 . For each depth interval, calculate a depth center of the depth interval based on the order of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value.

[0162] In this embodiment, the difference between the maximum depth value and the minimum depth value is calculated to obtain the total depth difference, and the total depth difference is divided by the number of depth intervals to obtain the interval range value of the depth interval. The interval range value is multiplied by the product of the sequence of the depth interval in the depth interval sequence, and the minimum depth value is added to obtain the depth center of the depth interval.

[0163] Taking the calculation of the depth center of the kth depth interval as an example, the depth center of the kth depth interval The calculation formula is formula (11):

[0164] (11).

[0165] S211 . For the target Gaussian ellipsoid in the ellipsoid subsequence of each depth interval, calculate the depth weight of the target Gaussian ellipsoid based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0166] In this embodiment, the depth weight of the target Gaussian ellipsoid is inversely correlated with the depth value.

[0167] In this embodiment, the depth weight of the i-th target Gaussian ellipsoid in the ellipsoid subsequence of the k-th depth interval is calculated. The formula for is shown in formula (12):

[0168] (12);

[0169] In the formula, represents the preset weight adjustment parameter corresponding to the kth depth interval, which is used to control the weight distribution of the interval. Represents the depth value of the i-th target Gaussian ellipsoid in the ellipsoid subsequence of the k-th depth interval.

[0170] S212: Project each target Gaussian ellipsoid onto the two-dimensional plane of the training image to obtain the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each target Gaussian ellipsoid.

[0171] In this embodiment, the Gaussian function of the target Gaussian ellipsoid of the key point is expressed as:

[0172] .

[0173] Correspondingly, the mean vector and covariance matrix are expressed as follows by the Gaussian function of the two-dimensional Gaussian ellipse:

[0174] ;

[0175] Among them, μ and Σ represent the mean vector and covariance matrix of the Gaussian ellipsoid, respectively. and They represent the mean vector and covariance matrix of the two-dimensional Gaussian ellipse after projection onto the Gaussian ellipsoid.

[0176] S213 . For each depth interval, based on the depth weight, color value, opacity of the target Gaussian ellipsoid in the ellipsoid subsequence of the depth interval and the mean vector and covariance matrix of the two-dimensional Gaussian ellipsoid, calculate the rendering color value of the depth interval.

[0177] In this embodiment, for a target Gaussian ellipsoid within the target depth interval, the weighted parameters of each previous Gaussian ellipsoid of the target Gaussian ellipsoid are accumulated to obtain a cumulative weighted parameter, where the previous Gaussian ellipsoid is a Gaussian ellipsoid whose rank is smaller than the target Gaussian ellipsoid in the ellipsoid subsequence in the target depth interval, and the cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the previous Gaussian ellipsoid, and the weighted two-dimensional opacity is equal to the depth weight of the previous Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse.

[0178] For a target Gaussian ellipsoid within the target depth interval, the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse is calculated to obtain the opacity parameter.

[0179] For a target Gaussian ellipsoid within the target depth range, the product of the color value of the target Gaussian ellipsoid and the opacity parameter is calculated as the color to be rendered.

[0180] The depth weight of each target Gaussian ellipsoid is used as the weight coefficient, and the colors to be rendered of each target Gaussian ellipsoid are weightedly added to obtain the rendering color value of the target depth interval. In this embodiment, taking the kth depth interval as an example, the rendering color value of the kth depth interval is The calculation formula of is shown in formula (13):

[0181] (13);

[0182] In the formula, The first pixel in the target Gaussian ellipsoid sequence The target Gaussian ellipsoid G The color value of Represents the target Gaussian ellipsoid G The depth weight of Represents the target Gaussian ellipsoid G The opacity of (also represents the opacity of the two-dimensional Gaussian ellipse after projection), Represents the target opacity. The calculation method of the target opacity is shown in formula (14). Represents the depth weight of the target Gaussian ellipsoid Gj (j is not greater than i), Indicates the opacity of the target Gaussian ellipsoid Gj.

[0183] (14);

[0184] in, Gaussian function representing a two-dimensional Gaussian ellipse.

[0185] It should be noted that, for a Gaussian ellipsoid or a two-dimensional Gaussian ellipse, the closer to the center, the greater the opacity, and the farther away from the center, the smaller the opacity. S214: For each pixel, calculate a rendering color value of the pixel based on the rendering color values ​​of each depth interval.

[0186] S215 . Rendering to obtain a rendered image of the training color image based on the rendered color value of each pixel of the nth color image.

[0187] S216. Based on the loss values ​​of the rendered image and the color image, the Gaussian ellipsoid set is updated using a back gradient propagation algorithm.

[0188] S217, judging whether the preset iteration termination condition is reached, if so, taking the updated Gaussian ellipsoid set as the target Gaussian ellipsoid set, if not, executing the next iteration.

[0189] In this embodiment, the iteration termination condition includes that the number of iterations reaches a preset iteration number threshold, for example, N=30000. When the number of iterations reaches 30000, the iteration stops. Otherwise, after the number of iterations is increased by 1, the next iteration is performed according to S206~217.

[0190] S218. Generate a three-dimensional reconstruction model of the target plant based on the target Gaussian ellipsoid set.

[0191] It can be seen from the above technical scheme that, unlike the traditional method of rendering in sequence according to the size of depth values, this scheme introduces the depth weight of the Gaussian ellipsoid calculated by depth image on the basis of traditional 3D Gaussian sputtering technology. The depth weight is inversely correlated with the depth value of the Gaussian ellipsoid, so that the key points close to the depth camera (that is, the key points with smaller depth values) have a greater influence factor on the rendering color of the pixel points. Introducing the depth weight into the rendering formula effectively reflects the true distribution of the point cloud in three-dimensional space, enhances the distinction of depth information in subtle areas, and more accurately controls the contribution of Gaussian points at different depth levels, ensuring that the detail information of the foreground is better retained during rendering, thereby improving the geometric accuracy and layering of the three-dimensional reconstructed model of the plant, and improving the restoration of the three-dimensional reconstructed model to the plant.

[0192] Furthermore, this solution divides the depth value of the target Gaussian ellipsoid into depth intervals and renders the color of the pixel points in units of depth intervals. This not only retains the global depth sorting characteristics, but also ensures the contribution of the target Gaussian ellipsoid in each depth interval. Furthermore, it more accurately reflects the relative position of the target Gaussian ellipsoid in the three-dimensional space. Through depth interval division, it can better handle areas with similar depths but complex geometric features, ultimately improving the geometric accuracy and visual consistency of the rendering results.

[0193] Furthermore, this scheme is based on the 3D Gaussian splash differentiable rendering technology suitable for depth image and color image fusion data, combining the image information of the depth image and the color image, thereby enhancing the usability of the 3D Gaussian splash technology and its application scenarios in the three-dimensional reconstruction of agricultural plants.

[0194] It should be noted that Figure 2 This is only a specific implementation process of a plant three-dimensional reconstruction method provided in an embodiment of the present application. The present application can also be implemented through a variety of other optional specific implementation processes. For example, S211~S214 is only an optional method for calculating the depth weight of the target Gaussian ellipsoid. In other optional embodiments, the depth weight of the target Gaussian ellipsoid is inversely proportional to the depth value of the target Gaussian ellipsoid. The quotient of 1 divided by the depth value of the target Gaussian ellipsoid can be used as the depth weight of the target Gaussian ellipsoid.

[0195] For another example, in an optional embodiment, a nonlinear method may be used to calculate the equation group of the three-dimensional coordinates of the key point P in the second color image. The specific method is as follows:

[0196] Define the energy function as shown in formula (15) and formula (16):

[0197] E = d(p1,K[I,0]P)+d(p2,K[R2,t2]P) (15);

[0198] P = argminE(16);

[0199] Among them, d(p1,K[I,0]P) represents the distance between the 3D point with world coordinates P projected to the pixel point on the image where p1 is located and p1, and d(p2,K[R2,t2]P) represents the distance between the 3D point with world coordinates P projected to the pixel point on the image where p2 is located and p2. The 3D coordinates of P are solved using Newton's method or Levenberg-Marquardt method.

[0200] For another example, S215 to S216 are only an optional method for obtaining the rendering color of a pixel point. In other optional embodiments, the depth weight of the target Gaussian ellipsoid is used as a weighting coefficient, and the Gaussian sputtering rendering results of each target Gaussian ellipsoid are weighted added to obtain the rendering color of the pixel point. Taking the target pixel point corresponding to M target Gaussian ellipsoids as an example, the rendering color of the target pixel point is calculated. The specific method is shown in formula (17):

[0201] (17);

[0202] in, The first pixel in the target Gaussian ellipsoid sequence represents the target pixel point. The target Gaussian ellipsoid G The color value of Represents the target Gaussian ellipsoid G The depth weight of Represents the target Gaussian ellipsoid G The opacity, Represents the opacity of the target Gaussian ellipsoid Gr, r is not greater than m.

[0203] For another example, S202~S205 is an optional method for initializing a Gaussian ellipsoid set based on a color image set of the target plant. In an optional embodiment, a multi-view point cloud stitching method can be used to obtain three-dimensional point cloud data or a SLAM algorithm can be used to obtain three-dimensional point cloud data. In another optional embodiment, algorithms such as ORB or SuperPoint can be used to extract key points and descriptors of key points in color images.

[0204] For another example, in the rendering process of 3D Gaussian sputtering, high-quality details and edge sharpening are crucial to the rendering effect. High-quality details can highlight details, make textures, colors and shapes easier to identify, enhance the realism and three-dimensionality of objects, while edge sharpening strengthens the outline of objects, making the boundaries between different objects more obvious, thereby reducing the sense of blur. High-quality images not only enhance visual appeal, but also optimize rendering performance. Therefore, in an optional embodiment, a rendering optimization step is added before the rendering of 3D Gaussian sputtering, and a high-resolution reconstruction network is used to improve image quality. Specifically, the high-resolution reconstruction network HRNet is a generative adversarial network model for image super-resolution. Image super-resolution technology can effectively reduce noise in images, thereby reducing the point cloud noise generated by the reconstruction system. Combined with the perceptual optimization strategy of the generative adversarial network, in order to enable the network model to output the original resolution result, the basic residual unit is replaced by a dense residual unit, and the upsampling unit and batch normalization unit are removed, so that the model can provide clearer edges and more visually attractive images, so that the radiation field can more effectively extract the structural features of the tomato plant, thereby more accurately completing the 3D reconstruction task of the tomato plant. Therefore, performing this rendering optimization step before the SFM initial point cloud reconstruction can not only improve the quality of the SFM reconstructed point cloud, but also improve the rendering effect of the image during the Gaussian sputtering process. Without changing the image resolution, the image detail quality and edge sharpness are improved. This high-quality image not only enhances the visual appeal, but also effectively improves the rendering effect of the 3D reconstructed plant model generated by the 3D Gaussian sputtering technology.

[0205] A three-dimensional plant reconstruction method provided in an embodiment of the present application is introduced above, and a device for executing the above-mentioned three-dimensional plant reconstruction method will be introduced below.

[0206] See also Figure 3 , Figure 3 The schematic diagram of the structure of a plant 3D reconstruction device provided in the embodiment of the present application is shown in FIG. The plant 3D reconstruction device is configured in the client, such as Figure 3 As shown, the plant three-dimensional reconstruction device 300 includes:

[0207] The point cloud data construction unit 301 is used to obtain three-dimensional point cloud data based on a target color image sequence of a target plant; wherein the target color image sequence includes a plurality of color images under different viewing angles arranged in time sequence, and the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of a plurality of key points;

[0208] A Gaussian initialization unit 302 is used to initialize the Gaussian ellipsoid of each of the key points according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set, wherein the Gaussian ellipsoid set includes the Gaussian ellipsoid of each of the key points, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity;

[0209] The Gaussian iterative updating unit 303 is used to perform iteration to update the Gaussian ellipsoid set until a preset iteration termination condition is reached to obtain the updated Gaussian ellipsoid set;

[0210] A three-dimensional modeling unit 304 is used to construct a three-dimensional reconstruction model of the target plant based on the updated Gaussian ellipsoid set;

[0211] Wherein, when the Gaussian update unit is used to perform the iteration, it is specifically used to:

[0212] A color image selected from the target color image sequence as the iterative training color image;

[0213] For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, the depth value of each target Gaussian ellipsoid is obtained; the target Gaussian ellipsoid of the pixel point is a Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point;

[0214] Calculating the depth weight of each of the target Gaussian ellipsoids based at least on the depth value; wherein the depth weight is inversely correlated with the depth value;

[0215] For each of the pixel points, based on the depth weight and Gaussian parameters of each of the target Gaussian ellipsoids, a rendering color value of the pixel point is calculated;

[0216] Rendering to obtain a rendered image of the target color image based on the rendered color value of each pixel of the training color image;

[0217] The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

[0218] In a possible implementation, the point cloud data construction unit is used to obtain three-dimensional point cloud data based on the target color image sequence of the target plant, specifically for:

[0219] Acquire the original color image sequence of the target plant through a depth camera, and filter the original color image sequence based on a preset image quality index to obtain the target color image sequence, wherein the image quality index includes a blur detection probability;

[0220] For each color image in the target color image sequence, using a scale-invariant feature transformation matching algorithm to extract key points of the color image and descriptors of the key points;

[0221] Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs of every two color images;

[0222] Three-dimensional point cloud data of the target plant is generated based on multiple pairs of key points of every two color images.

[0223] In a possible implementation, the point cloud data construction unit is used to generate the three-dimensional point cloud data of the target plant according to the multiple pairs of key points of each two color images, specifically for:

[0224] Starting from time sequence n=2, traverse the color images in the target color image sequence from small to large:

[0225] When n=2, based on the pixel coordinates of the key point pairs of the second color image and the first color image, an eight-point algorithm is used to calculate a basic matrix, and the basic matrix is ​​decomposed to obtain an extrinsic parameter matrix of the second color image, where the extrinsic parameter matrix includes a rotation matrix and a translation vector;

[0226] Calculate the three-dimensional coordinates of each key point in the second color image based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters and the extrinsic parameter matrix of the second color image using a triangulation method;

[0227] When n>2, based on the key point pairs of the nth color image and the n-1th color image, the perspective n-point PnP algorithm is used to calculate the extrinsic parameter matrix of the nth color image;

[0228] Calculate the three-dimensional coordinates of each key point in the nth color image based on the pixel coordinates of each key point pair in the nth color image and the n-1th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the n-1th color image, and the extrinsic parameter matrix of the nth color image using a triangulation method;

[0229] The three-dimensional point cloud data is generated according to the three-dimensional coordinate value and color value of each key point in each color image.

[0230] In a possible implementation, when the Gaussian iterative update unit is used to calculate the depth weight of each target Gaussian ellipsoid based on at least the depth value, it is specifically used to:

[0231] Divide a plurality of depth intervals according to a minimum depth value and a maximum depth value to obtain a depth interval sequence, wherein the depth interval sequence includes a plurality of depth intervals arranged from small to large according to the depth value;

[0232] For each of the depth intervals, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order according to the depth values ​​to obtain an ellipsoid subsequence corresponding to the depth interval;

[0233] For each of the depth intervals, based on the order of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value, calculate the depth center of the depth interval;

[0234] For each target Gaussian ellipsoid in the ellipsoid subsequence of each depth interval, a depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

[0235] In a possible implementation, the Gaussian iterative update unit is used to calculate the rendering color value of each pixel point based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid, specifically for:

[0236] Projecting each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtaining the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each of the target Gaussian ellipsoids based on the Gaussian parameters of the Gaussian ellipsoid;

[0237] Calculate the opacity of the two-dimensional Gaussian ellipse based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse;

[0238] For each of the depth intervals, based on the color value of the target Gaussian ellipsoid within the depth interval, the depth weight, and the opacity of the two-dimensional Gaussian ellipse, calculating a rendering color value of the depth interval;

[0239] The rendering color value of the pixel is calculated based on the rendering color values ​​of all the depth intervals of the pixel.

[0240] In a possible implementation, the Gaussian iterative update unit is used to calculate the rendering color value of the target depth interval based on the color value, depth weight, and opacity of the two-dimensional Gaussian ellipse in the target depth interval, and when the target depth interval is any one of the depth intervals, specifically for:

[0241] For a target Gaussian ellipsoid within the target depth interval, weighted parameters of each previous Gaussian ellipsoid of the target Gaussian ellipsoid are cumulatively multiplied to obtain a cumulative weighted parameter, wherein the previous Gaussian ellipsoid is a Gaussian ellipsoid with a smaller rank than the target Gaussian ellipsoid in the ellipsoid subsequence within the target depth interval, and the cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the previous Gaussian ellipsoid, and the weighted two-dimensional opacity is equal to the depth weight of the previous Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse;

[0242] For a target Gaussian ellipsoid within the target depth interval, calculating the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain an opacity parameter;

[0243] For a target Gaussian ellipsoid within the target depth interval, calculating the product of the color value and the opacity parameter of the target Gaussian ellipsoid as the color to be rendered;

[0244] The depth weight of each target Gaussian ellipsoid is used as a weight coefficient, and weighted addition is performed on the colors to be rendered of each target Gaussian ellipsoid to obtain a rendering color value of the target depth interval.

[0245] It should be noted that the specific structure and function of the plant 3D reconstruction device can be found in the above embodiments.

[0246] The present application also provides an electronic device in an embodiment. Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0247] like Figure 4 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 to a random access memory (RAM) 403. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 403. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0248] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a memory card, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0249] Also provided in an embodiment of the present application is a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the plant three-dimensional reconstruction methods provided in the embodiments of the present application.

[0250] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the plant three-dimensional reconstruction methods provided in the embodiments of the present application.

[0251] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0252] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0253] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0254] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for three-dimensional reconstruction of a plant, characterized in that: include: Acquire three-dimensional point cloud data based on a target color image sequence of a target plant; wherein the target color image sequence includes a plurality of color images at different viewing angles arranged in time sequence, and the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of a plurality of key points; Initializing the Gaussian ellipsoid of each of the key points according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set, wherein the Gaussian ellipsoid set includes the Gaussian ellipsoid of each of the key points, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity; Performing iteration to update the Gaussian ellipsoid set until a preset iteration termination condition is reached to obtain the updated Gaussian ellipsoid set; Based on the updated Gaussian ellipsoid set, constructing a three-dimensional reconstruction model of the target plant; The iteration includes: A color image selected from the target color image sequence as the iterative training color image; For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, the depth value of each target Gaussian ellipsoid is obtained; the target Gaussian ellipsoid of the pixel point is a Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point; Calculating the depth weight of each of the target Gaussian ellipsoids based at least on the depth value; wherein the depth weight is inversely correlated with the depth value; For each of the pixel points, based on the depth weight and Gaussian parameters of each of the target Gaussian ellipsoids, a rendering color value of the pixel point is calculated; Rendering to obtain a rendered image of the target color image based on the rendered color value of each pixel of the training color image; The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

2. The plant three-dimensional reconstruction method according to claim 1, characterized in that: The method of acquiring three-dimensional point cloud data based on a target color image sequence of a target plant comprises: Acquire the original color image sequence of the target plant through a depth camera, and filter the original color image sequence based on a preset image quality index to obtain the target color image sequence, wherein the image quality index includes a blur detection probability; For each color image in the target color image sequence, using a scale-invariant feature transformation matching algorithm to extract key points of the color image and descriptors of the key points; Based on the descriptor, key point matching is performed on every two color images in the target color image sequence to obtain multiple pairs of key point pairs of every two color images; Three-dimensional point cloud data of the target plant is generated based on multiple pairs of key points of every two color images.

3. The plant three-dimensional reconstruction method according to claim 2, characterized in that: The step of generating the three-dimensional point cloud data of the target plant based on the multiple pairs of key points of each two color images includes: Starting from time sequence n=2, traverse the color images in the target color image sequence from small to large: When n=2, based on the pixel coordinates of the key point pairs of the second color image and the first color image, an eight-point algorithm is used to calculate a basic matrix, and the basic matrix is ​​decomposed to obtain an extrinsic parameter matrix of the second color image, where the extrinsic parameter matrix includes a rotation matrix and a translation vector; Calculate the three-dimensional coordinates of each key point in the second color image based on the pixel coordinates of each key point pair in the second color image and the first color image, the camera intrinsic parameters and the extrinsic parameter matrix of the second color image using a triangulation method; When n>2, based on the key point pairs of the nth color image and the n-1th color image, the perspective n-point PnP algorithm is used to calculate the extrinsic parameter matrix of the nth color image; Calculate the three-dimensional coordinates of each key point in the nth color image based on the pixel coordinates of each key point pair in the nth color image and the n-1th color image, the camera intrinsic parameters, the extrinsic parameter matrix of the n-1th color image, and the extrinsic parameter matrix of the nth color image using a triangulation method; The three-dimensional point cloud data is generated according to the three-dimensional coordinate value and color value of each key point in each color image.

4. The plant three-dimensional reconstruction method according to claim 1, characterized in that: The step of calculating the depth weight of each target Gaussian ellipsoid at least based on the depth value comprises: Divide a plurality of depth intervals according to a minimum depth value and a maximum depth value to obtain a depth interval sequence, wherein the depth interval sequence includes a plurality of depth intervals arranged from small to large according to the depth value; For each of the depth intervals, the target Gaussian ellipsoids with depth values ​​within the depth interval are sorted in ascending order according to the depth values ​​to obtain an ellipsoid subsequence corresponding to the depth interval; For each of the depth intervals, based on the order of the depth interval in the depth interval sequence, the number of depth intervals, the minimum depth value, and the maximum depth value, calculate the depth center of the depth interval; For each target Gaussian ellipsoid in the ellipsoid subsequence of each depth interval, a depth weight of the target Gaussian ellipsoid is calculated based on the depth center of the depth interval and the depth value of the target Gaussian ellipsoid.

5. The plant 3D reconstruction method according to claim 4, characterized in that: The step of calculating, for each pixel point, a rendering color value of the pixel point based on the depth weight and Gaussian parameters of each target Gaussian ellipsoid comprises: Projecting each of the target Gaussian ellipsoids onto the two-dimensional plane of the training image, and obtaining the mean vector and covariance matrix of the two-dimensional Gaussian ellipse of each of the target Gaussian ellipsoids based on the Gaussian parameters of the Gaussian ellipsoid; Calculate the opacity of the two-dimensional Gaussian ellipse based on the opacity of the target Gaussian ellipsoid and the mean vector and covariance matrix of the two-dimensional Gaussian ellipse; For each of the depth intervals, based on the color value of the target Gaussian ellipsoid within the depth interval, the depth weight, and the opacity of the two-dimensional Gaussian ellipse, a rendering color value of the depth interval is calculated; The rendering color value of the pixel is calculated based on the rendering color values ​​of all the depth intervals of the pixel.

6. The plant three-dimensional reconstruction method according to claim 5, characterized in that: Calculating a rendering color value of a target depth interval based on a color value of the target Gaussian ellipsoid within the target depth interval, a depth weight, and an opacity of the two-dimensional Gaussian ellipse, wherein the target depth interval is any one of the depth intervals, including: For a target Gaussian ellipsoid within the target depth interval, weighted parameters of each previous Gaussian ellipsoid of the target Gaussian ellipsoid are cumulatively multiplied to obtain a cumulative weighted parameter, wherein the previous Gaussian ellipsoid is a Gaussian ellipsoid with a smaller rank than the target Gaussian ellipsoid in the ellipsoid subsequence within the target depth interval, and the cumulative weighted parameter is equal to 1 minus the weighted two-dimensional opacity of the previous Gaussian ellipsoid, and the weighted two-dimensional opacity is equal to the depth weight of the previous Gaussian ellipsoid multiplied by the opacity of the two-dimensional Gaussian ellipse; For a target Gaussian ellipsoid within the target depth interval, calculating the product of the cumulative weighted parameter of the target Gaussian ellipsoid and the opacity of the two-dimensional Gaussian ellipse to obtain an opacity parameter; For a target Gaussian ellipsoid within the target depth interval, calculating the product of the color value and the opacity parameter of the target Gaussian ellipsoid as the color to be rendered; The depth weight of each target Gaussian ellipsoid is used as a weight coefficient, and weighted addition is performed on the colors to be rendered of each target Gaussian ellipsoid to obtain a rendering color value of the target depth interval.

7. A plant three-dimensional reconstruction device, characterized in that: include: A point cloud data construction unit, used to obtain three-dimensional point cloud data based on a target color image sequence of a target plant; wherein the target color image sequence includes a plurality of color images under different viewing angles arranged in time sequence, and the three-dimensional point cloud data includes three-dimensional coordinate values ​​and color values ​​of a plurality of key points; A Gaussian initialization unit, used for initializing the Gaussian ellipsoid of each of the key points according to the three-dimensional point cloud data to obtain a Gaussian ellipsoid set, wherein the Gaussian ellipsoid set includes the Gaussian ellipsoids of each of the key points, and the Gaussian parameters of the Gaussian ellipsoid include a mean vector, a covariance matrix, a color value, and opacity; A Gaussian iterative updating unit, used for performing iteration to update the Gaussian ellipsoid set until a preset iteration termination condition is reached to obtain the updated Gaussian ellipsoid set; A three-dimensional modeling unit, used for constructing a three-dimensional reconstruction model of the target plant based on the updated Gaussian ellipsoid set; Wherein, when the Gaussian iterative update unit is used to perform the iteration, it is specifically used to: A color image selected from the target color image sequence as the iterative training color image; For each pixel point in the training color image, based on the depth image corresponding to each target Gaussian ellipsoid of the pixel point, the depth value of each target Gaussian ellipsoid is obtained; the target Gaussian ellipsoid of the pixel point is a Gaussian ellipsoid in the Gaussian ellipsoid set that covers the pixel point; Calculating the depth weight of each of the target Gaussian ellipsoids based at least on the depth value; wherein the depth weight is inversely correlated with the depth value; For each of the pixel points, based on the depth weight and Gaussian parameters of each of the target Gaussian ellipsoids, a rendering color value of the pixel point is calculated; Rendering to obtain a rendered image of the target color image based on the rendered color value of each pixel of the training color image; The Gaussian ellipsoid set is updated based on the loss values ​​of the training color image and the rendered image.

8. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the three-dimensional plant reconstruction method as claimed in any one of claims 1 to 6.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the plant three-dimensional reconstruction method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the plant three-dimensional reconstruction method as described in any one of claims 1 to 6.

Images