A tea bud leaf target three-dimensional point cloud prediction method based on depth information

Abstract

The application provides a tea bud leaf target three-dimensional point cloud prediction method based on depth information, which comprises collecting an RGB image sequence and a depth image sequence of tea bud leaves, and performing instance segmentation on the RGB image sequence by using a tea leaf visual model. Total matching loss is extracted according to the RGB image sequence and the depth image sequence, camera poses are estimated according to the total matching loss, and the camera poses are optimized. The three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system are judged to obtain tea bud leaf camera point cloud. The tea bud leaf camera point cloud is projected into a three-dimensional world coordinate system to obtain tea bud leaf target three-dimensional point cloud. The application minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain the optimized camera pose, and on this basis, the accurate three-dimensional information of the target in the field of view is obtained by combining the depth information, the point cloud data under multiple perspectives is spliced, and thus the more complete tea bud leaf three-dimensional point cloud is reconstructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a tea bud leaf target three-dimensional point cloud prediction method based on depth information. BACKGROUND

[0002] In an unstructured natural tea garden environment, a famous tea picking robot needs to plan a picking strategy according to the perception information of a vision system. The target position obtained from a two-dimensional image cannot guide the picking robot to work in a three-dimensional space. Therefore, the picking robot needs to have more accurate local three-dimensional perception of the picking target to obtain the local three-dimensional point cloud of the picking target. The local three-dimensional point cloud of the picking target can help the picking robot accurately identify the three-dimensional spatial position angle of the picking target and the relative position relationship between the picking target and the picking mechanism, thereby providing decision support for the picking robot to successfully complete the picking work.

[0003] Traditional target local three-dimensional perception methods usually adopt a feature fusion-based method, the core of which is to obtain local point cloud by using color, texture and shape features of the target. However, due to the complex lighting conditions of the tea garden and the growth characteristics of the tea leaves, the imaging scales of targets at different distances in the same field of view are different, and the poses are different, which causes the tea buds to present various morphological characteristics under the camera view. It is difficult for the vision system to obtain accurate three-dimensional information of the target in the field of view by relying only on the feature fusion-based method. At the same time, the images collected by the depth camera are often disturbed by noise, resulting in the loss of some depth information in the depth image. Therefore, it is a significant challenge to rely only on the image information obtained from a single view to construct a point cloud of the tea bud leaf with a complete morphology.

[0004] In summary, it is necessary to splice the point cloud data under multiple views to reconstruct a more complete three-dimensional point cloud of the tea bud leaf. SUMMARY

[0005] To overcome the problems in the related art, the purpose of the present application is to provide a tea bud leaf target three-dimensional point cloud prediction method based on depth information, which splices the point cloud data under multiple views to reconstruct a more complete three-dimensional point cloud of the tea bud leaf.

[0006] A tea bud leaf target three-dimensional point cloud prediction method based on depth information, comprising:

[0007] collecting an RGB image sequence and a depth image sequence of the tea bud leaf;

[0008] performing instance segmentation on the RGB image sequence by using a tea leaf vision model to obtain a tea bud leaf mask image sequence;

[0009] extracting a total matching loss according to the RGB image sequence and the depth image sequence;

[0010] estimate a camera pose according to the total matching loss;

[0011] optimize the camera pose to obtain an optimized camera pose;

[0012] determine three-dimensional coordinates of each pixel point in the tea leaf mask image in a camera coordinate system in combination with an RGB image sequence, a depth image sequence and a tea bud leaf mask image sequence, to obtain a tea bud leaf camera point cloud;

[0013] project the tea bud leaf camera point cloud into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud.

[0014] In the preferred technical solution of the present application, the total matching loss is extracted according to the RGB image sequence and the depth image sequence, comprising:

[0015] construct an image pyramid based on the RGB image sequence, the image pyramid being used to represent a scale space of the image;

[0016] extract ORB features of the RGB image in the image pyramid, and calculate a normalized ORB vector difference according to the ORB features;

[0017] extract gradient feature differences of the depth image from the depth image sequence;

[0018] weight and fuse the ORB vector difference and the gradient feature differences of the depth image to obtain the total matching loss.

[0019] In the preferred technical solution of the present application, the gradient feature differences of the depth image are extracted from the depth image sequence, comprising:

[0020] construct a horizontal convolution kernel and a vertical convolution kernel;

[0021] calculate horizontal gradients of the depth image by using the horizontal convolution kernel, and calculate vertical gradients of the depth image by using the vertical convolution kernel; wherein the depth image sequence comprises a plurality of depth images;

[0022] normalize the horizontal gradients to obtain normalized horizontal gradients, and normalize the vertical gradients to obtain normalized vertical gradients;

[0023] calculate an inverse tangent function of a ratio of the normalized vertical gradients to the normalized horizontal gradients to obtain a depth gradient direction angle;

[0024] calculate the gradient feature differences of the depth image according to the depth gradient direction angle.

[0025] In the preferred technical solution of the present application, the normalized ORB vector difference is calculated according to the ORB features, comprising:

[0026] The normalized ORB vector difference between different frame RGB images is calculated according to the following formula:

[0027] ;

[0028] d H (g k1 ,g k2 )= ;

[0029] wherein g k1 represents the k1th ORB vector, g k2 represents the k2th ORB vector, represents the length of the k1th ORB vector, represents the ORB vector difference between the k1th ORB vector and the k2th ORB vector, d H (g k1 ,g k2 ) represents the normalized ORB vector difference between the k1th ORB vector and the k2th ORB vector; i represents the element sequence number in the ORB vector, and m represents the total number of elements in the ORB vector, represents the i-th element in the k1th ORB vector, represents the i-th element in the k2th ORB vector.

[0030] In the preferred technical scheme of the present application, the camera pose is estimated according to the total matching loss, comprising:

[0031] searching for the minimum value in all total matching losses to obtain the minimum total matching loss;

[0032] screening out the first target ORB vector and the second target ORB vector corresponding to the minimum total matching loss;

[0033] counting the matching point pairs of the first target ORB vector and the second target ORB vector;

[0034] based on the matching point pairs, using the re-projection error to inversely deduce the rotation matrix and the translation matrix of the depth camera; wherein the rotation matrix and the translation matrix constitute the camera pose.

[0035] In the preferred technical scheme of the present application, the camera pose is optimized to obtain an optimized camera pose, comprising:

[0036] constructing a relationship between the camera pose and the re-projection error;

[0037] calculating the sum of all re-projection errors to obtain a total re-projection error;

[0038] Minimize the total weight projection error, and use the camera pose corresponding to the minimum value of the total weight projection error as the optimized camera pose.

[0039] In a preferred embodiment of the present invention, the step of using reprojection error to derive the rotation and translation matrices of the depth camera includes:

[0040] Based on the following formula, the relationship between camera pose and reprojection error is constructed using the bundle adjustment algorithm:

[0041] ;

[0042] Among them, e i Let u represent the i-th reprojection error. i s represents the projected coordinates of the i-th pixel in the world coordinate system of the tea bud and leaf mask image. i Let P represent the scale factor, K represent the camera intrinsic parameter matrix, T represent the camera pose matrix, and P represent the camera pose matrix. i The i-th pixel in the tea bud and leaf mask image is represented by the camera pose matrix, which includes a rotation matrix and a translation matrix.

[0043] Based on the i-th reprojection error, the i-th rotation matrix and the i-th translation matrix of the depth camera are derived in reverse.

[0044] In a preferred embodiment of the present invention, the step of combining the RGB image sequence, the depth image sequence, and the tea bud and leaf mask image sequence to determine the three-dimensional coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system, and obtaining the tea bud and leaf camera point cloud, includes:

[0045] The depth image sequence is matched with the tea bud and leaf mask image sequence, wherein the depth image sequence includes multiple depth images and the tea bud and leaf mask image sequence includes multiple tea bud and leaf mask images;

[0046] Obtain the depth value of each pixel in the corresponding tea bud and leaf mask image from the depth image;

[0047] Based on the depth value, the three-dimensional spatial coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system are calculated using the following formula:

[0048] ;

[0049] in, Let be the z-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let x be the x-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let y be the y-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system;

[0050] The coordinates of all pixels in the camera coordinate system are converted into a point cloud of tea buds and leaves; the point cloud of tea buds and leaves is used to represent the complete shape of the tea bud and leaf target.

[0051] In a preferred embodiment of the present invention, the step of calculating the gradient feature difference of the depth image based on the depth gradient direction angle includes:

[0052] The gradient feature differences of depth images are calculated using the following formula:

[0053] ;

[0054] ;

[0055] in, This represents the difference in depth gradient direction angle between the i-th depth image and the j-th depth image, with min indicating the minimum value operation. This represents the depth gradient direction angle of the i-th depth image. This represents the depth gradient orientation angle of the j-th depth image; Let represent the gradient feature difference between the i-th depth image and the j-th depth image, and let cos represent the cosine function.

[0056] In a preferred embodiment of the present invention, after obtaining the three-dimensional point cloud of the tea bud and leaf target, the method further includes:

[0057] Bilateral filtering was used to remove noise from the 3D point cloud of tea buds and leaves, resulting in a denoised 3D point cloud.

[0058] Statistical filtering is used to remove outliers from the denoised 3D point cloud to obtain an optimized 3D point cloud.

[0059] The beneficial effects of this invention are as follows:

[0060] The present invention provides a method for predicting 3D point clouds of tea buds and leaves based on depth information. This method involves acquiring RGB image sequences and depth image sequences of tea buds and leaves. The depth image sequence is a collection of multiple consecutive depth images, each representing the distance from different regions in the scene to the depth camera. A tea leaf visual model is used to perform instance segmentation on the RGB image sequences, resulting in a tea bud and leaf mask image sequence. Instance segmentation aims to achieve target detection and pixel classification, i.e., identifying the object category in each RGB image and generating an independent binary mask for it. A total matching loss is extracted from the RGB image sequence and the depth image sequence, and the camera pose is estimated based on the total matching loss. ORB features are an efficient feature detection and description method, possessing both rotation invariance and scale invariance. The advantages of ORB features are utilized to match associated frames, thereby accurately estimating the camera pose. By combining the RGB image sequence, the depth image sequence, and the tea bud and leaf mask image sequence, the 3D coordinates of each pixel in the tea leaf mask image in the camera coordinate system are determined, resulting in the tea bud and leaf camera point cloud. The tea bud and leaf camera point cloud is projected onto a 3D world coordinate system to obtain the 3D point cloud of the tea bud and leaf target. This invention minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain the optimized camera pose. Based on this, it combines depth information to obtain accurate three-dimensional information of the target within the field of view. It then stitches together point cloud data from multiple perspectives to reconstruct a more complete three-dimensional point cloud of tea buds and leaves. Attached Figure Description

[0061] Figure 1 This is a flowchart of the three-dimensional point cloud prediction method for tea buds and leaves based on depth information according to the present invention;

[0062] Figure 2 This is a flowchart of the total matching loss for extracting the masked image sequence of tea buds and leaves according to the present invention;

[0063] Figure 3 This is a flowchart of the present invention for estimating camera pose based on total matching loss;

[0064] Figure 4 This is a diagram showing the result of inter-frame matching based on the total matching loss according to the present invention. Detailed Implementation

[0065] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0066] Example 1

[0067] likeFigure 1 As shown, this embodiment provides a method for predicting the three-dimensional point cloud of tea buds and leaves based on depth information, including:

[0068] S1: Acquire RGB image sequences and depth image sequences of tea buds and leaves;

[0069] S2: Use a tea visual model to segment the RGB image sequence to obtain a tea bud and leaf mask image sequence;

[0070] S3: Extract the total matching loss based on the RGB image sequence and the depth image sequence;

[0071] S4: Estimate the camera pose based on the total matching loss;

[0072] S5: Optimize the camera pose to obtain the optimized camera pose;

[0073] S6: Combine the RGB image sequence, depth image sequence and tea bud and leaf mask image sequence to determine the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system, and obtain the camera point cloud of tea buds and leaves.

[0074] S7: Project the point cloud of the tea bud and leaf camera onto the three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target.

[0075] like Figure 2 As shown, the step of extracting the total matching loss based on the RGB image sequence and the depth image sequence includes:

[0076] S31: Construct an image pyramid based on an RGB image sequence, wherein the image pyramid is used to represent the scale space of the image;

[0077] S32: Extract the ORB features of the RGB image from the image pyramid, and calculate the difference of the normalized ORB vector based on the ORB features;

[0078] S33: Extract gradient feature differences from depth image sequences;

[0079] S34: Weighted fusion of ORB vector differences and gradient feature differences of depth images to obtain the total matching loss.

[0080] The image pyramid is composed of multi-resolution sub-images of the same original image, with high-resolution features at the bottom and low-resolution features at the top. In this embodiment, each tea bud and leaf mask image in the tea bud and leaf mask image sequence is downsampled multiple times. Each downsampling reduces the length and width of the image to half of their original values, i.e., the total number of pixels is reduced to 1 / 4 of the original value, thereby extracting features at different resolutions.

[0081] ORB is a high-efficiency feature point detection and description algorithm that combines the advantages of FAST corner detection and BRIEF descriptors, and optimizes for rotation invariance and computational efficiency. The ORB algorithm calculates the feature point orientation using the gray-scale centroid method, taking the offset vector between the feature point and the centroid as the orientation, thus achieving rotation invariance. Multi-scale detection is achieved by constructing a Gaussian pyramid, making the algorithm scale invariant. The ORB algorithm utilizes the orientation information from FAST corner detection to rotate the sampling pattern of the point pairs, thereby enhancing rotation invariance.

[0082] The RGB image sequence contains multiple RGB images targeting tea buds and leaves; the depth image sequence contains multiple depth images reflecting the depth information of tea buds and leaves; and the tea bud and leaf mask image sequence contains multiple mask images of tea buds and leaves. The ORB feature of each pixel in the RGB images is calculated, resulting in multiple ORB features. The number of ORB features is the same as the total number of pixels in the RGB images, and each ORB feature corresponds to a single pixel in the RGB images. All ORB features of an RGB image are combined into an ORB vector; for example, the k1-th ORB vector is represented as... The k2th ORB vector is represented as The total matching loss is calculated using the following formula:

[0083] ;

[0084] Among them, D total This represents the total matching loss. Represents the first fusion coefficient. This represents the second fusion coefficient. + =1, This represents the ORB vector difference between the i-th RGB image and the j-th RGB image. This represents the gradient feature difference between the i-th depth image and the j-th depth image. The ORB vector difference reflects the two-dimensional pose difference of tea buds and leaves in different frames of RGB images, and the gradient feature difference of the depth images reflects the pose difference of tea buds and leaves in the depth direction in different frames of depth images, thus accurately representing the pose of tea buds and leaves in three-dimensional space.

[0085] The step of calculating the difference in the normalized ORB vector based on the ORB features includes:

[0086] The difference in normalized ORB vectors between different frames of RGB images is calculated using the following formula:

[0087] ;

[0088] d H (g k1 ,gk2 )= ;

[0089] Among them, g k1 Let g represent the k1-th ORB vector. k2 This represents the k2th ORB vector. This represents the length of the k1-th ORB vector. d represents the difference between the k1-th and k2-th ORB vectors. H (g k1 ,g k2 The expression () represents the difference between the normalized ORB vectors of the k1-th and k2-th ORB vectors; i represents the element index in the ORB vector, and m represents the total number of elements in the ORB vector. This represents the i-th element in the k1-th ORB vector. This represents the i-th element in the k2-th ORB vector. Based on the ORB vector composed of multiple ORB features, the distance between two ORB vectors is measured by the norm of the difference between them. Then, it is divided by the length of the corresponding ORB vector to obtain the normalized ORB vector difference, which has the advantage of high computational efficiency.

[0090] The extraction of gradient feature differences from depth image sequences includes:

[0091] S331: Construct horizontal and vertical convolutional kernels;

[0092] S332: The horizontal gradient of the depth image is calculated using the horizontal convolution kernel, and the vertical gradient of the depth image is calculated using the vertical convolution kernel; wherein, the depth image sequence includes multiple depth images;

[0093] S333: Normalize the horizontal gradient to obtain a normalized horizontal gradient; normalize the vertical gradient to obtain a normalized vertical gradient;

[0094] S334: Calculate the arctangent function of the ratio of the normalized vertical gradient to the normalized horizontal gradient to obtain the depth gradient direction angle;

[0095] S335: Calculate the gradient feature difference of the depth image based on the depth gradient direction angle.

[0096] The horizontal and vertical gradients of the depth image are calculated using the following formulas:

[0097] ;

[0098] ;

[0099] ;

[0100] ;

[0101] Among them, S x For horizontal convolution kernels, S y Z is a vertical convolution kernel, Z is the depth image, Z x Z represents the horizontal gradient of the depth image. y The vertical gradient of the depth image. The horizontal convolution kernel S. x Used to detect gradient changes in the depth image along the x-direction, vertical convolution kernel S y Used to detect gradient changes in a depth image along the y-direction.

[0102] The horizontal and vertical gradients are normalized using the following formulas:

[0103] ;

[0104] ;

[0105] ;

[0106] in, This represents the normalized horizontal gradient. This represents the normalized vertical gradient. u represents the horizontal coordinate (row number) of the depth image, and v represents the vertical coordinate (column number). This represents the pixel neighborhood centered at feature point p with a radius of r. This represents the value of the point in the u-th row and v-th column of the horizontal gradient. This represents the value of the point in the u-th row and v-th column of the vertical gradient. Let represent the Gaussian weight, exp represent the exponential function, u0 be the x-coordinate of feature point p, and v0 be the y-coordinate of feature point p. This represents the Gaussian weighting adjustment parameter. .

[0107] The depth gradient direction angle is calculated using the following formula:

[0108] ;

[0109] in, This represents the depth gradient direction angle of feature point p. The value of is [0, π). This represents the arctangent function.

[0110] The step of calculating the gradient feature difference of the depth image based on the depth gradient direction angle includes:

[0111] The gradient feature differences of depth images are calculated using the following formula:

[0112] ;

[0113] ;

[0114] in, This represents the difference in depth gradient direction angle between the i-th depth image and the j-th depth image, with min indicating the minimum value operation. This represents the depth gradient direction angle of the i-th depth image. This represents the depth gradient orientation angle of the j-th depth image; Let represent the gradient feature difference between the i-th depth image and the j-th depth image, and let cos represent the cosine function.

[0115] The depth gradient orientation angle has both direction and extension. This embodiment combines two absolute value operations and a minimum value operation to limit the depth gradient orientation angle of two depth images to the range of 0-π / 2, thereby measuring the difference in the true gradient features of the two depth images. The cosine function of the difference in depth gradient orientation angle is calculated to normalize the difference in gradient features of the depth images to the range of 0-1, so as to unify the metric and characterize the pose difference of tea buds and leaves in the two depth images in terms of depth.

[0116] The method for predicting 3D point cloud images of tea buds and leaves based on depth information provided in this embodiment includes acquiring RGB image sequences and depth image sequences of tea buds and leaves. The depth image sequence is an image set composed of multiple consecutive depth images, each representing the distance from different regions in the scene to the depth camera. A tea leaf visual model is used to perform instance segmentation on the RGB image sequence to obtain a tea bud and leaf mask image sequence. Instance segmentation aims to achieve target detection and pixel classification, that is, to identify the object category in each RGB image and generate an independent binary mask for it. The total matching loss is extracted based on the RGB image sequence and the depth image sequence. Specifically, the two RGB image frames with the minimum total matching loss are paired, and the two depth image frames with corresponding numbers are also paired. For example, if the total matching loss is minimum between the 3rd and 5th frames, then the 3rd and 5th RGB image frames are paired, and the 3rd and 5th depth image frames are also paired. The camera pose is estimated based on the total matching loss. Figure 4 This is a diagram showing the result of inter-frame matching based on the total matching loss according to the present invention. Figure 4 As shown, Figure 4 The marked points in the left image and Figure 4 The points marked in the image on the right correspond one-to-one.

[0117] ORB features are an efficient feature detection and description method, possessing both rotation invariance and scale invariance. Utilizing these advantages, ORB features are matched with associated frames to accurately estimate camera pose. By combining RGB image sequences, depth image sequences, and tea bud mask image sequences, the 3D coordinates of each pixel in the tea bud mask image in the camera coordinate system are determined, resulting in a tea bud camera point cloud. This point cloud is then projected onto a 3D world coordinate system to obtain a 3D point cloud of the tea bud target. This invention uses iterative optimization to minimize the distance between the theoretical and actual projection points, obtaining an optimized camera pose. Based on this, depth information is combined to acquire accurate 3D information of the target within the field of view. The point cloud data from multiple perspectives are stitched together to reconstruct a more complete 3D point cloud of the tea bud.

[0118] This embodiment provides a method for predicting 3D point clouds of tea buds and leaves based on depth information. This includes acquiring RGB image sequences and depth image sequences of tea buds and leaves. The depth image sequence is a collection of multiple consecutive depth images, each representing the distance from different regions in the scene to the depth camera. A tea leaf visual model is used to perform instance segmentation on the RGB image sequence, resulting in a tea bud and leaf mask image sequence. Instance segmentation aims to achieve target detection and pixel classification, i.e., identifying the object category in each RGB image and generating an independent binary mask for it. A total matching loss is extracted from the RGB image sequence and the depth image sequence. The camera pose is estimated based on the total matching loss and optimized to obtain the optimized camera pose. ORB features are an efficient feature detection and description method, possessing both rotation invariance and scale invariance. The advantages of ORB features are utilized to match associated frames, thereby accurately estimating the camera pose. By combining the RGB image sequence, the depth image sequence, and the tea bud and leaf mask image sequence, the 3D coordinates of each pixel in the tea leaf mask image in the camera coordinate system are determined, resulting in the tea bud and leaf camera point cloud. The point cloud of tea buds and leaves is projected onto a three-dimensional world coordinate system to obtain a three-dimensional point cloud of the tea bud and leaf target. This invention minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain the optimized camera pose. Based on this, combined with depth information, accurate three-dimensional information of the target within the field of view is obtained. The point cloud data from multiple perspectives are stitched together to reconstruct a more complete three-dimensional point cloud of tea buds and leaves.

[0119] Example 2

[0120] like Figure 1 As shown, this embodiment describes the differences from Embodiment 1. This embodiment provides a method for predicting three-dimensional point clouds of tea buds and leaves based on depth information, including:

[0121] S1: Acquire RGB image sequences and depth image sequences of tea buds and leaves;

[0122] S2: Use a tea visual model to segment the RGB image sequence to obtain a tea bud and leaf mask image sequence;

[0123] S3: Extract the total matching loss based on the RGB image sequence and the depth image sequence;

[0124] S4: Estimate the camera pose based on the total matching loss;

[0125] S5: Optimize the camera pose to obtain the optimized camera pose;

[0126] S6: Combine the RGB image sequence, depth image sequence and tea bud and leaf mask image sequence to determine the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system, and obtain the camera point cloud of tea buds and leaves.

[0127] S7: Project the point cloud of the tea bud and leaf camera onto the three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target.

[0128] like Figure 3 As shown, estimating the camera pose based on the total matching loss includes:

[0129] S41: Search for the minimum value among all total matching losses to obtain the minimum total matching loss;

[0130] S42: Select the first target ORB vector and the second target ORB vector corresponding to the minimum total matching loss;

[0131] S43: Count the matching point pairs between the first target ORB vector and the second target ORB vector;

[0132] S44: Based on the matching point pair, the rotation matrix and translation matrix of the depth camera are derived in reverse using the reprojection error; wherein, the rotation matrix and the translation matrix constitute the camera pose.

[0133] The smaller the total matching loss, the more similar the ORB vectors of the two targets are, indicating a stronger temporal and spatial correlation between the RGB images of the two tea buds. The pixel coordinates and depth information of the matched point pairs are combined to obtain the 3D point coordinates. The reprojection error measures the deviation between the theoretical position of a point projected onto the 2D image and its actual observed position; minimizing the reprojection error optimizes the camera pose.

[0134] The optimization of the camera pose to obtain the optimized camera pose includes:

[0135] S51: Calculate the sum of all reprojection errors to obtain the total reprojection error;

[0136] S52: Minimize the total weight projection error, and take the camera pose corresponding to the minimum value of the total weight projection error as the optimized camera pose.

[0137] The relationship between camera pose and reprojection error is as follows:

[0138] ;

[0139] Among them, e i Let u represent the i-th reprojection error. i s represents the projected coordinates of the i-th pixel in the world coordinate system of the tea bud and leaf mask image. i Let P represent the scale factor, K represent the camera intrinsic parameter matrix, T represent the camera pose matrix, and P represent the camera pose matrix. i This represents the i-th pixel in the tea bud and leaf mask image. The camera pose matrix includes a rotation matrix and a translation matrix. R represents the rotation matrix, t represents the translation matrix, and the i-th rotation matrix and i-th translation matrix of the depth camera are derived by reverse engineering based on the i-th reprojection error.

[0140] Add all reprojection errors together, i.e. sum the polynomials, to obtain the total projection error. Take the camera pose corresponding to the minimum value of the total projection error as the optimized camera pose. The optimized camera pose is applicable to each frame of the RGB image sequence.

[0141] This embodiment uses a bundle adjustment algorithm to construct the relationship between camera pose and reprojection error. Bundle adjustment adjusts the camera pose and 3D point coordinates, including the camera's rotation and translation matrices, to minimize the sum of squared errors between all observed 2D image points and reprojection points, thereby achieving joint optimization of camera pose and 3D coordinate points.

[0142] Example 3

[0143] like Figure 1 As shown, this embodiment describes the differences from Embodiment 1. This embodiment provides a method for predicting three-dimensional point clouds of tea buds and leaves based on depth information, including:

[0144] S1: Acquire RGB image sequences and depth image sequences of tea buds and leaves;

[0145] S2: Use a tea visual model to segment the RGB image sequence to obtain a tea bud and leaf mask image sequence;

[0146] S3: Extract the total matching loss based on the RGB image sequence and the depth image sequence;

[0147] S4: Estimate the camera pose based on the total matching loss;

[0148] S5: Optimize the camera pose to obtain the optimized camera pose;

[0149] S6: Combine the RGB image sequence, depth image sequence and tea bud and leaf mask image sequence to determine the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system, and obtain the camera point cloud of tea buds and leaves.

[0150] S7: Project the point cloud of the tea bud and leaf camera onto the three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target.

[0151] The process of combining RGB image sequences, depth image sequences, and tea bud and leaf mask image sequences to determine the three-dimensional coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system, thereby obtaining the camera point cloud of tea buds and leaves, includes:

[0152] S61: Match the depth image sequence with the tea bud and leaf mask image sequence, wherein the depth image sequence includes multiple depth images and the tea bud and leaf mask image sequence includes multiple tea bud and leaf mask images;

[0153] S62: Obtain the depth value of each pixel in the corresponding tea bud and leaf mask image from the depth image;

[0154] S63: Based on the depth value, calculate the three-dimensional spatial coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system using the following formula:

[0155] ;

[0156] in, Let be the z-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let x be the x-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let y be the y-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system;

[0157] S64: Convert the coordinates of all pixels in the camera coordinate system into a tea bud and leaf camera point cloud; the tea bud and leaf camera point cloud is used to represent the complete shape of the tea bud and leaf target.

[0158] The RGB image is segmented using a tea leaf visual model to obtain a mask image of tea buds and leaves. , where P i P represents the pixels of tea buds and leaves in the masked image of tea buds and leaves. s This represents all pixels in the RGB image. Extract the target pixel-level contour from the tea bud and leaf mask image, preserving the point cloud data within the target area of ​​the tea bud and leaf while removing background and other interfering information. For each tea bud and leaf target pixel coordinate P in the tea bud and leaf mask image...i For each (u, v), the depth value z of the pixel location can be obtained from the corresponding depth image. Therefore, the three-dimensional spatial coordinates of that point in the camera coordinate system are... The formula is as follows:

[0159] ;

[0160] ;

[0161] Where K is the camera intrinsic parameter matrix, f x f is the focal length in the x-direction. y c is the focal length in the y-direction. x Let c be the intersection of the optical axis in the x-direction and the imaging plane. y The point where the optical axis in the y-direction intersects the imaging plane. Let be the coordinates of the i-th pixel in the camera coordinate system along the z-direction. Let be the coordinates of the i-th pixel in the camera coordinate system along the x-direction. Let be the coordinate of the i-th pixel in the camera coordinate system along the y-direction. Let z be the depth value of the i-th pixel in the tea bud and leaf mask image, depthscale be the depth size factor, u be the x-coordinate of the tea bud and leaf target in the tea bud and leaf mask image in the image coordinate system, v be the y-coordinate of the tea bud and leaf target in the tea bud and leaf mask image in the image coordinate system, and P be the depth value of the tea bud and leaf mask image. i,z Let be the coordinates of the i-th pixel in the z-direction in the world coordinate system. The x-axis of the image coordinate system is parallel to the x-axis of the camera coordinate system, and the y-axis of the image coordinate system is parallel to the y-axis of the camera coordinate system.

[0162] The step of projecting the tea bud and leaf camera point cloud onto a three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target includes:

[0163] S71: The following formula is used to transform the tea bud and leaf camera point cloud using rotation and translation matrices, so as to project the tea bud and leaf camera point cloud onto the world coordinate system:

[0164] ;

[0165] Among them, R i Let t be the i-th rotation matrix. i Let i be the translation matrix. Let i be the i-th 3D point in the camera coordinate system. Let i be the i-th three-dimensional point in the world coordinate system;

[0166] S72: Combine all three-dimensional spatial points in the world coordinate system to form a three-dimensional point cloud of tea buds and leaves.

[0167] The target local point cloud of tea buds and leaves in each keyframe of the RGB image sequence is projected onto a 3D world coordinate system and then stitched and fused. For the i-th spatial point in the camera coordinate system... The spatial point projected onto the world coordinate system is . The calculation formula is as follows:

[0168] ;

[0169] in, Let R be the three-dimensional coordinates of the i-th point in the world coordinate system. i Let t be the i-th rotation matrix. i Let i be the translation matrix. Let R be the 3D coordinates of the i-th point in the camera coordinate system, calculated using visual odometry. i and t i .

[0170] Visual odometry is a core module in SLAM (Simultaneous Localization and Mapping), used to estimate camera motion based on consecutive image frames. The rotation matrix describes the camera's rotation about a certain axis, while the translation matrix describes the camera's displacement in space.

[0171] After obtaining the three-dimensional point cloud of the tea bud and leaf target, the method further includes:

[0172] S73: Bilateral filtering is used to remove noise from the 3D point cloud of tea buds and leaves to obtain the denoised 3D point cloud;

[0173] S74: Statistical filtering is used to remove outliers from the denoised 3D point cloud to obtain the optimized 3D point cloud.

[0174] Bilateral filtering is a nonlinear filtering method that combines spatial proximity and pixel value similarity, which can effectively remove noise in point clouds while preserving edge features. Statistical filtering is an effective method for removing outliers based on statistical analysis. It calculates the average distance between each point and its neighboring points, and then calculates the standard deviation of the global average distance based on the average distance, using the standard deviation of the global average distance as the judgment criterion. The process of removing outliers in the denoised 3D point cloud using statistical filtering is as follows: (1) For the denoised 3D point cloud set {T1,T2,...,T... total The average distance between the target point cloud and points in a specific neighborhood:

[0175] ;

[0176] Where, d i T represents the denoised 3D point cloud.i (x i ,y i ,z i The average distance from x to num points in the neighborhood. i Let x represent the coordinates of the center of the i-th point cloud in the x-direction, and y represent the coordinates of the center of the i-th point cloud in the x-direction. i Let z represent the coordinates of the center of the i-th point cloud in the y-direction, and z represent the coordinates of the center of the i-th point cloud in the z-direction. i Let x represent the coordinates of the center of the i-th point cloud in the z-direction. j Let y represent the coordinates of the j-th point in the x-direction within the neighborhood of the i-th point cloud. j Let z represent the coordinates of the j-th point in the y-direction within the neighborhood of the i-th point cloud. j Let represent the coordinates of the j-th point in the z-direction within the neighborhood of the i-th point cloud.

[0177] (2) Calculate the global mean and standard deviation of the average distance using the following formulas:

[0178] ;

[0179] ;

[0180] in, This represents the global average distance between the total number of centers of the denoised 3D point cloud. The standard deviation of the global average distance is represented by , and total represents the total number of denoised 3D point clouds in the denoised 3D point cloud set.

[0181] (3) If the denoised 3D point cloud T of the i-th generation i d i In the interval [ -λ , +λ If the value is within the specified range, the point cloud is retained; otherwise, it is discarded.

[0182] This embodiment

[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0184] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for predicting three-dimensional point clouds of tea buds and leaves based on depth information, characterized in that, include: Collect RGB image sequences and depth image sequences of tea buds and leaves; A tea leaf visual model was used to segment RGB image sequences to obtain a sequence of tea bud and leaf mask images. Extract the total matching loss from the RGB image sequence and the depth image sequence; The camera pose is estimated based on the total matching loss. The camera pose is optimized to obtain the optimized camera pose. By combining RGB image sequences, depth image sequences, and tea bud and leaf mask image sequences, the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system are determined, and the camera point cloud of tea buds and leaves is obtained. The point cloud of the tea bud and leaf camera is projected onto the three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target. The process of combining RGB image sequences, depth image sequences, and tea bud and leaf mask image sequences to determine the three-dimensional coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system, thereby obtaining the camera point cloud of tea buds and leaves, includes: The depth image sequence is matched with the tea bud and leaf mask image sequence, wherein the depth image sequence includes multiple depth images and the tea bud and leaf mask image sequence includes multiple tea bud and leaf mask images; Obtain the depth value of each pixel in the corresponding tea bud and leaf mask image from the depth image; Based on the depth value, the three-dimensional spatial coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system are calculated using the following formula: ； in, Let be the z-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let x be the x-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. f is the y-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system; x f is the focal length in the x-direction. y Let c be the focal length in the y-direction. x Let c be the intersection of the optical axis in the x-direction and the imaging plane. y y is the intersection of the optical axis in the y-direction and the imaging plane; z is the depth value of the i-th pixel in the tea bud and leaf mask image, depthscale is the depth size factor, u is the abscissa of the tea bud and leaf target in the tea bud and leaf mask image in the image coordinate system, v is the ordinate of the tea bud and leaf target in the tea bud and leaf mask image in the image coordinate system, P i,z Let be the coordinates of the i-th pixel in the z-direction in the world coordinate system; The coordinates of all pixels in the camera coordinate system are converted into a point cloud of tea buds and leaves; the point cloud of tea buds and leaves is used to represent the complete shape of the tea bud and leaf target.

2. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, The step of extracting the total matching loss based on the RGB image sequence and the depth image sequence includes: An image pyramid is constructed based on RGB image sequences, and the image pyramid is used to represent the scale space of the image; ORB features of the RGB image are extracted from the image pyramid, and the difference of the normalized ORB vector is calculated based on the ORB features. Extracting gradient feature differences from depth image sequences; The ORB vector differences and the gradient feature differences of the depth image are weighted and fused to obtain the total matching loss.

3. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The extraction of gradient feature differences from depth image sequences includes: Construct horizontal and vertical convolutional kernels; The horizontal gradient of the depth image is calculated using the horizontal convolution kernel, and the vertical gradient of the depth image is calculated using the vertical convolution kernel; wherein, the depth image sequence includes multiple depth images; The horizontal gradient is normalized to obtain a normalized horizontal gradient; the vertical gradient is normalized to obtain a normalized vertical gradient. The arctangent function of the ratio of the normalized vertical gradient to the normalized horizontal gradient is used to obtain the depth gradient direction angle. The gradient feature differences of the depth image are calculated based on the depth gradient direction angle.

4. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The step of calculating the difference in the normalized ORB vector based on the ORB features includes: The difference in normalized ORB vectors between different frames of RGB images is calculated using the following formula: ； d H (g k1 ,g k2 )= ; Among them, g k1 Let g represent the k1-th ORB vector. k2 This represents the k2th ORB vector. This represents the length of the k1-th ORB vector. d represents the difference between the k1-th and k2-th ORB vectors. H (g k1 ,g k2 The expression () represents the difference between the normalized ORB vectors of the k1-th and k2-th ORB vectors; i represents the element index in the ORB vector, and m represents the total number of elements in the ORB vector. This represents the i-th element in the k1-th ORB vector. This represents the i-th element in the k2-th ORB vector.

5. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The step of estimating the camera pose based on the total matching loss includes: The minimum total matching loss is obtained by searching for the minimum value among all total matching losses. Select the first target ORB vector and the second target ORB vector corresponding to the minimum total matching loss; Count the matching point pairs between the first target ORB vector and the second target ORB vector; Based on the matching point pairs, the rotation and translation matrices of the depth camera are derived in reverse using the reprojection error; wherein, the rotation matrix and the translation matrix constitute the camera pose.

6. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, The optimization of the camera pose to obtain the optimized camera pose includes: Calculate the sum of all reprojection errors to obtain the total reprojection error; Minimize the total weight projection error, and use the camera pose corresponding to the minimum value of the total weight projection error as the optimized camera pose.

7. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 5, characterized in that, The process of deriving the rotation and translation matrices of the depth camera using reprojection errors includes: Based on the following formula, the relationship between camera pose and reprojection error is constructed using the bundle adjustment algorithm: ； Among them, e i Let u represent the i-th reprojection error. i s represents the projected coordinates of the i-th pixel in the world coordinate system of the tea bud and leaf mask image. i Let P represent the scale factor, K represent the camera intrinsic parameter matrix, T represent the camera pose matrix, and P represent the camera pose matrix. i The i-th pixel in the tea bud and leaf mask image is represented by the camera pose matrix, which includes a rotation matrix and a translation matrix. Based on the i-th reprojection error, the i-th rotation matrix and the i-th translation matrix of the depth camera are derived in reverse.

8. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 3, characterized in that, The step of calculating the gradient feature difference of the depth image based on the depth gradient direction angle includes: The gradient feature differences of depth images are calculated using the following formula: ； ； in, This represents the difference in depth gradient direction angle between the i-th depth image and the j-th depth image, with min indicating the minimum value operation. This represents the depth gradient direction angle of the i-th depth image. This represents the depth gradient orientation angle of the j-th depth image; Let represent the gradient feature difference between the i-th depth image and the j-th depth image, and let cos represent the cosine function.

9. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, After obtaining the three-dimensional point cloud of the tea bud and leaf target, the method further includes: Bilateral filtering was used to remove noise from the 3D point cloud of tea buds and leaves, resulting in a denoised 3D point cloud. Statistical filtering is used to remove outliers from the denoised 3D point cloud to obtain an optimized 3D point cloud.

Images