Tea bud and leaf target three-dimensional point cloud prediction method based on depth information

Abstract

The invention provides a tea bud and leaf target three-dimensional point cloud prediction method based on depth information, and the method comprises the steps: collecting an RGB image sequence and a depth image sequence of tea buds and leaves, and carrying out the instance segmentation of the RGB image sequence through a tea visual model. And extracting total matching loss according to the RGB image sequence and the depth image sequence, estimating a camera pose according to the total matching loss, and optimizing the camera pose. And judging the three-dimensional coordinate of each pixel point in the tea leaf mask image in a camera coordinate system to obtain a tea bud leaf camera point cloud. And projecting the tea bud and leaf camera point cloud to a three-dimensional world coordinate system to obtain a tea bud and leaf target three-dimensional point cloud. According to the method, the distance between a theoretical projection point and an actual projection point is minimized through iterative optimization, the optimized camera pose is obtained, on this basis, depth information is combined, accurate three-dimensional information of a target in a visual field is obtained, point cloud data under multiple visual angles are spliced, and therefore the tea bud leaf three-dimensional point cloud with a more complete shape is reconstructed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a tea bud leaf target three-dimensional point cloud prediction method based on depth information. BACKGROUND

[0002] In an unstructured natural tea garden environment, a famous tea picking robot needs to plan a picking strategy according to the perception information of a vision system. The target position obtained from a two-dimensional image cannot guide the picking robot to work in a three-dimensional space. Therefore, the picking robot needs to have more accurate local three-dimensional perception of the picking target to obtain the local three-dimensional point cloud of the picking target. The local three-dimensional point cloud of the picking target can help the picking robot accurately identify the three-dimensional spatial position angle of the picking target and the relative position relationship between the picking target and the picking mechanism, thereby providing decision support for the picking robot to successfully complete the picking work.

[0003] Traditional target local three-dimensional perception methods usually adopt a feature fusion-based method, the core of which is to obtain local point cloud by using features such as color, texture, and shape of the target. However, due to the complex lighting conditions of the tea garden and the growth characteristics of the tea leaves, the imaging scales of targets at different distances in the same field of view are different, and the poses are different, which causes the tea buds to present various morphological characteristics under the camera view. It is difficult for the vision system to obtain accurate three-dimensional information of the target in the field of view by relying only on the feature fusion-based method. At the same time, the images collected by the depth camera are often disturbed by noise, resulting in the loss of some depth information in the depth image. Therefore, it is a significant challenge to rely only on the image information obtained from a single view to construct a point cloud of the tea bud leaf with a complete morphology.

[0004] In summary, it is necessary to splice the point cloud data under multiple views to reconstruct a more complete three-dimensional point cloud of the tea bud leaf. SUMMARY

[0005] To overcome the problems in the related art, the purpose of the present application is to provide a tea bud leaf target three-dimensional point cloud prediction method based on depth information, which splices the point cloud data under multiple views to reconstruct a more complete three-dimensional point cloud of the tea bud leaf.

[0006] A tea bud leaf target three-dimensional point cloud prediction method based on depth information, comprising: collecting an RGB image sequence and a depth image sequence of tea buds; performing instance segmentation on the RGB image sequence by using a tea leaf vision model to obtain a tea bud leaf mask image sequence; extracting a total matching loss according to the RGB image sequence and the depth image sequence; estimating a camera pose according to the total matching loss; Optimize the camera pose to obtain an optimized camera pose; In combination with the RGB image sequence, the depth image sequence and the tea bud leaf mask image sequence, the three-dimensional coordinates of each pixel point in the tea leaf mask image under the camera coordinate system are determined to obtain a tea bud leaf camera point cloud; The tea bud leaf camera point cloud is projected into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud.

[0007] In the preferred technical solution of the present application, the total matching loss is extracted according to the RGB image sequence and the depth image sequence, comprising: An image pyramid is constructed based on the RGB image sequence, and the image pyramid is used to represent the scale space of the image; The ORB features of the RGB image are extracted in the image pyramid, and the normalized ORB vector difference is calculated according to the ORB features; The gradient feature difference of the depth image is extracted from the depth image sequence; The ORB vector difference and the gradient feature difference of the depth image are weighted and fused to obtain the total matching loss.

[0008] In the preferred technical solution of the present application, the gradient feature difference of the depth image is extracted from the depth image sequence, comprising: Horizontal and vertical convolution kernels are constructed; The horizontal gradient of the depth image is calculated using the horizontal convolution kernel, and the vertical gradient of the depth image is calculated using the vertical convolution kernel; wherein the depth image sequence comprises a plurality of depth images; The horizontal gradient is normalized to obtain a normalized horizontal gradient; and the vertical gradient is normalized to obtain a normalized vertical gradient; The inverse tangent function of the ratio of the normalized vertical gradient to the normalized horizontal gradient is calculated to obtain a depth gradient direction angle; The gradient feature difference of the depth image is calculated according to the depth gradient direction angle.

[0009] In the preferred technical solution of the present application, the normalized ORB vector difference is calculated according to the ORB features, comprising: The normalized ORB vector difference between different frames of RGB images is calculated according to the following formula: ; d H (g k1 ,g k2 )= ; Wherein g k1 represents the k1th ORB vector, and g k2denotes the k2th ORB vector, denotes the length of the k1th ORB vector, denotes the ORB vector difference between the k1th ORB vector and the k2th ORB vector, d H (g k1 , g k2 denotes the normalized ORB vector difference between the k1th ORB vector and the k2th ORB vector; i denotes the element sequence number in the ORB vector, and m denotes the total number of elements in the ORB vector, denotes the i-th element in the k1th ORB vector, denotes the i-th element in the k2th ORB vector.

[0010] In the preferred technical scheme of the present application, the camera pose is estimated according to the total matching loss, comprising: searching for the minimum value in all total matching losses to obtain the minimum total matching loss; screening out the first target ORB vector and the second target ORB vector corresponding to the minimum total matching loss; counting the matching point pairs of the first target ORB vector and the second target ORB vector; based on the matching point pairs, the rotation matrix and the translation matrix of the depth camera are inversely deduced using the re-projection error; wherein the rotation matrix and the translation matrix constitute the camera pose.

[0011] In the preferred technical scheme of the present application, the camera pose is optimized to obtain an optimized camera pose, comprising: constructing a relationship between the camera pose and the re-projection error; calculating the sum of all re-projection errors to obtain a total re-projection error; minimizing the total re-projection error, and taking the camera pose corresponding to the minimum value of the total re-projection error as the optimized camera pose.

[0012] In the preferred technical scheme of the present application, the rotation matrix and the translation matrix of the depth camera are inversely deduced using the re-projection error, comprising: according to the following formula, the relationship between the camera pose and the re-projection error is constructed by using the bundle adjustment algorithm: ; wherein e i denotes the i-th re-projection error, u i denotes the projection coordinates of the i-th pixel point of the tea bud leaf mask image in the world coordinate system, s i denotes the scale factor, K denotes the camera intrinsic matrix, T denotes the camera pose matrix, P ithe i-th pixel point of the tea bud leaf mask image, the camera pose matrix comprises a rotation matrix and a translation matrix; the i-th rotation matrix and the i-th translation matrix of the depth camera are deduced reversely according to the i-th re-projection error.

[0013] In the preferred technical scheme of the present application, the combination of the RGB image sequence, the depth image sequence and the tea bud leaf mask image sequence is used to determine the three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system, and a tea bud leaf camera point cloud is obtained, comprising: The depth image sequence and the tea bud leaf mask image sequence are matched, the depth image sequence comprises a plurality of depth images, and the tea bud leaf mask image sequence comprises a plurality of tea bud leaf mask images. The depth value of each pixel point in the corresponding tea bud leaf mask image is obtained from the depth image. According to the depth value, the three-dimensional space coordinates of each pixel point in the tea bud leaf mask image in the camera coordinate system are calculated by using the following formula: ; wherein, z is the z-axis coordinate of the i-th pixel point in the tea bud leaf mask image in the camera coordinate system, x is the x-axis coordinate of the i-th pixel point in the tea bud leaf mask image in the camera coordinate system, y is the y-axis coordinate of the i-th pixel point in the tea bud leaf mask image in the camera coordinate system. The coordinates of all pixel points in the camera coordinate system are converted into a tea bud leaf camera point cloud; the tea bud leaf camera point cloud is used to represent the complete morphology of the tea bud leaf target.

[0014] In the preferred technical scheme of the present application, the calculation of the gradient feature difference of the depth image according to the depth gradient direction angle comprises: The gradient feature difference of the depth image is calculated by using the following formula: ; ; wherein, represents the depth gradient direction angle difference between the i-th depth image and the j-th depth image, and min represents the minimum value operation, represents the depth gradient direction angle of the i-th depth image, represents the depth gradient direction angle of the j-th depth image. represents the gradient feature difference between the i-th depth image and the j-th depth image, and cos represents the cosine function.

[0015] In the preferred technical scheme of the present application, after obtaining the tea bud leaf target three-dimensional point cloud, the following steps are further included: Adopt bilateral filtering to remove the noise in the tea bud leaf target three-dimensional point cloud, and obtain the three-dimensional point cloud after denoising; Adopt statistical filtering to remove outliers in the three-dimensional point cloud after denoising, and obtain the optimized three-dimensional point cloud.

[0016] The beneficial effects of the present application are: The tea bud leaf target three-dimensional point cloud prediction method based on depth information provided by the present application comprises collecting an RGB image sequence and a depth image sequence of tea bud leaves, the depth image sequence being an image set composed of multiple consecutive depth images, each depth image being used to represent the distance from different regions in the scene to the depth camera. A tea leaf visual model is used to perform instance segmentation on the RGB image sequence to obtain a tea bud leaf mask image sequence. Instance segmentation aims to realize target detection and pixel classification, that is, to identify the object category in each RGB image and generate an independent binary mask therefor. The total matching loss is extracted according to the RGB image sequence and the depth image sequence, and the camera pose is estimated according to the total matching loss. ORB feature is a high-efficiency feature detection and description method, which has rotation invariance and scale invariance. The above advantages of ORB feature are used to match and associate frames, so as to accurately estimate the camera pose. The three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system are judged in combination with the RGB image sequence, the depth image sequence and the tea bud leaf mask image sequence, and the tea bud leaf camera point cloud is obtained. The tea bud leaf camera point cloud is projected into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud. The present application minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain the optimized camera pose, and on this basis, the accurate three-dimensional information of the target in the field of view is obtained in combination with the depth information, the point cloud data under multiple perspectives is spliced, and thus a more complete tea bud leaf three-dimensional point cloud is reconstructed. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a flowchart of the tea bud leaf target three-dimensional point cloud prediction method based on depth information of the present application; Figure 2 is a flowchart of the total matching loss of the tea bud leaf mask image sequence extraction of the present application; Figure 3 is a flowchart of the camera pose estimation according to the total matching loss of the present application; Figure 4 is a result diagram of the inter-frame matching according to the total matching loss of the present application. DETAILED DESCRIPTION

[0018] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a method for predicting the three-dimensional point cloud of tea buds and leaves based on depth information, including: S1: Acquire RGB image sequences and depth image sequences of tea buds and leaves; S2: Use a tea visual model to segment the RGB image sequence to obtain a tea bud and leaf mask image sequence; S3: Extract the total matching loss based on the RGB image sequence and the depth image sequence; S4: Estimate the camera pose based on the total matching loss; S5: Optimize the camera pose to obtain the optimized camera pose; S6: Combine the RGB image sequence, depth image sequence and tea bud and leaf mask image sequence to determine the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system, and obtain the camera point cloud of tea buds and leaves. S7: Project the point cloud of the tea bud and leaf camera onto the three-dimensional world coordinate system to obtain the three-dimensional point cloud of the tea bud and leaf target.

[0020] like Figure 2 As shown, the step of extracting the total matching loss based on the RGB image sequence and the depth image sequence includes: S31: Construct an image pyramid based on an RGB image sequence, wherein the image pyramid is used to represent the scale space of the image; S32: Extract the ORB features of the RGB image from the image pyramid, and calculate the difference of the normalized ORB vector based on the ORB features; S33: Extract gradient feature differences from depth image sequences; S34: Weighted fusion of ORB vector differences and gradient feature differences of depth images to obtain the total matching loss.

[0021] The image pyramid is composed of multi-resolution sub-images of the same original image, with high-resolution features at the bottom and low-resolution features at the top. In this embodiment, each tea bud and leaf mask image in the tea bud and leaf mask image sequence is downsampled multiple times. Each downsampling reduces the length and width of the image to half of their original values, i.e., the total number of pixels is reduced to 1 / 4 of the original value, thereby extracting features at different resolutions.

[0022] ORB is a high-efficiency feature point detection and description algorithm, which combines the advantages of FAST corner detection and BRIEF descriptor, and is optimized for rotation invariance and computational efficiency. ORB algorithm calculates the feature point direction by gray centroid method, and takes the feature point and centroid offset vector as the direction, so that the algorithm has rotation invariance. By constructing a Gaussian pyramid, multi-scale detection is realized, so that the algorithm has scale invariance. ORB algorithm uses the direction information of FAST corner detection to rotate the point pair sampling mode, thereby enhancing the rotation invariance.

[0023] The RGB image sequence includes multiple RGB images with tea bud leaves as the target, the depth image sequence includes multiple depth images reflecting the depth information of the tea bud leaves, and the tea bud leaf mask image sequence includes multiple tea bud leaf mask images. The ORB features of each pixel point in the RGB image are calculated to obtain multiple ORB features, the number of ORB features is the same as the total number of pixel points in the RGB image, and each ORB feature corresponds to a pixel point in the RGB image. All ORB features of an RGB image form an ORB vector, for example, the k1th ORB vector is represented as , and the k2th ORB vector is represented as . The total matching loss is calculated according to the following formula: ; where D total represents the total matching loss, represents the first fusion coefficient, represents the second fusion coefficient, + =1, represents the ORB vector difference between the i th RGB image and the j th RGB image, represents the gradient feature difference between the i th depth image and the j th depth image. The ORB vector difference reflects the two-dimensional tea bud leaf pose difference between different frames of RGB images, and the gradient feature difference of the depth image reflects the pose difference of the tea bud leaf in the depth direction between different frames of depth images, thereby accurately representing the pose of the tea bud leaf in three-dimensional space.

[0024] The normalized ORB vector difference is calculated according to the ORB features, which includes: The normalized ORB vector difference between different frames of RGB images is calculated according to the following formula: ; d H (g k1 ,g k2 )= ; where g k1denotes the k1th ORB vector, g k2 denotes the k2th ORB vector, denotes the length of the k1th ORB vector, denotes the ORB vector difference between the k1th ORB vector and the k2th ORB vector, d H (g k1 ,g k2 ) denotes the normalized ORB vector difference between the k1th ORB vector and the k2th ORB vector; i denotes the element sequence number in the ORB vector, and m denotes the total number of elements in the ORB vector, denotes the i-th element in the k1th ORB vector, denotes the i-th element in the k2th ORB vector; on the basis of grouping multiple ORB features into ORB vectors, the distance between the ORB vectors is measured by using the norm of the difference between the two ORB vectors, and then divided by the length of the corresponding ORB vector to obtain the normalized ORB vector difference, which has the advantage of high operation efficiency.

[0025] The gradient feature difference of the depth image is extracted from the sequence of depth images, comprising: S331: constructing a horizontal convolution kernel and a vertical convolution kernel; S332: using the horizontal convolution kernel to calculate the horizontal gradient of the depth image, and using the vertical convolution kernel to calculate the vertical gradient of the depth image; wherein the sequence of depth images comprises a plurality of depth images; S333: normalizing the horizontal gradient to obtain a normalized horizontal gradient; and normalizing the vertical gradient to obtain a normalized vertical gradient; S334: calculating the arctangent function of the ratio of the normalized vertical gradient to the normalized horizontal gradient to obtain a depth gradient direction angle; S335: calculating the gradient feature difference of the depth image according to the depth gradient direction angle.

[0026] The horizontal gradient and the vertical gradient of the depth image are calculated by using the following formula: ; ; ; ; wherein S x is a horizontal convolution kernel, S y is a vertical convolution kernel, Z is a depth image, Z x is the horizontal gradient of the depth image, and Z y is the vertical gradient of the depth image. The horizontal convolution kernel S xA vertical convolution kernel S for detecting gradient change of a depth image in an x direction y A vertical convolution kernel S for detecting gradient change of a depth image in an x direction

[0027] The horizontal gradient and the vertical gradient are normalized respectively by using the following formulas: ; ; ; wherein, denotes the normalized horizontal gradient, denotes the normalized vertical gradient. u denotes the horizontal coordinate of the depth image, i.e. the row number, and v denotes the vertical coordinate of the depth image, i.e. the column number, denotes the pixel neighborhood with the feature point p as the center and the radius r. denotes the value of the point in the u-th row and the v-th column of the horizontal gradient, denotes the value of the point in the u-th row and the v-th column of the vertical gradient. denotes the Gaussian weight, exp denotes the exponential function, u0 is the horizontal coordinate of the feature point p, and v0 is the vertical coordinate of the feature point p. denotes the Gaussian weight adjustment parameter, .

[0028] The depth gradient direction angle is calculated by using the following formula: ; wherein, denotes the depth gradient direction angle of the feature point p, the value of is [0, π), denotes the inverse tangent function.

[0029] The gradient feature difference of the depth image is calculated according to the depth gradient direction angle, which comprises: The gradient feature difference of the depth image is calculated by using the following formula: ; ; wherein, denotes the depth gradient direction angle difference between the i-th depth image and the j-th depth image, min denotes the minimum value operation, denotes the depth gradient direction angle of the i-th depth image, denotes the depth gradient direction angle of the j-th depth image; denotes the gradient feature difference between the i-th depth image and the j-th depth image, and cos denotes the cosine function.

[0030] The depth gradient direction angle has both a direction and can be extended. In the embodiment, two absolute value operations and a minimum value operation are combined to limit the depth gradient direction angle of two depth images in the range of 0-π / 2, so as to measure the real gradient feature difference of the two depth images. The cosine function of the depth gradient direction angle difference is calculated, the gradient feature difference of the depth image is normalized in the range of 0-1, the unified measurement standard is unified, and the pose difference of the tea bud leaf target in the depth in the two depth images is represented.

[0031] The tea bud leaf target three-dimensional point cloud prediction method based on depth information provided in the embodiment comprises collecting an RGB image sequence and a depth image sequence of the tea bud leaf. The depth image sequence is an image set composed of a plurality of continuous depth images, and each depth image is used to represent the distance from different regions in the scene to the depth camera. The tea leaf visual model is used to perform instance segmentation on the RGB image sequence to obtain a tea bud leaf mask image sequence. The instance segmentation aims to realize target detection and pixel classification, that is, to identify the object category in each RGB image and generate an independent binary mask for the object category. The total matching loss is extracted according to the RGB image sequence and the depth image sequence, that is, the two frames of RGB images with the minimum total matching loss are paired, and at the same time, the two frames of depth images with the corresponding serial numbers are paired. For example, if the total matching loss of the third frame image and the fifth frame image is the smallest, the third frame RGB image and the fifth frame RGB image are paired, and at the same time, the third frame depth image and the fifth frame depth image are paired. The camera pose is estimated according to the total matching loss. Figure 4 The result image of the inter-frame matching according to the total matching loss of the present application is shown in Figure 4 , wherein Figure 4 the points marked in the left image correspond one by one to the points marked in the right image. Figure 4

[0032] The ORB feature is a high-efficiency feature detection and description method, which has rotation invariance and scale invariance. The above advantages of the ORB feature are used to match and associate the frames, so as to accurately estimate the camera pose. The three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system are judged in combination with the RGB image sequence, the depth image sequence and the tea leaf mask image sequence, and the tea bud leaf camera point cloud is obtained. The tea bud leaf camera point cloud is projected into the three-dimensional world coordinate system, and the tea bud leaf target three-dimensional point cloud is obtained. The present application minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain the optimized camera pose. On this basis, the accurate three-dimensional information of the target in the field of view is obtained in combination with the depth information, the point cloud data under multiple perspectives is spliced, and the more complete tea bud leaf three-dimensional point cloud is reconstructed.

[0033] ​The tea bud leaf target three-dimensional point cloud prediction method based on depth information provided in the embodiment comprises collecting an RGB image sequence and a depth image sequence of tea bud leaves, the depth image sequence being an image set composed of multiple frames of continuous depth images, each depth image being used to represent the distance from different regions in the scene to the depth camera. A tea leaf visual model is used to perform instance segmentation on the RGB image sequence to obtain a tea bud leaf mask image sequence. Instance segmentation aims to realize target detection and pixel classification, that is, to identify the object category in each RGB image and generate an independent binary mask therefor. A total matching loss is extracted according to the RGB image sequence and the depth image sequence, a camera pose is estimated according to the total matching loss, and the camera pose is optimized to obtain an optimized camera pose. ORB features are a kind of efficient feature detection and description method, which have rotation invariance and scale invariance. The above advantages of ORB features are used to match and associate frames to accurately estimate the camera pose. In combination with the RGB image sequence, the depth image sequence and the tea bud leaf mask image sequence, the three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system are judged to obtain a tea bud leaf camera point cloud. The tea bud leaf camera point cloud is projected into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud. The present application minimizes the distance between the theoretical projection point and the actual projection point through iterative optimization to obtain an optimized camera pose, and on this basis, the accurate three-dimensional information of the target in the field of view is obtained in combination with the depth information, the point cloud data under multiple perspectives is spliced, and thus a more complete tea bud leaf three-dimensional point cloud is reconstructed.

[0034] Embodiment 2 As Figure 1 shown, the embodiment is based on embodiment 1, and describes the difference from embodiment 1. The embodiment provides a tea bud leaf target three-dimensional point cloud prediction method based on depth information, comprising: S1: collecting an RGB image sequence and a depth image sequence of tea bud leaves; S2: using a tea leaf visual model to perform instance segmentation on the RGB image sequence to obtain a tea bud leaf mask image sequence; S3: extracting a total matching loss according to the RGB image sequence and the depth image sequence; S4: estimating a camera pose according to the total matching loss; S5: optimizing the camera pose to obtain an optimized camera pose; S6: in combination with the RGB image sequence, the depth image sequence and the tea bud leaf mask image sequence, judging the three-dimensional coordinates of each pixel point in the tea leaf mask image in the camera coordinate system to obtain a tea bud leaf camera point cloud; S7: projecting the tea bud leaf camera point cloud into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud.

[0035] As Figure 3estimating the camera pose according to the total matching loss, comprising: S41: searching for a minimum value in all total matching losses, to obtain a minimum total matching loss; S42: screening out a first target ORB vector and a second target ORB vector corresponding to the minimum total matching loss; S43: counting matching point pairs of the first target ORB vector and the second target ORB vector; S44: based on the matching point pairs, using a re-projection error to inversely deduce a rotation matrix and a translation matrix of the depth camera; wherein the rotation matrix and the translation matrix constitute the camera pose.

[0036] The smaller the total matching loss is, the more similar the two target ORB vectors are, that is, the stronger the correlation of the two frames of RGB images of the tea bud leaves in time and space is. The pixel point coordinates and depth information of the matching point pairs are combined to obtain three-dimensional point coordinates. The re-projection error is used to measure the deviation between the theoretical position and the actual observed position of the three-dimensional space point projected onto the two-dimensional image, and the camera pose is optimized by minimizing the re-projection error.

[0037] optimizing the camera pose to obtain an optimized camera pose, comprising: S51: calculating a sum of all re-projection errors to obtain a total re-projection error; S52: minimizing the total re-projection error, and taking the camera pose corresponding to the minimum value of the total re-projection error as the optimized camera pose.

[0038] The relationship between the camera pose and the re-projection error is as follows: ; wherein e i represents the i-th re-projection error, u i represents the projection coordinates of the i-th pixel point of the tea bud leaf mask image in the world coordinate system, s i represents a scale factor, K represents a camera intrinsic matrix, T represents a camera pose matrix, P i represents the i-th pixel point of the tea bud leaf mask image, the camera pose matrix comprises a rotation matrix and a translation matrix, R represents the rotation matrix, t represents the translation matrix, and the i-th rotation matrix and the i-th translation matrix of the depth camera are inversely deduced according to the i-th re-projection error.

[0039] All re-projection errors are added, that is, the polynomials are summed, to obtain a total re-projection error, and the camera pose corresponding to the minimum value of the total re-projection error is taken as the optimized camera pose, which is applicable to each frame of RGB image in the sequence of RGB images.

[0040] The embodiment adopts a bundle adjustment algorithm to construct a relationship between the camera pose and the re-projection error. The bundle adjustment adjusts the camera pose and the three-dimensional point coordinates, and the camera pose includes the rotation matrix and the translation matrix of the camera, so that the sum of squares of errors between all observed two-dimensional image points and re-projection points is minimized, thereby realizing joint optimization of the camera pose and the three-dimensional coordinate points.

[0041] Embodiment 3 As Figure 1 shown, the embodiment describes the differences from embodiment 1 on the basis of embodiment 1. The embodiment provides a tea bud leaf target three-dimensional point cloud prediction method based on depth information, including: S1: collecting an RGB image sequence and a depth image sequence of tea bud leaves; S2: performing instance segmentation on the RGB image sequence by using a tea leaf visual model to obtain a tea bud leaf mask image sequence; S3: extracting a total matching loss according to the RGB image sequence and the depth image sequence; S4: estimating a camera pose according to the total matching loss; S5: optimizing the camera pose to obtain an optimized camera pose; S6: combining the RGB image sequence, the depth image sequence, and the tea bud leaf mask image sequence to determine the three-dimensional coordinates of each pixel point in the tea leaf mask image under the camera coordinate system, and obtaining a tea bud leaf camera point cloud; S7: projecting the tea bud leaf camera point cloud into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud.

[0042] The combination of the RGB image sequence, the depth image sequence, and the tea bud leaf mask image sequence to determine the three-dimensional coordinates of each pixel point in the tea leaf mask image under the camera coordinate system to obtain a tea bud leaf camera point cloud includes: S61: matching the depth image sequence and the tea bud leaf mask image sequence, the depth image sequence including multiple depth images, and the tea bud leaf mask image sequence including multiple tea bud leaf mask images; S62: obtaining the depth value of each pixel point in the corresponding tea bud leaf mask image from the depth image; S63: according to the depth value, using the following formula to calculate the three-dimensional space coordinates of each pixel point in the tea bud leaf mask image under the camera coordinate system: ; wherein, is the z-axis coordinate of the i-th pixel point in the tea bud leaf mask image under the camera coordinate system, is the x-axis coordinate of the i-th pixel point in the tea bud leaf mask image under the camera coordinate system, y-coordinate of the i-th pixel point in the tea bud leaf mask image in the camera coordinate system; S64: convert the coordinates of all pixel points in the camera coordinate system into a tea bud leaf camera point cloud; the tea bud leaf camera point cloud is used to represent the complete morphology of the tea bud leaf target.

[0043] The tea bud leaf mask image is obtained by segmenting the RGB image using the tea leaf visual model, that is, where P i is a pixel point of the tea bud leaf in the tea bud leaf mask image, P s is the total pixel point of the RGB image. The target pixel-level contour in the tea bud leaf mask image is extracted, the point cloud data in the tea bud leaf target area is retained, and the background and other interference information is removed. For each tea bud leaf target pixel coordinate P i (u,v) in the tea bud leaf mask image, the depth value z of the pixel position can be obtained from the corresponding depth image, and the three-dimensional space coordinates of the point in the camera coordinate system The formula is as follows: ; ; where K is the camera intrinsic matrix, f x is the focal length in the x direction, f y is the focal length in the y direction, c x is the intersection of the x direction optical axis and the imaging plane, and c y is the intersection of the y direction optical axis and the imaging plane. is the z direction coordinate of the i-th pixel point in the camera coordinate system, is the x direction coordinate of the i-th pixel point in the camera coordinate system, is the y direction coordinate of the i-th pixel point in the camera coordinate system. z is the depth value of the i-th pixel point in the tea bud leaf mask image, depthscale is the depth size factor, u is the horizontal coordinate of the tea bud leaf target in the image coordinate system in the tea bud leaf mask image, v is the vertical coordinate of the tea bud leaf target in the image coordinate system in the tea bud leaf mask image, P i,z is the z direction coordinate of the i-th pixel point in the world coordinate system. The x axis of the image coordinate system is parallel to the x axis of the camera coordinate system, and the y axis of the image coordinate system is parallel to the y axis of the camera coordinate system.

[0044] The tea bud leaf camera point cloud is projected into a three-dimensional world coordinate system to obtain a tea bud leaf target three-dimensional point cloud, comprising: S71: using the following formula, using a rotation matrix and a translation matrix to transform the tea bud leaf camera point cloud to project the tea bud leaf camera point cloud into the world coordinate system: ; wherein R i is the i-th rotation matrix, t i is the i-th translation matrix, is the i-th three-dimensional space point in the camera coordinate system, and is the i-th three-dimensional space point in the world coordinate system. S72: Grouping all three-dimensional space points in the world coordinate system into a tea bud leaf target three-dimensional point cloud.

[0045] Projecting the target local point cloud of the tea bud leaf of each key frame in the RGB image sequence into the three-dimensional world coordinate system for splicing and fusion, for the i-th space point in the camera coordinate system , the space point in the world coordinate system is . The calculation formula of is as follows: ; wherein, is the three-dimensional coordinate of the i-th point in the world coordinate system, R i is the i-th rotation matrix, t i is the i-th translation matrix, is the three-dimensional coordinate of the i-th point in the camera coordinate system, R i and t i are calculated according to the visual odometry.

[0046] The visual odometry is a core module in SLAM (Simultaneous Localization and Mapping), which is used to estimate the motion of the camera according to the continuous image frames. The rotation matrix is used to describe the rotation of the camera around an axis, and the translation matrix is used to describe the displacement of the camera in space.

[0047] After obtaining the tea bud leaf target three-dimensional point cloud, the method further comprises: S73: Removing noise in the tea bud leaf target three-dimensional point cloud by using bilateral filtering to obtain a denoised three-dimensional point cloud; S74: Removing outliers in the denoised three-dimensional point cloud by using statistical filtering to obtain an optimized three-dimensional point cloud.

[0048] Bilateral filtering is a nonlinear filtering method that combines spatial proximity and pixel value similarity, which can effectively remove noise in point cloud while preserving edge features. Statistical filtering is an effective method based on statistical analysis to remove outliers, which calculates the average distance of each point to the points in the neighborhood, and then calculates the standard deviation of the global average distance according to the average distance, and takes the standard deviation of the global average distance as the judgment standard. The process of removing outliers in the denoised three-dimensional point cloud using statistical filtering is as follows: (1) For the target point cloud in the denoised three-dimensional point cloud set {T1, T2,..., T total} and the average distance of the points in a certain neighborhood: ; Where d i represents the average distance of the i-th denoised three-dimensional point cloud T i (x i ,y i ,z i ) to the num points in the neighborhood, x i represents the coordinate of the center of the i-th point cloud in the x direction, y i represents the coordinate of the center of the i-th point cloud in the y direction, and z i represents the coordinate of the center of the i-th point cloud in the z direction. x j represents the x coordinate of the j-th point in the neighborhood of the i-th point cloud, y j represents the y coordinate of the j-th point in the neighborhood of the i-th point cloud, and z j represents the z coordinate of the j-th point in the neighborhood of the i-th point cloud.

[0049] (2) Calculate the global mean and standard deviation of the average distance according to the following formula: ; ; Where represents the global average distance of the total centers of the denoised three-dimensional point cloud set, and represents the standard deviation of the global average distance, and total represents the total number of denoised three-dimensional point clouds in the denoised three-dimensional point cloud set.

[0050] (3) If the d i of the i-th denoised three-dimensional point cloud T i is in the interval [ -λ , +λ ], then the point cloud is retained, otherwise the point cloud is removed.

[0051] The embodiment It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a", "comprising", or "comprises" does not, without further restriction, exclude the existence of additional elements of the process, method, article, or apparatus that comprises the element.

[0052] The preferred embodiments of the present application have been described above with the specific details of the preferred embodiments to provide complete understanding of the application. However, it will be apparent to persons having ordinary skill in the technology that, without departing from the spirit and scope of the application, they can make equivalent changes or modifications to the preferred embodiments. Therefore, the patent protection scope of the present application should be subject to the protection scope of the patent claims.

Claims

1. A method for predicting three-dimensional point clouds of tea buds and leaves based on depth information, characterized in that, include: Collect RGB image sequences and depth image sequences of tea buds and leaves; A tea leaf visual model was used to segment RGB image sequences to obtain a sequence of tea bud and leaf mask images. Extract the total matching loss from the RGB image sequence and the depth image sequence; The camera pose is estimated based on the total matching loss. The camera pose is optimized to obtain the optimized camera pose. By combining RGB image sequences, depth image sequences, and tea bud and leaf mask image sequences, the three-dimensional coordinates of each pixel in the tea leaf mask image in the camera coordinate system are determined, and the camera point cloud of tea buds and leaves is obtained. The point cloud of the tea bud and leaf camera is projected onto a three-dimensional world coordinate system to obtain a three-dimensional point cloud of the tea bud and leaf target.

2. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, The step of extracting the total matching loss based on the RGB image sequence and the depth image sequence includes: An image pyramid is constructed based on RGB image sequences, and the image pyramid is used to represent the scale space of the image; ORB features of the RGB image are extracted from the image pyramid, and the difference of the normalized ORB vector is calculated based on the ORB features. Extracting gradient feature differences from depth image sequences; The ORB vector differences and the gradient feature differences of the depth image are weighted and fused to obtain the total matching loss.

3. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The extraction of gradient feature differences from depth image sequences includes: Construct horizontal and vertical convolutional kernels; The horizontal gradient of the depth image is calculated using the horizontal convolution kernel, and the vertical gradient of the depth image is calculated using the vertical convolution kernel; wherein, the depth image sequence includes multiple depth images; The horizontal gradient is normalized to obtain a normalized horizontal gradient; the vertical gradient is normalized to obtain a normalized vertical gradient. The arctangent function of the ratio of the normalized vertical gradient to the normalized horizontal gradient is used to obtain the depth gradient direction angle. The gradient feature differences of the depth image are calculated based on the depth gradient direction angle.

4. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The step of calculating the difference in the normalized ORB vector based on the ORB features includes: The difference in normalized ORB vectors between different frames of RGB images is calculated using the following formula: ； d H (g k1 ,g k2 )= ; Among them, g k1 Let g represent the k1-th ORB vector. k2 This represents the k2th ORB vector. This represents the length of the k1-th ORB vector. d represents the difference between the k1-th and k2-th ORB vectors. H (g k1 ,g k2 The expression () represents the difference between the normalized ORB vectors of the k1-th and k2-th ORB vectors; i represents the element index in the ORB vector, and m represents the total number of elements in the ORB vector. This represents the i-th element in the k1-th ORB vector. This represents the i-th element in the k2-th ORB vector.

5. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 2, characterized in that, The step of estimating the camera pose based on the total matching loss includes: The minimum total matching loss is obtained by searching for the minimum value among all total matching losses. Select the first target ORB vector and the second target ORB vector corresponding to the minimum total matching loss; Count the matching point pairs between the first target ORB vector and the second target ORB vector; Based on the matching point pairs, the rotation and translation matrices of the depth camera are derived in reverse using the reprojection error; wherein, the rotation matrix and the translation matrix constitute the camera pose.

6. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, The optimization of the camera pose to obtain the optimized camera pose includes: Calculate the sum of all reprojection errors to obtain the total reprojection error; Minimize the total weight projection error, and use the camera pose corresponding to the minimum value of the total weight projection error as the optimized camera pose.

7. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 5, characterized in that, The process of deriving the rotation and translation matrices of the depth camera using reprojection errors includes: Based on the following formula, the relationship between camera pose and reprojection error is constructed using the bundle adjustment algorithm: ； Among them, e i Let u represent the i-th reprojection error. i s represents the projected coordinates of the i-th pixel in the world coordinate system of the tea bud and leaf mask image. i Let P represent the scale factor, K represent the camera intrinsic parameter matrix, T represent the camera pose matrix, and P represent the camera pose matrix. i The i-th pixel in the tea bud and leaf mask image is represented by the camera pose matrix, which includes a rotation matrix and a translation matrix. Based on the i-th reprojection error, the i-th rotation matrix and the i-th translation matrix of the depth camera are derived in reverse.

8. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, The process of combining RGB image sequences, depth image sequences, and tea bud and leaf mask image sequences to determine the three-dimensional coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system, thereby obtaining the camera point cloud of tea buds and leaves, includes: The depth image sequence is matched with the tea bud and leaf mask image sequence, wherein the depth image sequence includes multiple depth images and the tea bud and leaf mask image sequence includes multiple tea bud and leaf mask images; Obtain the depth value of each pixel in the corresponding tea bud and leaf mask image from the depth image; Based on the depth value, the three-dimensional spatial coordinates of each pixel in the tea bud and leaf mask image in the camera coordinate system are calculated using the following formula: ； in, Let be the z-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let x be the x-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system. Let y be the y-coordinate of the i-th pixel in the tea bud and leaf mask image in the camera coordinate system; The coordinates of all pixels in the camera coordinate system are converted into a point cloud of tea buds and leaves; the point cloud of tea buds and leaves is used to represent the complete shape of the tea bud and leaf target.

9. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 3, characterized in that, The step of calculating the gradient feature difference of the depth image based on the depth gradient direction angle includes: The gradient feature differences of depth images are calculated using the following formula: ； ； in, This represents the difference in depth gradient direction angle between the i-th depth image and the j-th depth image, with min indicating the minimum value operation. This represents the depth gradient direction angle of the i-th depth image. This represents the depth gradient orientation angle of the j-th depth image; Let represent the gradient feature difference between the i-th depth image and the j-th depth image, and let cos represent the cosine function.

10. The method for predicting three-dimensional point clouds of tea buds and leaves based on depth information according to claim 1, characterized in that, After obtaining the three-dimensional point cloud of the tea bud and leaf target, the method further includes: Bilateral filtering was used to remove noise from the 3D point cloud of tea buds and leaves, resulting in a denoised 3D point cloud. Statistical filtering is used to remove outliers from the denoised 3D point cloud to obtain an optimized 3D point cloud.

Images