Tea picking point positioning method based on virtual and real depth fusion

Abstract

The invention relates to a tea leaf picking point positioning method based on virtual and real depth fusion, and belongs to the technical field of intelligent tea leaf picking. According to the method, through combination of an RGB-D camera, deep learning and biomorphological modeling, accurate positioning of a picking point and picking suitability in a complex tea garden environment is realized. According to the method, through multi-sensor information fusion, the growth state, morphological characteristics and environmental data of the tea leaves are obtained, machine vision and a deep learning model are combined, picking key points are accurately positioned, the picking opportunity is judged through a growth model, and therefore a more efficient and more intelligent tea leaf picking solution is provided. Through deep learning and biomorphic modeling technologies, growth characteristics and real-time image data of tea trees are comprehensively utilized, the accuracy and efficiency of tea leaf picking are greatly improved, and intelligent upgrading of the tea industry is further promoted.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent tea picking and relates to a tea picking point positioning method based on virtual-real deep fusion. BACKGROUND

[0002] With the continuous progress of intelligent agricultural technology, the automation and intelligentization of the tea production process have become an important direction to promote the sustainable development of the tea industry. In recent years, the growing environment of tea trees has become increasingly complex, and tea picking, as the most labor-intensive link in the tea industry, still mainly relies on manual operation. However, traditional manual picking is not only low in efficiency but also labor-intensive, especially in the production process of famous tea, picking operation accounts for a large part of tea garden management labor. With the continuous advancement of agricultural modernization, the demand for mechanized and intelligent picking in the tea industry has also increased. In recent years, the rapid development of computer vision, deep learning and robotics technology has been applied in tea picking. Through image processing and machine learning technology, the positioning of tea picking points has achieved certain results, but due to the complexity of the tea garden environment, the similarity between tea and weeds, branches and leaves is high, and a single vision system cannot effectively cope with problems such as occlusion and environmental interference, resulting in that the picking accuracy and efficiency still cannot meet the actual demand. SUMMARY

[0003] Therefore, the purpose of the present application is to provide a tea picking point positioning method based on virtual-real deep fusion, which combines RGB-D cameras, deep learning and biological morphology modeling to accurately position the picking point and picking suitability in a complex tea garden environment.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0005] A tea picking point positioning method based on virtual-real deep fusion, the method comprising:

[0006] Collecting physiological and ecological multi-source information of tea tree growth, extracting tea tree bud leaf time series morphological features and quantifying; learning biological-morphological coupling representation through multi-modal autoencoder to generate time series coupling feature vector; based on the feature vector, using Transformer encoder to predict future growth trend of bud leaf, to distinguish current growth stage and to evaluate picking suitability, combining the predicted information to establish a virtual three-dimensional model of tea leaf meta-universe, thereby predicting an ideal key point set conforming to the current stage;

[0007] Synchronously capture color images and depth images of tea leaves and fuse to obtain color-depth information; model tea tree tender shoots as a structure prior graph based on biological structure prior and directed structure graph; based on color-depth information and structure prior graph, use a graph convolution network to extract structure key points from the structure prior graph and form a structured key point set; according to the topological connection of the structure prior graph, perform structure verification on each point in the structured key point set, and sort the structure key points that pass the structure verification to generate a candidate point set;

[0008] Perform visibility analysis on each candidate point in the candidate point set, if the point is visible, it is an actual candidate key point; if the point is not visible, according to the pre-defined matching relationship between the ideal key point set and the candidate point set, the corresponding point in the ideal key point set is mapped to the current three-dimensional scene and is taken as a virtual candidate key point, wherein the pre-defined matching relationship is obtained through a key point matching function; assign a comprehensive confidence to each actual candidate key point and virtual candidate key point through a confidence scoring function, and sort according to the comprehensive confidence to obtain a picking point set;

[0009] Set a temporal stability factor to evaluate the stability of each picking point in multiple consecutive frames, which is calculated based on the variance of the position fluctuation of the picking point in multiple consecutive frames; combine the temporal stability, comprehensive confidence and physiological suitability of each picking point to calculate a multi-factor comprehensive score, select the picking point with the highest multi-factor comprehensive score as the optimal picking point, and return the three-dimensional coordinates of the optimal picking point.

[0010] Further, through the environment sensor, physiological and ecological multi-source information including temperature T, humidity H, light intensity L, CO2 concentration C, soil nutrient N, soil water content W, leaf water content LW and chlorophyll content LC is collected, to obtain the environmental and physiological feature vector E(t i ) of each moment t i ) = [T(t i ), H(t i ), L(t i ), C(t i ), N(t i ), W(t i ), LW(t i ), LC(t i )], construct a physiological data set ε(t) = {E(t1),..., E(t i ),..., E(t n )}, n is the time length; continuously capture the time sequence images I of tea tree buds and leaves using an RGB-D camera, extract morphological features bud length L b , growth posture angle θ and curvature radius κ, and define the time sequence morphological feature vector M(t i ) = [L b (ti ),κ(t i ),θ(t i )], and obtain the morphological time series dataset

[0011] Through two sets of parallel autoencoders, E(t) and After feature compression, a bidirectional gated cross-fusion module is introduced to achieve bidirectional attention perception between modalities. The cross-fused representation is encoded again to obtain a high-order coupled embedding:

[0012]

[0013] Where σ is the Sigmoid gate activation function; ⊙ represents the Hadamard product; and are the feature vectors after the first encoding; and are the feature vectors after cross fusion, W E With W M is the transformation matrix of cross fusion; and are the encoding functions of the feature vectors after cross fusion; and Respectively for and High-order coupling embedding vector after secondary encoding;

[0014] The residual fusion mechanism is used to unify the high-order representations of the two modalities into a coupled embedding vector Z(t):

[0015]

[0016] Where LN represents layer normalization; then the embedding vectors of all time points are combined into a temporal input sequence Z(t) = {Z(t1), Z(t2), ..., Z(t n )} is input to the time series modeling module, and the fusion time coding outputs the time series coupling feature vector F(t)={F(t1),F(t2),...,F(t n )}.

[0017] Furthermore, the obtained temporal coupling feature vector F(t) is input into the Transformer encoder Learning global dynamic representation: F(t i ) Input stage discriminant head gets Through the linear layer F(t i ) to perform regression and predict the picking suitability score Introducing the spatial regression head to F(t i) performing linear mapping to regress out keypoint positions

[0018] based on keypoint regression prediction results obtain the three-dimensional key structure of the bud leaf, each key point contains (X, Y, Z) three-dimensional coordinates; use cubic spline interpolation to spatially fit the key point sequence to generate a continuous and smooth bud leaf main axis curve B(s); by normal extension on both sides of the main axis curve, using the physiological modeling parameters of leaf width and thickness, the leaf surface contour point set is constructed; according to the stage classification result indicated growth stage, select different levels of detail to reconstruct the morphology of the bud leaf; according to the picking suitability score control the freshness parameter of the rendering model; dynamically adjust the rendering attribute vector R(t i ), the attribute value is mapped to the physical rendering material parameter through linear interpolation; wherein the bud leaf skeleton curve function B(s) and the attribute vector R(t i ) are respectively represented as:

[0019]

[0020] In the formula, s is the normalized arc length parameter; M(t i ) is the moisture content simulation coefficient, C(t i ) is the color saturation adjustment coefficient, and G(t i ) is the surface smoothness coefficient.

[0021] Finally, the completed three-dimensional point cloud surface is generated into a mesh model through the PSR algorithm, and a complete three-dimensional virtual bud leaf is formed through the Unity meta-universe development platform, and an ideal key point set P conforming to the current stage is obtained.

[0022] Further, color images and depth images of tea leaves are synchronously collected, each pixel is mapped to three-dimensional space based on joint registration of depth images and color images, and a dense point cloud D of the tender shoot structure is reconstructed; biological structure prior is introduced, and the tea tree tender shoot is modeled as a directed graph structure composed of bud, stem and leaf, the geometric connection relationship of the tender shoot structure is represented by a structure prior graph G=(V, E), V represents the center key point node of each part, and E represents the bud-stem and stem-leaf connection relationship; the direction between nodes is determined by the growth axis;

[0023] Based on the constructed dense point cloud D and the structure prior graph G, a graph representation of key structure points is constructed based on the structure prior graph G, the local topological dependence between candidate parts is extracted by using the GCN network, and the perception ability of the features to the structure connectivity is enhanced; based on the GCN output, channel attention and spatial attention are introduced to weight the significance of different channels and regions, and the response ability of the network to the key structure region is improved; each node in the structure graph G is regarded as a learnable structure query vector, and a structure perception Transformer decoder with graph position encoding is used to capture the structure-related context information in the global feature graph; each structure query outputs the spatial position (x i ,y i ,z i ) and confidence score c i of a key point; a multi-source supervised target is introduced for supervised learning to enhance the geometric constraint ability and structure consistency of the network, and finally a structured key point set m is the number of key points.

[0024] Further, according to the topological connection of the structure prior graph, the points in the structured key point set are structure-verified, and for each key point p i , the connectivity with other nodes in the structure prior graph is calculated, and for the nodes with valid connection paths, the geometric consistency index on each connection path is calculated, if the connection is invalid or the spatial structure is abnormal, the point p i is removed; the angle θ i between the vector of the key point p i and the principal axis is calculated, if θ i meets the requirements, it is considered that the orientation of the key point p i meets the natural growth direction, otherwise, the point p i is removed or processed with a reduced weight;

[0025] The structure key points verified by the structure are taken as candidate points, and a candidate point set K is generated according to the confidence degree of each candidate point.

[0026] Further, a predefined matching relationship between the ideal key point set and the candidate point set is obtained through a key point matching function, wherein the key point matching function is:

[0027]

[0028] In the formula, p i is the three-dimensional coordinate information of the ideal key point, k i is the three-dimensional coordinate information of the candidate point, c i is the confidence score; Θ(k i ) is an occlusion penalty term, if k iis 1 if the key point is occluded, otherwise 0; alpha, beta and delta are weight coefficients.

[0029] Further, a comprehensive confidence is assigned to each actual candidate key point and virtual candidate key point by a confidence score function, as shown in the following formula:

[0030]

[0031] In the formula, S f (k i ) is the comprehensive confidence, v(k i ) is the visibility score, represents the spatial rationality score, c i is the confidence score, and lambda1, lambda2 and lambda3 are weight coefficients; according to the ranking of the comprehensive confidence S f (k i ), a certain number of points with the highest comprehensive confidence are reserved to obtain a picking point set.

[0032] Further, a multi-factor comprehensive score is calculated in combination with the time sequence stability, comprehensive confidence and physiological suitability of each picking point:

[0033]

[0034] In the formula, X is the coordinate information of the corresponding ideal picking target point; Omega i is the time sequence stability factor, rho(k i ) is the physiological suitability score; sigma(·) is a Sigmoid normalization; gamma1, gamma2 and gamma3 are weight parameters; the picking point with the highest multi-factor comprehensive score is selected as the optimal picking point, and the three-dimensional coordinates of the optimal picking point are returned.

[0035] The present application has the following advantages:

[0036] (1) Breaking through the environmental limitations, improving the positioning accuracy and robustness: the present application fuses RGB-D depth information, time sequence morphological features and biological growth models to construct a multi-modal perception system. Through the tender shoot joint posture analysis network, the tea leaf structure is modeled in three dimensions, effectively overcoming the defects of traditional single visual system which is easily affected by branch and leaf occlusion, weed interference and light changes, significantly enhancing the accuracy and environmental adaptability of key point detection in complex scenes.

[0037] (2) Realize the growth-driven intelligent decision: the application adopts the growth prediction model based on "tea leaf meta universe", uses multi-source sensor data and transformer time series modeling, quantifies the physiological and ecological characteristics of bud leaves and predicts the growth stage. Combined with the biological-morphological coupling characteristics, the picking suitability and ideal key point position are dynamically evaluated, so that the positioning process is in line with the growth law of tea trees, and the passive recognition is upgraded to active decision dominated by the growth state, which greatly optimizes the scientificity of picking time judgment.

[0038] (3) Fault-tolerant mechanism of virtual-real cooperation: the application proposes a visibility-guided fusion strategy. For each candidate point, the real-time RGB-D image is used to determine its visibility in the current scene. When the point is invisible, the corresponding ideal point position coordinates are mapped to a virtual point to ensure the continuous traceability of the picking target. Through this mechanism, the operation continuity of the picking robot under extreme conditions such as local occlusion or limited view angle is significantly improved, and the risk of missing detection is reduced.

[0039] (4) Multi-dimensional optimization of picking efficiency and quality: the application fuses spatial coordinate confidence, growth suitability and picking feasibility and other multi-dimensional indicators through time series stability factor and comprehensive score function to select the optimal picking point. This strategy takes into account the short-term positioning stability and long-term physiological rationality, while ensuring picking accuracy and minimizing invalid operations, providing high-reliability execution instructions for the picking robot and comprehensively improving work efficiency and tea product quality.

[0040] Other advantages, objects, and features of the present application will be in part apparent and in part pointed out below in the specification, in which reference will be made to the appended drawings. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the specification and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred detailed description of the present application will be given below in combination with the drawings, in which:

[0042] Figure 1 The flowchart of the tea picking point positioning method based on virtual-real depth fusion provided by an embodiment of the present application. DETAILED DESCRIPTION

[0043] Following, the advantages and effects of the present application can be easily understood by those skilled in the art from the description disclosed herein. The present application can also be implemented or applied by different specific embodiments, and the details in the description can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] In view of the problems of complex tea garden environment, high similarity between tea leaves and weeds, branches and leaves, and single visual system cannot effectively cope with occlusion and environmental interference in the prior art, the present application proposes a tea leaf picking point positioning method, which includes two core parts of tea leaf growth model construction and virtual-real key point positioning. The positioning process mainly includes three main stages of real-time data perception and key point preliminary acquisition, virtual-real key point depth fusion based on visibility, and optimal picking point determination and output combined with timing stability and multi-factor evaluation. In the data acquisition and processing stage, the physiological and morphological data of tea trees are collected, the biological-morphological coupling feature extraction and timing modeling technology is used to construct the tea leaf growth prediction model, and the precise prediction of key points, stages and picking suitability is realized. In the virtual-real key point depth fusion stage based on visibility, the virtual key point is formed by executing the virtual-real key point fusion strategy based on visibility analysis. In the optimal picking point determination and output stage combined with timing stability and multi-factor evaluation, the key point with the highest joint screening score is selected as the final picking target by combining timing stability, fusion confidence, physiological and ecological picking suitability, and estimated picking feasibility, so as to realize efficient and intelligent automatic picking of tea leaves.

[0045] As Figure 1 shown, it is an embodiment of the method proposed by the present application, which is described as follows:

[0046] I. A growth prediction model based on tea leaf meta-universe is proposed. First, multi-source information collection of tea tree growth and extraction and quantification of time-series morphological characteristics of tea tree buds and leaves are performed. By deploying multiple sensors, time-series ecological data composed of physiological indicators of tea trees are obtained; at the same time, a machine vision system is used to capture time-series images of tea tree buds and leaves, and image processing techniques are used to quantify key morphological indicators such as bud length, growth posture, and construct time-series morphological data. Second, a biological-morphological coupling feature extraction method based on multi-source time-series data is proposed, which learns the physiological-morphological coupling representation by constructing a multi-modal autoencoder to generate time-series coupling feature vectors. Further, based on the vector sequence, a tea leaf life cycle growth model is constructed using Transformer to train the model to predict future growth trends, determine the current growth stage, and evaluate the picking suitability, and use this information to construct a virtual three-dimensional model of tea leaf meta-universe, and directly predict the ideal key point position list P that meets the current stage.

[0047] The steps are as follows:

[0048] 1: Micro-environmental factor data collection: define the sampling time sequence τ = {t1, t2,..., t n}, deploy environmental sensors at each time point t i , i ∈ [1, n] to obtain temperature T, humidity H, light intensity L, CO2 concentration C, soil nutrients N, soil water content W, leaf water content LW, and chlorophyll content LC. Thus, the environmental and physiological feature vector E(t i ) = [T(t i ), H(t i ), L(t i ), C(t i ), N(t i ), W(t i ), LW(t i ), LC(t i )] can be obtained. Accordingly, a physiological data set ε(t) = {E(t1), E(t2),..., E(t n )} can be constructed. Wherein,

[0049] 2: Time-series image collection and morphological feature extraction of tea tree buds and leaves: use RGB-D camera to continuously capture time-series images I of tea tree buds and leaves, and extract morphological features (including bud length L b , growth posture angle θ, curvature radius κ). Bud length L bis the Euclidean distance from the tip to the base of the shoot. The shoot leaf growth posture angle θ reflects the degree of spatial bending and the direction of the main axis, which is calculated by the angle between the main axis and the horizontal plane. The curvature radius κ is calculated by fitting the shoot leaf skeleton curve. The image acquisition time is set to every morning, noon, and evening to enhance the generalization of the data. The morphological time series feature vector M(t i ) is defined as b (t i ), κ(t i ), θ(t i )] to obtain the morphological time series dataset wherein the calculation of κ is as follows:

[0050]

[0051] wherein the greater κ indicates the more bending of the shoot leaf; Q(s) is expressed in the form of a two-dimensional parameter of the skeleton curve with respect to the arc length; s0 and s1 represent the arc lengths at the start and end of the skeleton, respectively; represents the modulus of the local curvature vector, and s is the arc length parameter used to describe the skeleton curve, and each point on the skeleton can be uniquely determined by the length s along the curve from the starting point.

[0052] The morphological time series dataset is coupled with the physiological dataset ε(t) = {E(t1), E(t2),..., E(t n ) as the input for subsequent biological-morphological coupling modeling.

[0053] 3: Based on the obtained dataset E(t) and , a hierarchical nested cross-fusion autoencoder is proposed to learn coupled features of multi-source data and extract biological-morphological fusion time series features.

[0054] Specifically, for each time point t i , there is a corresponding physiological feature vector and a time series morphological feature vector Two sets of parallel autoencoders are used to compress E(t) and respectively, and then a bidirectional gated cross-fusion module is introduced to realize bidirectional attention perception between modalities, thereby enhancing the perception ability of morphology to physiology and physiology to morphology. The cross-fused representation is encoded again to obtain high-order coupled embedding. The process is as follows:

[0055]

[0056] wherein σ is the Sigmoid gate activation; and ⊙ represents the Hadamard product (element multiplication); and are the feature vectors after the first encoding, respectively; and are the cross-fused feature vectors, respectively, W E and M is the cross-fused transformation matrix; and are the encoding functions for the cross-fused feature vectors, respectively; and are the first-order coupling embedding vectors, respectively; and are the second-order coupling embedding vectors.

[0057] 4: Use the residual fusion mechanism to unify the two modal high-order representations into a coupling embedding vector Z(t), and then group the embedding vectors of all time points into a time sequence input sequence Z(t) = {Z(t1), Z(t2),..., Z(t n ) and input it to the time modeling module, to obtain the time encoding output time sequence coupling feature vector F(t) = {F(t1), F(t2),..., F(t n )}. Wherein, Z(t) is represented as follows:

[0058]

[0059] In the formula, LN represents layer normalization, d Z is the embedding feature dimension after fusion.

[0060] 5: Input the obtained time sequence coupling feature vector F(t) into the Transformer encoder to learn the global dynamic representation, and input F(t i ) into the stage discrimination head to obtain Further, F(t i ) is regressed through a linear layer to predict the picking suitability score Finally, a spatial regression head is introduced to perform linear mapping on F(t i ) to regress the key point position. The specific expression is:

[0061]

[0062] Wherein, F(t i ) represents the feature vector fused with the information of physiology, environment and morphology corresponding to time t i ; represents the stage classification result of time t i ; SM is a Softmax function for normalizing the output; W s , b s are the weights and bias of stage discrimination, respectively; and σ is a sigmoid function. The score of whether the current time point is suitable for picking, 0 means extremely unsuitable, 1 means most suitable; W r , b r are the regression weight and bias of the regression key point respectively. p , b p are the regression weight and bias of the regression key point respectively.

[0063] 6: Prediction result based on key point regression Obtain the three-dimensional key structure of the bud leaf, where n p represents the number of predicted key points, each key point contains (X, Y, Z) three-dimensional coordinates. The key point sequence is spatially fitted by using cubic spline interpolation to generate a continuous and smooth bud leaf main axis curve B(s). By normal extension on both sides of the main axis curve, the physiological modeling parameters of leaf width and thickness are used to construct the leaf surface contour point set.

[0064] According to the stage classification result indicated growth stage, different levels of detail are selected for morphological reconstruction: (1) bud stage: sparse key point fitting is used, surface details are simplified, and the tender bud texture is enhanced; (2) growth period: high-density key point fitting is used, and local curvature modulation is introduced to simulate natural bending; (3) mature period: micro-features such as fine lines, edge serrations, etc. are added to the main axis to reconstruct the complete vein structure.

[0065] According to the picking suitability score control the freshness parameters of the model rendering, including leaf water content, color saturation, texture glossiness, etc. The rendering attribute vector R(t i ) is dynamically adjusted, and the attribute value is mapped to the physical rendering material parameters through linear interpolation.

[0066] Finally, the completed three-dimensional point cloud surface is generated into a mesh model through the PSR algorithm, and through the Unity meta-universe development platform, a complete three-dimensional virtual bud leaf is formed and the ideal key point set P of the current stage is obtained.

[0067] The bud leaf skeleton curve function B(s) and the attribute vector R(t i ) are defined as follows:

[0068]

[0069] where s is the normalized arc length parameter; M(t i ) is the water content simulation coefficient, C(t i ) is the color saturation adjustment coefficient, and G(t i ) is the surface smoothness coefficient.

[0070] II. A real-time keypoint prediction method based on the analysis of tender shoot joint posture is proposed. First, the RGB-D camera is used to collect the image of the tea leaves to be picked in real time, and the synchronous RGB and depth information is obtained. Second, the keypoint prediction network based on the analysis of tender shoot joint posture is used to process the RGB-D data. This network regards the tender shoot of tea as a joint structure composed of buds, stems, leaves, etc. It deduces the three-dimensional spatial posture of the tender shoot of tea from the RGB-D information by referring to human pose estimation. Finally, the three-dimensional coordinate positions of the key points and their corresponding posture analysis are derived from the analyzed three-dimensional posture to form the candidate point set K observed by vision.

[0071] The steps include:

[0072] 1: The RGB-D camera is used to synchronously collect the color image I R and the depth image I D ; based on the joint registration of the depth image I D and the color image I R , each pixel is mapped to a three-dimensional space, and a dense point cloud D of the tender shoot structure is reconstructed.

[0073] 2: The biological structure prior is introduced, and the tender shoot of tea is modeled as a directed graph structure composed of buds, stems, and leaves. The overall modeling idea refers to the skeleton representation idea in human pose estimation, and the tender shoot structure is regarded as a kind of "plant skeleton". The geometric connection relationship is represented by the structure prior graph G=(V, E), where V represents the center key point nodes of each part, E represents the connection relationship between buds and stems, and stems and leaves, and is used to constrain the spatial topology. The direction between nodes is determined by the growth principal axis to realize the semantic modeling of spatial structure.

[0074] 3: Based on the constructed point cloud D and the structure prior graph G, the graph representation of key structure points is constructed based on the structure prior graph G=(V, E), the local topological dependence between candidate parts is extracted by using the graph convolution network (GCN), and the perception ability of features to structure connectivity is enhanced. On the basis of the GCN output, channel attention and spatial attention are introduced to weight the significance of different channels and regions, and the response ability of the network to key structure regions is improved. In order to realize efficient regression of structure point positions, each node in the structure graph is regarded as a learnable structure query vector, and a structure perception Transformer decoder with graph position coding is used to capture structure-related context information in the global feature map; each structure query outputs the spatial position (x i , y i , z i ) and confidence score c i of a key point. In order to enhance the geometric constraint ability and structure consistency of the network, multi-source supervision targets are introduced for supervised learning, and finally the structured keypoint set K

[0075] where the multi-source supervision target is as follows:

[0076]

[0077] is the position regression loss, representing the Euclidean distance between the key point position and the labeled point. is the confidence score loss, using Binary Cross Entropy to c i Classify whether it is a target picking point. is the graph structure consistency loss, used to constrain the distance change between adjacent key points and the angle change of the main axis, to prevent pose distortion. λ1, λ2, λ3 are loss weight coefficients, and m is the number of key points. is the labeled point.

[0078] 4: According to the topological connection in the structure graph G=(V, E), the point set V is pre-grouped in the following way:

[0079] First, the points with local topological consistency are clustered into a "structure candidate set"; then the spatial center position in each structure group is selected as the representative point; finally, non-structural outliers with a main axis angle exceeding the threshold (60°) are removed.

[0080] Learn from the "upper and lower limb symmetry" and "trunk connectivity" principles in the human skeleton, and use structure consistency as the key point retention standard. For each key point p i in the structured key point set, calculate its connectivity with other nodes in the structure graph. For nodes with valid connection paths (bud-stem-leaf), calculate the geometric consistency index (point distance in the range of [2mm, 50mm]) on each connection path. If the connection is invalid (i.e., the topological structure is incomplete, and some expected connections (edges E) in the pre-defined structure graph G=(V, E) cannot be formed in the actual detected key points) or the spatial structure is abnormal (leaf tip floating), the point is removed. Verify whether the key point p satisfies the natural growth direction by estimating the main axis vector i of the tender shoot. i If θ i ∈[0°, 15°], it is considered to be reasonable in direction, otherwise it is considered to be an abnormal pose or from a blocked point, and is removed or given a lower weight.

[0081] For key points that pass the structure verification, they are used as candidate points and a candidate point set K is generated according to the confidence ranking of each candidate point, where the confidence μ i ​​​For the original pose confidence c i The fusion result of the structure score θ i The fusion result of the structure score θ i The three-dimensional spatial coordinates of the key points; The geometric attribute vector.

[0082] Thirdly, the strategy integrates the ideal key point set P and the candidate point set K, so that the serial numbers of the points in P and K are aligned. Secondly, for each candidate point k, the visibility of k in the current scene is determined using real-time RGB-D images. If k is visible, it is taken as an actual candidate key point; if k is not visible, a virtual candidate key point list is formed by projecting the ideal position into the current three-dimensional scene using the pre-defined matching relationship between the ideal key point set P and the candidate point set K. Finally, a comprehensive confidence score S f is assigned to all candidate key points, and the picking point set is formed according to the score ranking.

[0083] The steps are as follows:

[0084] 1: Using real-time RGB-D images, the visibility of each candidate point k i is analyzed, and for each candidate point k i ∈K(i), according to its three-dimensional coordinates and the occlusion relationship of the depth image, the visibility v i of k in the current scene is determined, v i =1 indicates that it is visible in the depth image, and v i =0 indicates that it is completely invisible, all visible points constitute the visible key point set K vis (i)={k i ′|v i =1}, and all invisible points constitute the invisible key point set K inv (i)={k″ i |v i =0}. Among them, for the visible key points, their observed three-dimensional coordinates and confidence are directly retained and taken as actual candidate key points.

[0085] 2: For the invisible key points k″ i ∈K inv (i), a virtual candidate key point generation method based on structure constraint projection is proposed, specifically, a key point matching function O(p,k) is defined to learn the matching relationship between the point set P={p1,p2,...,p n} predicted by the growth model when the candidate point set K is visible and the point set K={k1,k2,...,k n} detected by the pose analysis. The key point matching function O(p,k) is defined as follows:

[0086]

[0087] Among them, p i is the three-dimensional coordinate information of the ideal key point, k i is the three-dimensional coordinate information of the candidate point, c i is the confidence score of the point, Θ(k i ) is the occlusion penalty term, if k i If it is blocked, it is 1, otherwise it is 0. α, β, and δ are the weight coefficients of each item.

[0088] 3: In order to further select the most valuable key points for picking decisions, for each candidate key point k″′ i (including actual candidate key points and virtual candidate key points) Calculate the confidence score function S after comprehensively considering the three aspects fi (k″′ i ), and finally the score S fi (k″′ i ) are sorted and the points with the highest scores are retained as the picking point set.

[0089]

[0090] Among them, v(k″′ i ) is the visibility score, according to the candidate key point k″′ i Occlusion level classification, represents the spatial rationality score, c i is the confidence score, λ1, λ2, and λ3 are the weight coefficients of each item.

[0091] Fourth, an optimization method for optimal picking points based on temporal stability and multi-factor comprehensive scoring is proposed. First, the temporal stability factor Ω is proposed. i To measure the stability of each picking point under multi-frame observation; on this basis, a comprehensive scoring function is constructed to conduct a multi-dimensional evaluation of all picking points, comprehensively considering factors such as temporal stability, confidence, physiological suitability and picking feasibility, and screening the picking point with the highest score as the final picking point; finally, the precise three-dimensional coordinates used to guide robot operation are output.

[0092] The steps include:

[0093] 1: An optimal picking point optimization method based on timing stability and multi-factor comprehensive score is proposed. After the construction of the integrated picking point list, due to environmental factors such as wind, shading, and light changes, some key points may change dramatically in a short period of time. These changes will lead to unstable picking targets, and thus affect the accuracy and efficiency of the robot picking. Therefore, it is necessary to further filter out the picking points with timing stability, continuous visibility, and high confidence as the final target.

[0094] By setting the timing stability factor Ω i , which reflects the visibility, position fluctuation, and confidence fluctuation of the ith picking point in the past t frames. It measures the stability of the point in multiple frames. The better the stability, the more consistent the position and confidence at different times, and the more suitable it is as the final picking target. i is defined as follows:

[0095]

[0096] where, υ(K i (j)) is the position variance, which represents the coordinate fluctuation of picking point i in the jth frame. K i (j) is the coordinate of the ith picking point in the jth frame, represents the average value of the position of the ith picking point in the past t frames. The variance υ(K i (j)) measures the amplitude of the position change. The smaller the value, the smaller the position fluctuation and the better the stability.

[0097] 2: Calculate the multi-factor comprehensive score by combining the timing stability, comprehensive confidence, and physiological suitability of each picking point:

[0098]

[0099] where, is the coordinate information of the corresponding ideal picking target point i; ρ(k i ) represents the physiological suitability score based on the growth model; σ(·) is the Sigmoid normalization; γ1, γ2, γ3 are weighting parameters.

[0100] 3: Sort in descending order , get the picking point with the highest score as the optimal picking point. Output the final optimal picking point and return the three-dimensional coordinate as the action target of the picking robot.

[0101] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A tea picking point positioning method based on virtual-real depth fusion, characterized in that: The method includes: Multi-source physiological and ecological information on tea plant growth is collected to extract and quantify the temporal morphological characteristics of tea buds and leaves. A multimodal autoencoder learns the biomorphic coupling representation to generate a temporal coupling feature vector. Based on this feature vector, a Transformer encoder is used to predict the future growth trend of buds and leaves, identify the current growth stage, and assess picking suitability. Combined with the predicted information, a virtual three-dimensional model of the tea metaverse is established to predict the ideal set of key points that match the current stage. Color and depth images of tea leaves are simultaneously collected and fused to obtain color-depth information. Tea tree shoots are modeled as a structural prior graph based on biological structural priors and a directed structural graph. Based on the color-depth information and the structural prior graph, a graph convolutional network is used to extract structural key points from the structural prior graph and form a structured key point set. Each point in the structured key point set is structurally verified based on the topological connections of the structural prior graph, and the structural key points that pass the structural verification are sorted to generate a candidate point set. A visibility analysis is performed on each candidate point in the candidate point set. If the point is visible, it is used as the actual candidate key point. If the point is not visible, the corresponding point in the ideal key point set is mapped to the current 3D scene according to the predefined matching relationship between the ideal key point set and the candidate point set, and used as the virtual candidate key point. The predefined matching relationship is obtained through the key point matching function. A confidence scoring function is used to assign a comprehensive confidence to each actual candidate key point and virtual candidate key point, and the picking point set is obtained by sorting according to the comprehensive confidence. A temporal stability factor is set to evaluate the stability of each picking point within multiple consecutive frames. The temporal stability factor is calculated based on the variance of the position fluctuations of the picking point within multiple consecutive frames. A multi-factor comprehensive score is calculated by combining the temporal stability, comprehensive confidence, and physiological suitability of each picking point. The picking point with the highest multi-factor comprehensive score is selected as the optimal picking point, and the three-dimensional coordinates of the optimal picking point are returned.

2. The method according to claim 1, characterized in that The environmental sensors collect physiological and ecological multi-source information including temperature T, humidity H, light intensity L, CO2 concentration C, soil nutrients N, soil moisture content W, leaf water content LW and chlorophyll content LC to obtain the physiological and ecological multi-source information at each moment t i The environmental and physiological characteristic vector E(t i )=[T(t i ),H(t i ),L(t i ),C(t i ),N(t i ),W(t i ),LW(t i ),LC(t i )], construct the physiological data set ε(t)={E(t1),...,E(t i ),...,E(t n )}, n is the time length; the RGB-D camera is used to continuously capture the time sequence image I of the tea tree buds and leaves, and the morphological feature bud length L is extracted b , growth posture angle θ and curvature radius κ, define the time series morphological feature vector M(t i )=[L b (t i ),κ(t i ),θ(t i )], and obtain the morphological time series dataset Through two sets of parallel autoencoders, E(t) and After feature compression, a bidirectional gated cross-fusion module is introduced to achieve bidirectional attention perception between modalities. The cross-fused representation is encoded again to obtain a high-order coupled embedding: Where σ is the Sigmoid gate activation function; ⊙ represents the Hadamard product; and are the feature vectors after the first encoding; and are the feature vectors after cross fusion, W E With W M is the transformation matrix of cross fusion; and are the encoding functions of the feature vectors after cross fusion; and Respectively for and High-order coupling embedding vector after secondary encoding; The residual fusion mechanism is used to unify the high-order representations of the two modalities into a coupled embedding vector Z(t): Where LN represents layer normalization; then the embedding vectors of all time points are combined into a temporal input sequence Z(t) = {Z(t1), Z(t2), ..., Z(t n )} is input to the time series modeling module, and the fusion time coding outputs the time series coupling feature vector F(t) = {F(t1), F(t2), ..., F(t n )}.

3. The method according to claim 2, characterized in that The obtained temporal coupling feature vector F(t) is input into the Transformer encoder Learning global dynamic representation: F(t i ) Input stage discriminant head gets Through the linear layer F(t i ) to perform regression and predict the picking suitability score Introducing the spatial regression head to F(t i ) performs linear mapping to regress the key point positions Based on the key point regression prediction results The three-dimensional key structure of the bud and leaf is obtained, and each key point contains (X, Y, Z) three-dimensional coordinates; the key point sequence is spatially fitted using cubic spline interpolation to generate a continuous and smooth bud and leaf main axis curve B(s); by extending the main axis curve on both sides in the normal direction, using the physiological modeling parameters of leaf width and thickness, a leaf surface contour point set is constructed; according to the stage classification results According to the growth stage indicated, different levels of detail are selected to reconstruct the morphology of the buds and leaves; according to the picking suitability score Control the freshness parameters of model rendering; dynamically adjust the rendering attribute vector R(t i ), the attribute value is mapped to the physical rendering material parameter through linear interpolation; among them, the bud leaf skeleton curve function B(s) and the attribute vector R(t i ) are respectively expressed as: Where s is the normalized arc length parameter; M(t i ) is the moisture content simulation coefficient, C(t i ) is the color saturation adjustment coefficient, G(t i ) is the surface smoothness coefficient; Finally, the completed three-dimensional point cloud surface is used to generate a mesh model through the PSR algorithm, and through the Unity metaverse development platform, a complete three-dimensional virtual bud and leaf is formed to obtain the ideal key point set P that meets the current stage.

4. The method according to claim 1, wherein Color and depth images of tea leaves are collected simultaneously. Each pixel is mapped to three-dimensional space based on joint registration of the depth and color images to reconstruct a dense point cloud D of the shoot structure. A biological structural prior is introduced to model the shoot as a directed graph consisting of buds, stems, and leaves. The geometric connectivity of the shoot structure is represented by a structural prior graph G = (V, E), where V represents the central key node of each part and E represents the bud-stem and stem-leaf connections. The directions between nodes are determined by the main growth axis. Based on the constructed dense point cloud D and the structural prior graph G, a graph representation of key structural points is constructed based on the structural prior graph G. The GCN network is used to extract the local topological dependencies between candidate parts and enhance the feature perception of structural connectivity. Based on the GCN output, channel attention and spatial attention are introduced to weight the significance of different channels and regions, thereby improving the network's response to key structural regions. Each node in the structural graph G is regarded as a learnable structural query vector, and a structure-aware Transformer decoder with graph position encoding is used to capture structural contextual information in the global feature graph. Each structural query output corresponds to the spatial position (x) of a key point. i ,y i ,z i ) and confidence score c i ; Introducing multi-source supervision targets for supervised learning to enhance the geometric constraint ability and structural consistency of the network, and finally forming a structured key point set m is the number of key points.

5. The method according to claim 4, characterized in that According to the topological connection of the structure prior graph, the structure of each key point set is checked. i , calculate its connectivity with other nodes in the structural prior graph, and for nodes with valid connection paths, calculate the geometric consistency index on each connection path. If the connection fails or the spatial structure is abnormal, the point p is eliminated. i ; Calculate the key point p i Vector With spindle The angle θ i , if θ i If it meets the requirements, the key point p i The direction is consistent with the natural growth direction, otherwise the point p is removed i or downgrade the rights; The key structural points that pass the structural verification are taken as candidate points, and the candidate point set K is generated according to the confidence ranking of each candidate point.

6. The method according to claim 1, characterized in that The predefined matching relationship between the ideal key point set and the candidate point set is obtained through the key point matching function, where the key point matching function is: Where p i is the three-dimensional coordinate information of the ideal key point, k i is the three-dimensional coordinate information of the candidate point, c i is the confidence score; Θ(k i ) is the occlusion penalty term, if k i If blocked, it is 1, otherwise it is 0; α, β, δ are weight coefficients.

7. The method according to claim 1, characterized in that The confidence scoring function is used to assign comprehensive confidence to each actual candidate key point and virtual candidate key point, as shown in the following formula: Where S fi (k″′ i ) is the comprehensive confidence, v(k″′ i ) is the key point k″′ to be selected i Visibility score, represents the spatial rationality score, c i is the confidence score, λ1, λ2, λ3 are weight coefficients; according to the comprehensive confidence S fi (k″′ i ) sorting, retaining several points with the highest comprehensive confidence to obtain the picking point set.

8. The method according to claim 7, characterized in that The multi-factor comprehensive score was calculated by combining the temporal stability, comprehensive confidence and physiological suitability of each picking point: Where, is the coordinate information of the corresponding ideal picking target point; Ω i is the temporal stability factor, ρ(k i ) is the physiological suitability score; σ(·) is the Sigmoid normalization; γ1, γ2, and γ3 are weighting parameters; the picking point with the highest multi-factor comprehensive score is selected as the optimal picking point, and the three-dimensional coordinates of the optimal picking point are returned.

Images