Intelligent robot tea tender shoot picking point identification method based on multi-source data fusion

Abstract

This invention relates to a method for identifying tea shoot picking points using an intelligent robot based on multi-source data fusion, belonging to the field of intelligent tea picking technology. First, nighttime images of tea leaves are acquired, and a nighttime image enhancement model based on a decomposition model is proposed to address the problems of numerous noise points and insufficient brightness in nighttime images. Next, a DFOCC-YOLOv8 model is proposed to obtain two-dimensional detection boxes for tea leaves, compensating for channel and spatial features to solve the problem of missed detection of small tea targets. Furthermore, a TeaReLU activation function and an adaptive occlusion loss function are proposed to improve the problem of inaccurate identification of tea shoots due to occlusion. Then, a tea shoot segmentation model based on heterogeneous multi-feature weighted fusion is constructed to remove background irrelevant to the tea leaves. Finally, point cloud data from a LiDAR scanner is introduced to obtain depth information, and a picking point localization system based on depth data optimization and wind speed correction is constructed to determine the location of the tea shoot picking point.

Description

Technical Field

[0001] This invention belongs to the field of intelligent tea picking technology, and relates to an intelligent robot method for identifying tea shoot picking points based on multi-source data fusion. Background Technology

[0002] Currently, the production of premium teas relies heavily on manual harvesting, with a low rate of mechanization. The mechanization rate for tea garden management nationwide is less than 10%, a significant gap compared to field crops. Furthermore, the harvesting of premium teas is highly seasonal, with a very short harvesting period, which places a huge demand on labor.

[0003] Currently, most methods for locating premium teas rely on two-dimensional images. Successful location depends on the clear visibility of the tender shoot picking point. However, in actual tea garden environments, the tea stems for picking premium teas are often obscured. Research on occlusion tasks focuses on restoring spatial image data to gain visual inertia, rather than estimating shape and position. These methods consume significant time and computational resources. While they can improve system robustness, they clearly cannot meet the needs of agricultural harvesting tasks. Therefore, there is an urgent need to extract target features for specific task objects and conduct research on shape restoration and position estimation. Furthermore, due to the short tea harvesting cycle, high labor costs, and tea occlusion issues, proposing a method for automatically picking tea at night by identifying picking points is of great significance. This method can enable fully automated tea harvesting at night, saving costs, improving tea quality, and contributing to the rapid development of the tea industry. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an intelligent robot method for identifying tea shoot picking points based on multi-source data fusion. This addresses the problems of numerous noise points and insufficient brightness in images under nighttime conditions. The invention then proposes a DFOCC-YOLOv8 model to obtain two-dimensional detection boxes for tea leaves, compensating for channel and spatial features to solve the problem of missed detections of small tea targets and obtain more accurate tea leaf location information. An adaptive occlusion loss function is introduced to improve the problem of inaccurate identification due to occlusion. Next, a multi-source feature adaptive fusion tea shoot segmentation model is constructed, proposing feature extraction methods based on color gradient analysis, multi-scale texture response, dynamic shape symmetry analysis, and multi-directional tea growth characteristics. These methods process the two-dimensional tea shoot boxes obtained by S2, and the adaptive fusion mechanism improves the model's segmentation accuracy and robustness in complex tea leaf images. Finally, point cloud data from LiDAR is introduced to obtain depth information, and a picking point positioning system based on depth data optimization and wind speed correction is constructed. By modeling the growth characteristics of tea leaves and combining depth data optimization techniques, the depth of the tea shoot picking point is inferred using the fusion of leaf growth angle and point cloud data. In addition, considering the influence of wind speed, wind direction and environmental factors on depth measurement, wind error correction technology was introduced. By adjusting the depth data in real time, the error caused by wind speed fluctuations was eliminated, thereby significantly improving the harvesting accuracy and stability.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for identifying tea shoot picking points using intelligent robots based on multi-source data fusion, comprising the following steps:

[0007] S1: Acquire nighttime images of tea leaves and enhance the images based on an image decomposition model to address the issues of numerous noise points and insufficient brightness in nighttime images.

[0008] S2: Use the DFOCC-YOLOv8 model to detect tea shoots in the enhanced image, obtain two-dimensional detection boxes for tea leaves, and filter the detection boxes to remove tea leaves that do not meet the requirements.

[0009] S3: Construct a tea shoot segmentation model using heterogeneous multi-feature weighted fusion, process the two-dimensional detection box of tea obtained in step S2, remove background unrelated to tea, and obtain the tea shoot segmentation image;

[0010] S4: Introduce point cloud data from LiDAR to obtain depth information, and combine it with segmented images of tea shoots for data synchronization and registration; construct a picking point positioning system based on depth data optimization and wind speed correction, optimize and correct the acquired depth data, and determine the three-dimensional coordinates of the tea shoot picking point.

[0011] Furthermore, S1 specifically includes the following steps:

[0012] S11: Mount a camera on the arm of the tea-picking robot, tilt it downwards above the tea ridge, 30-50cm away from the tea leaves, at an angle of [30°, 60°], to obtain images of the tea leaves in a nighttime environment; install a lidar next to the camera to obtain point cloud data of the tea leaves in a nighttime environment.

[0013] S12: A nighttime image enhancement method based on an image decomposition model is proposed, which decomposes the acquired nighttime image into a structure layer and a texture layer; a global brightness factor is constructed to enhance the brightness of the structure map, as shown in Equation (1):

[0014]

[0015] Among them, L g (x,y) represents the global brightness factor, I'(x,y) represents the brightness value of the input image, and μ image σ represents the global mean brightness of the image. image The standard deviation of image brightness is represented by ν, which is a small constant to avoid division by zero. η controls the intensity of brightness enhancement, γ is a non-linear exponential adjustment parameter, and I' is the standard deviation of image brightness. max The maximum brightness value of the image is 255, β is the scaling factor, λ is a parameter that adjusts the enhancement intensity of low-light areas, and σ is a parameter that adjusts the enhancement intensity of low-light areas. max This represents the maximum standard deviation of the image's brightness.

[0016] The pixel values ​​of the input image are standardized, and the local contrast is calculated by combining the global mean and standard deviation.

[0017] By using a Gaussian distribution function to assign weights based on the distance between pixel values ​​and the maximum pixel value, the target brightness range can be highlighted.

[0018] By combining local texture or brightness gradient information, the dynamic range is compressed through logarithmic transformation to optimize the details and edge representation of the image;

[0019] S13: Construct a local brightness factor to enhance the details of the texture map, as shown in Equation (2):

[0020]

[0021] Among them, L local (x,y) represents the local brightness factor, I(x,y) represents the pixel value of the input image at position (x,y), and N k (x,y) represents the k-th layer filter of the Gaussian pyramid, where k is a positive integer, and μ local (x,y), σ local(x, y) represent the mean and standard deviation of the local region of the image at position (x, y), respectively. The local region is a circular region with a radius of 2 centered at (x, y). The gradient at position (x, y) is represented by K, and K represents the total number of filters.

[0022] The brightness difference between a pixel and its neighborhood is used to measure local changes, while gradient information is introduced to suppress abrupt changes in order to reduce noise interference.

[0023] By locally normalizing pixel values, details are highlighted and local contrast is enhanced.

[0024] Balance local detail enhancement with overall smoothing effect using adjustable parameters;

[0025] S14: The structure layer and texture layer processed by the night image enhancement method based on the image decomposition model are fused together for subsequent processing.

[0026] Furthermore, S2 specifically includes:

[0027] S21: YOLOv8 is used to extract features from the image to obtain the location information of the tea leaves; a DFOCC-YOLOv8 model is constructed to obtain the two-dimensional detection box of the tea leaves; a dual-feature wrapping model is constructed on the basis of YOLOv8 to wrap and compensate the spatial features and channel features; data augmentation is first performed on the dataset to improve the generalization ability of the model; then the three feature maps of different scales output by YOLOv8 are fed into the dual-feature wrapping module.

[0028] S22: Perform channel feature extraction, as shown in equation (3):

[0029]

[0030] Where Conv represents one-dimensional convolution, Conv k The subscript k in F represents the kernel size. c (e,u) represents the feature map of the e-th row, u-th column, and c-th channel, H represents the height of the feature map, W represents the width of the feature map, σ represents the Sigmoid activation function, F1 represents the feature map after channel feature recalibration, Conv1 represents a one-dimensional convolution with a 1×1 kernel, and Conv3 represents a one-dimensional convolution with a 3×3 kernel.

[0031] Global average pooling is performed on the input feature map to obtain a feature map with a scale of 1×1×c. Cross-channel interaction is used to improve the feature correlation between different channels. Features are extracted again using 1×1 convolution, and the corresponding weight coefficients are generated using the Sigmoid function for channel-weighted fusion of the input feature map.

[0032] S23: Spatial feature extraction, as shown in equation (4);

[0033] F2=σ(Conv21(ξ(Conv23(Tea ReLU(Conv1(F c (e,u)))))) (4)

[0034] The input feature map is convolved to halve the number of channels; spatial information is extracted using 3×3 convolution, and ReLU is used for activation to enhance the nonlinearity of the model; then 1×1 convolution is used to reduce the dimensionality of the channels, and TeaReLU activation function is constructed to filter features, improve speed and reduce the number of parameters, as shown in Equation (5):

[0035]

[0036] Where Conv2 represents two-dimensional convolution, Conv... k In the diagram, the subscript k represents the kernel size, σ represents the Sigmoid activation function, F2 represents the feature map after spatial feature recalibration, F represents the original feature map, ξ represents the ReLU function, Conv21 represents a 2D convolution with a 1×1 kernel, Conv23 represents a 2D convolution with a 3×3 kernel, and x represents the input value of the neuron. It is a threshold parameter that controls the inflection point of nonlinearity.

[0037] Finally, the Sigmoid function is used to generate the corresponding weight coefficients, which are used for spatial weighted fusion of the input feature maps.

[0038] S24: The feature map obtained after being wrapped by channel features and spatial features is sent to the DFOCC-YOLOv8 model to detect the tea shoot bounding box and calculate the length of the tea shoot bounding box. If it does not meet the required length, it is discarded. In order to improve the accuracy, the DFOCC-YOLOv8 model constructs an adaptive occlusion loss function, as shown in Equation (6). The smaller IoU(a,b) is, the more severe the occlusion is, the greater the difficulty of sample learning, and the larger the corresponding weight.

[0039]

[0040] Where 1-IoU(a,b) is the occlusion weight, L occ Let χ denote the occlusion loss function, a represent the predicted bounding box, b represent the ground truth bounding box, and χ² represent the occlusion loss function. pos κ is the location loss weighting coefficient. cls These are the weight coefficients of the classification loss function. Represents the true label (0 or 1) of the nth class, logc n c represents the predicted probability of the nth class. nThe logarithm of S is used to measure the uncertainty between the predicted probability and the true label. a S represents the area of ​​the prediction box. b N represents the area of ​​the true bounding box, and N represents the total number of categories.

[0041] Furthermore, S3 specifically includes the following steps:

[0042] S31: Extract the color feature information of tea leaves; first, introduce gradient information by calculating the rate of change of the intensity values ​​of the three color channels (red, green, and blue) in the horizontal and vertical directions in the image to detect areas of drastic color change; then, combine local pixel differences and statistical characteristics, highlight details and enhance contrast through local normalization, and use adjustable parameters to balance local detail enhancement and overall smoothing effect to optimize image details and texture, adapt to visual tasks in complex environments, as shown in Equation (7):

[0043]

[0044] Among them, I R (x,y), I G (x,y), I B (x, y) represent the red, green, and blue channel color values ​​of a pixel (x, y) in the image, respectively. These represent the color differences between the pixel and its neighboring pixels in the horizontal and vertical directions, respectively, f. color This indicates the weight of the color characteristics of tea leaves in image segmentation;

[0045] S32: Extract the texture feature information of tea leaves; First, obtain image texture-related information by summing operations on different directions and scales; Next, calculate the sum of local pixel intensity in the denominator, subtract a specific value from it and take the absolute value to measure local differences; Then, process this difference through exponential functions and parameters to suppress abrupt changes and reduce noise; Finally, use the formula parts and adjustable parameters to balance local detail enhancement and overall smoothing effect, and optimize image texture, as shown in Equation (8):

[0046]

[0047] Among them, R d (x,y,s) represents the texture response, L s(x,y,d) represents local contrast or local texture intensity, n' represents the size of the complexity measurement region, ε represents the learning rate of the weighting factor, I(x+i',y+j') represents the pixel value at position (x+i',y+j') in the image, d represents the number of directions, referring to how many different directions are considered to capture texture variation information, s represents the number of scales, referring to the number of regions of different sizes or resolutions selected when extracting texture features, i' represents the offset in the x-direction relative to the current position, j' represents the offset in the y-direction relative to the current position, and f texture The weight of the texture characteristics of tea leaves in image segmentation is represented by D, which represents the total number of different directions selected when extracting texture features, and S represents the total number of different region sizes selected when extracting texture features.

[0048] S33: To reflect the morphological changes of tea leaves, a new method based on shape manifold and local symmetry analysis is proposed, as shown in equation (9):

[0049]

[0050] in, It is the second derivative, representing the curvature of the edge; y(q) represents the vertical coordinate value of the q-th point of the shape profile; L is the total length of the profile; f shape This indicates the weight of the shape characteristics of tea leaves in image segmentation;

[0051] First, the second derivative part is similar to "gradient information," which measures the rate of shape change, including the difference in brightness between a pixel and its neighborhood to measure local changes.

[0052] Next, the summation part combines "local pixel differences" and "statistical properties" to highlight the local changes of the function;

[0053] Finally, the formula combines the two to balance the rate of shape change and local differences, thereby achieving an assessment of the overall shape characteristics.

[0054] S34: A novel method for extracting the growth pattern features of tea leaves is proposed, as shown in Equation (10). It focuses on the differences in growth direction, local spatial consistency, and correlation of growth patterns of tea leaves. By quantifying the differences in growth direction between adjacent pixels, measuring the growth consistency of local areas, and introducing an angle difference metric, the spatial structure changes of tea leaves at different growth stages are captured.

[0055]

[0056] Where x and y represent the position of the current pixel in the image, i and j represent the coordinates of adjacent pixels, and Δθ ijΔG represents the difference in growth direction between pixel i and pixel j, g represents the Gaussian-weighted standard deviation, used to control the influence range of the difference in growth direction within a local region. ij G represents the difference in growth pattern between pixel i and pixel j. L θ represents the standard deviation of the local spatial weighting, used to control the weighting range. ij This represents the growth direction angle between adjacent pixels i and j, describing the difference in the growth angle of tea leaves. g P This represents the standard deviation weighted by growth patterns, used to control the weighted range of correlation between growth patterns. The standard deviation of the fusion weights is used to control the influence range of each feature during the fusion process. grow This indicates the weight of tea plant growth characteristics in image segmentation;

[0057] S35: Construct a tea shoot segmentation model based on multi-source feature adaptive fusion, as shown in Equation (11), and design a learning mechanism to predict the importance of each pixel in different feature dimensions.

[0058] Specifically, the weight of each feature is predicted by calculating the features of local image regions and combining them with contextual information.

[0059] First, the mapping function learned through the convolutional neural network can determine the feature weights based on local features, and the sigmoid activation function suppresses the interference caused by local differences.

[0060] Next, the different feature maps are processed and multiplied with parameters to highlight the changes in different features and enhance local contrast.

[0061] Finally, by comprehensively processing different features and using segmentation algorithms to balance the enhancement of local details with the overall fusion effect, the image's comprehensive features are optimized.

[0062] f fusion =UNet(σ(φ) color (f color ))·P+σ(φ texture (f texture ))·P+σ(φ shape (f shape ))·P+σ(φ grow (f grow ))·P)(11)

[0063] Among them, f fusion (x,y) is the segmented feature map, σ represents the Sigmoid activation function, and φ color φ texture φ shape φ growIt is a mapping function learned through a convolutional neural network (CNN), which predicts the weight of each feature based on the local features of the input image. UNet represents the segmentation algorithm, and P represents the detection box of tea shoots obtained by S24.

[0064] Furthermore, S4 specifically includes:

[0065] S41: Extract key points from the tea leaf segmentation image obtained in S35; use the HRNet key point detection method to extract the b-th (b∈N) key point. + The coordinates of the intersection points of the tea shoot leaves are used to determine the two-dimensional coordinates of the tea shoot picking point by taking 1-2 mm down from the intersection points. Then, the segmented image and the point cloud data from the LiDAR are fused to achieve data synchronization and registration; and the LiDAR data is projected onto the segmented image to obtain the depth value of the tea leaf portion.

[0066] S42: Using depth data optimization technology, combining the growth characteristics of tea leaves with the depth data of the point cloud of tea shoots and leaves, as shown in Equation (12); Specifically, by modeling the growth angle of tea leaves, adjusting the depth value of the leaves, overcoming the depth deviation caused by tea growth, and inferring the depth of the picking point; At the same time, considering that the leaf points that are closer to the centroid of the leaf are on the same plane as the picking point, these points are given higher weights to avoid contradictions when the location of the picking point is unknown;

[0067]

[0068] in, This represents the depth value of the i-th point on the b-th tea shoot leaf. This represents the depth value of the b-th tea shoot picking point. This represents the growth angle of the leaf of the b-th tea shoot. This indicates that a model is established for the growth characteristics of the b-th tea shoot leaf to obtain the depth of the tea shoot picking point. c is a constant that controls the influence of distance on the weight, and n represents the total number of point clouds on the b-th tea shoot leaf.

[0069] S43: In the tea picking task, the correction of depth by wind speed and wind direction is considered; as shown in Equation (13), by dynamically considering the real-time changes of wind speed, wind direction and environmental factors, and combining the correction function to accurately adjust the depth data, wind error is eliminated, and picking accuracy and stability are improved.

[0070]

[0071] Where, ΔZ wind(t) represents the wind error correction value, indicating the amount of correction for depth by wind speed and direction; v(t) represents the wind speed at time t; θ(t) represents the wind direction angle at time t; w1 represents the initial influence coefficient of wind speed and direction on depth error; w2 is an exponential decay coefficient used to represent the weakening of the influence of wind speed over time; w3 is a time-dependent correction function used to control the influence of wind speed changes on depth over time. The time constant of the wind speed error is represented by f(E(t)), which represents the environmental correction function. Considering the influence of environmental factors such as humidity and temperature on the correction of wind speed and wind direction, the environmental correction function f(E(t)) is used to adjust the wind speed error, where E(t) represents the environmental variable at time t.

[0072] S44: After completing the depth correction, determine the location of the picking point for the tender tea shoots.

[0073] A smart robot system for identifying tea shoot picking points based on multi-source data fusion includes:

[0074] The image acquisition module is used to acquire nighttime images of tea leaves;

[0075] The image enhancement module is used to enhance images based on an image decomposition model.

[0076] The tea shoot detection module is used to detect tea shoots in the enhanced image and obtain a two-dimensional detection box for the tea leaves.

[0077] The tea shoot segmentation module is used to process the two-dimensional detection box of tea leaves, remove backgrounds unrelated to tea leaves, and obtain segmented images of tea shoots.

[0078] The depth information acquisition module is used to acquire depth information by introducing point cloud data from LiDAR and to perform data synchronization and registration by combining it with segmented images of tea shoots.

[0079] The picking point positioning module is used to build a picking point positioning system based on depth data optimization and wind speed correction. It optimizes and corrects the depth data to determine the three-dimensional coordinates of the picking point of tea shoots.

[0080] The beneficial effects of this invention are as follows:

[0081] (1) This invention proposes a nighttime image enhancement model based on a decomposition model to solve the problem of a large number of noise points and insufficient brightness in images under nighttime conditions, so that tea-picking robots can work at night, reduce labor costs, and avoid missing the best time for tea picking.

[0082] (2) Compared to traditional mechanical tea picking, which suffers from damage to tender buds and the presence of numerous impurities within them, this invention proposes a DFOCC-YOLOv8 model to obtain a two-dimensional detection box for tea leaves. Calculating the size of the two-dimensional detection box can filter out tea leaves that are too small or too old. A dual-channel winding model is constructed based on YOLOv8, complementing spatial and channel features to solve the problem of missed detection of small tea targets. An occlusion loss function is introduced to improve the problem of inaccurate identification due to occlusion.

[0083] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0084] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0085] Figure 1 This is the main flowchart of the intelligent robot tea shoot identification method based on multi-source data fusion described in this embodiment of the invention;

[0086] Figure 2 This is a structural diagram of the dual-channel winding module described in an embodiment of the invention. Detailed Implementation

[0087] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0088] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0089] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0090] In this embodiment, a 32-line Velodyne lidar is selected as the depth sensor, and a binocular depth camera d435i is used as the image data sensor. The algorithm is implemented in the Ubuntu system.

[0091] A method for identifying tea shoot picking points using intelligent robots based on multi-source data fusion, comprising the following steps:

[0092] S1: A nighttime image enhancement method based on image decomposition is proposed. This method decomposes low-light nighttime images into a structure layer and a texture layer, optimizing large-scale object contours and small-scale details respectively. A global brightness factor optimizes the brightness of the structure layer, highlighting the target brightness range; a local brightness factor enhances texture layer details, suppresses noise, and strengthens local contrast.

[0093] S2: A method for detecting tea shoot tenderness based on the DFOCC-YOLOv8 model is proposed. This model addresses the problem of small target detection in tea shoot tenderness by constructing a dual-feature wrapping module to compensate for spatial and channel features. A TeaReLU activation function is proposed to smooth low-value inputs and retain high-value inputs, enhancing the extraction of detailed features. Subsequently, an adaptive occlusion loss function is proposed to dynamically adjust the weights of occluded samples.

[0094] S3: A tea shoot segmentation model based on multi-source feature adaptive fusion is proposed. This model proposes a color feature extraction method based on gradient information and local normalization, introduces shape manifold and local symmetry analysis methods, and designs a feature extraction method that focuses on the differences in tea growth patterns to capture the spatial structural changes of tea at different growth stages.

[0095] S4: A picking point positioning system based on depth data optimization and wind speed correction is proposed. This system addresses the problem of lidar not being able to scan areas of the stems by using depth data optimization technology combined with tea growth characteristics to infer the depth of the picking point. Then, considering the dynamic changes in wind speed and direction, a correction function is constructed to adjust the depth data and eliminate wind-related errors.

[0096] like Figure 1 As shown, step S1 specifically includes the following steps:

[0097] S11: Mount the camera on the arm of the tea-picking robot, tilt it downwards above the tea ridge, 30-50cm away from the tea leaves, at an angle of [30°, 60°], to obtain images of the tea leaves in a nighttime environment; the lidar can be installed next to the camera to obtain point cloud data of the tea leaves in a nighttime environment.

[0098] S12: A nighttime image enhancement method based on an image decomposition model is proposed, which decomposes the acquired nighttime image into a structure layer and a texture layer. Since nighttime images usually contain a lot of noise and insufficient brightness, a global brightness factor is constructed, as shown in Equation (1), to enhance the brightness of the structure map. First, the pixel values ​​of the input image are standardized, and the local contrast is calculated by combining the global mean and standard deviation; then, the Gaussian distribution function is used to assign weights based on the distance between the pixel value and the maximum pixel value to highlight the target brightness range; finally, the dynamic range is compressed by logarithmic transformation by combining local texture or brightness gradient information to optimize the details and edge performance of the image.

[0099]

[0100] Among them, L g (x,y) represents the global brightness factor, I'(x,y) represents the brightness value of the input image, and μ image σ represents the global mean brightness of the image. image The standard deviation of image brightness is represented by ν, which is a small constant to avoid division by zero. η controls the intensity of brightness enhancement, γ is a non-linear exponential adjustment parameter, and I' is the standard deviation of image brightness. max The maximum brightness value of the image is 255, β is the scaling factor, λ is a parameter that adjusts the enhancement intensity of low-light areas, and σ is a parameter that adjusts the enhancement intensity of low-light areas. max This represents the maximum standard deviation of the image's brightness.

[0101] S13: Construct a local brightness factor, as shown in Equation (2). To enhance the detail of the texture map, firstly, the brightness difference between the pixel and its neighborhood is used to measure local changes, and gradient information is introduced to suppress abrupt changes in order to reduce noise interference. Then, by locally normalizing the pixel values, detail changes are highlighted and local contrast is enhanced. Finally, adjustable parameters are used to balance the local detail enhancement and the overall smoothing effect.

[0102]

[0103] Among them, L local (x,y) represents the local brightness factor, I(x,y) represents the pixel value of the input image at position (x,y), and N k(x,y) represents the k-th layer filter of the Gaussian pyramid, where k is a positive integer, and μ local (x,y), σ local (x, y) represent the mean and standard deviation of the local region of the image at position (x, y), respectively. The local region is a circular region with a radius of 2 centered at (x, y). Let K represent the gradient at position (x, y), and K represent the total number of filters.

[0104] S14: Effectively fuse the structure layer and texture layer after processing by the night image enhancement method based on the image decomposition model for subsequent processing.

[0105] S2 specifically includes:

[0106] S21: YOLOv8 is used to extract features from the image to obtain the location information of the tea leaves. Since tea leaves are small targets, directly using YOLOv8 for detection will lead to missed detections and false positives. Therefore, a DFOCC-YOLOv8 model is constructed to obtain the two-dimensional detection box for the tea leaves. A dual-feature wrapping model is built on top of YOLOv8 to intertwine and compensate for spatial and channel features. First, data augmentation is performed on the dataset to improve the model's generalization ability. Then, the three feature maps at different scales output by YOLOv8 are fed into the dual-feature wrapping module, the structure of which is as follows: Figure 2 As shown:

[0107] S22: Perform channel feature extraction as shown in Equation (3). First, perform global average pooling on the input feature map to obtain a feature map with a scale of 1×1×c. Then, use cross-channel interaction to improve the feature correlation between different channels. Use 1×1 convolution to extract features again. Finally, use the Sigmoid function to generate the corresponding weight coefficients for channel-weighted fusion of the input feature map.

[0108]

[0109] Where Conv represents one-dimensional convolution, Conv k The subscript k in F represents the kernel size. c (e,u) represents the feature map of the e-th row, u-th column, and c-th channel, H represents the height of the feature map, W represents the width of the feature map, σ represents the Sigmoid activation function, F1 represents the feature map after channel feature recalibration, Conv1 represents a one-dimensional convolution with a 1×1 kernel, and Conv3 represents a one-dimensional convolution with a 3×3 kernel.

[0110] S23: Spatial feature extraction, as shown in Equation (4). First, convolution is performed on the input feature map to halve the number of channels; then, 3×3 convolution is used to extract spatial information, and ReLU function is used for activation to enhance the nonlinearity of the model; then, 1×1 convolution is used to reduce the dimensionality of the number of channels, and TeaReLU activation function is constructed to filter features, as shown in Equation (5), which improves speed and reduces the number of parameters. Finally, the Sigmoid function is used to generate the corresponding weight coefficients for spatial weighted fusion of the input feature map.

[0111] F2=σ(Conv21(ξ(Conv23(Tea ReLU(Conv1(F c (e,u)))))) (4)

[0112]

[0113] Where Conv2 represents two-dimensional convolution, Conv... k In the diagram, the subscript k represents the kernel size, σ represents the Sigmoid activation function, F2 represents the feature map after spatial feature recalibration, F represents the original feature map, ξ represents the ReLU function, Conv21 represents a 2D convolution with a 1×1 kernel, Conv23 represents a 2D convolution with a 3×3 kernel, and x represents the input value of the neuron. It is a threshold parameter that controls the inflection point of nonlinearity.

[0114] S24: The feature map obtained after the channel features and spatial features are wrapped together is sent to the DFOCC-YOLOv8 model to detect the tea shoot bounding boxes and calculate the length of the tea shoot bounding boxes. If the length does not meet the required length, the bounding boxes are discarded. In order to further improve the accuracy, the model also constructs an adaptive occlusion loss function, as shown in Equation (6), where the smaller the IoU(a,b), the more severe the occlusion, the greater the difficulty of sample learning, and the larger the corresponding weight.

[0115]

[0116] Where 1-IoU(a,b) is the occlusion weight, L occ Let χ denote the occlusion loss function, a represent the predicted bounding box, b represent the ground truth bounding box, and χ² represent the occlusion loss function. pos κ is the location loss weighting coefficient. cls These are the weight coefficients of the classification loss function. Represents the true label (0 or 1) of the nth class, logc n c represents the predicted probability of the nth class. n The logarithm of the equation is used to measure the uncertainty between the predicted probability and the true label.

[0117] Sa S represents the area of ​​the prediction box. b N represents the area of ​​the true bounding box, and N represents the total number of categories.

[0118] S3 specifically includes the following steps:

[0119] S31: Extract the color feature information of tea leaves. First, gradient information is introduced by calculating the rate of change of the intensity values ​​of the red, green and blue color channels in the horizontal and vertical directions to detect areas of drastic color change. Then, by combining local pixel differences and statistical characteristics, local normalization and other operations are performed to highlight details and enhance contrast. Adjustable parameters are used to balance the local detail enhancement and overall smoothing effect to optimize image details and texture and adapt to visual tasks in complex environments, as shown in Equation (7).

[0120]

[0121] Among them, I R (x,y), I G (x,y), I B (x, y) represent the red, green, and blue channel color values ​​of a pixel (x, y) in the image, respectively. These represent the color differences between the pixel and its neighboring pixels in the horizontal and vertical directions, respectively, f. color This indicates the weight of the color characteristics of tea leaves in image segmentation.

[0122] S32: Extracting the texture features of tea leaves. First, image texture-related information is obtained by summing operations on different directions and scales. Next, the sum of local pixel intensities is calculated in the denominator, and a specific value is subtracted from it and the absolute value is taken to measure local differences. Then, this difference is processed by an exponential function and parameters to suppress abrupt changes and reduce noise. Finally, the local detail enhancement and overall smoothing effect are balanced using the various parts of the formula and adjustable parameters to optimize the image texture, as shown in Equation (8).

[0123]

[0124] Among them, R d (x,y,s) represents the texture response, L s(x,y,d) represents local contrast or local texture intensity, n' represents the size of the complexity measurement region, ε represents the learning rate of the weighting factor, I(x+i',y+j') represents the pixel value at position (x+i',y+j') in the image, d represents the number of directions, referring to how many different directions are considered to capture texture variation information, s represents the number of scales, referring to the number of regions (or resolutions) of different sizes selected when extracting texture features, i' represents the offset in the x-direction relative to the current position, j' represents the offset in the y-direction relative to the current position, and f texture The weight of the tea leaf texture characteristics in image segmentation is represented by D, which represents the total number of different directions selected when extracting texture features, and S represents the total number of different region sizes selected when extracting texture features.

[0125] S33: To more accurately reflect the morphological changes of tea leaves, a new method based on shape manifold and local symmetry analysis is proposed, as shown in Equation (9). First, the second derivative part is similar to "gradient information" to measure the rate of shape change, such as the difference in brightness between a pixel and its neighborhood to measure local changes. Next, the summation part combines "local pixel differences" and "statistical characteristics" to highlight the local changes of the function. Finally, the formula integrates the two to balance the rate of shape change and local differences, thereby achieving an evaluation of the overall shape characteristics.

[0126]

[0127] in, It is the second derivative, representing the curvature of the edge; y(q) represents the vertical coordinate value of the q-th point of the shape profile; L is the total length of the profile; f shape This represents the weight of the shape characteristics of tea leaves in image segmentation.

[0128] S34: A novel method for extracting the growth pattern features of tea leaves is proposed, as shown in Equation (10), which focuses on the differences in growth direction, local spatial consistency, and correlation of growth patterns of tea leaves. By quantifying the differences in growth direction between adjacent pixels, measuring the growth consistency of local regions, and introducing an angle difference metric, the spatial structure changes of tea leaves at different growth stages are captured.

[0129]

[0130] Where x and y represent the position of the current pixel in the image, i and j represent the coordinates of adjacent pixels, and Δθ ij ΔG represents the difference in growth direction between pixel i and pixel j, g represents the Gaussian-weighted standard deviation, used to control the influence range of the difference in growth direction within a local region. ij G represents the difference in growth pattern between pixel i and pixel j. Lθ represents the standard deviation of the local spatial weighting, used to control the weighting range. ij This represents the growth direction angle between adjacent pixels i and j, describing the difference in the growth angle of tea leaves. g P This represents the standard deviation weighted by growth patterns, used to control the weighted range of correlation between growth patterns. The standard deviation of the fusion weights is used to control the influence range of each feature during the fusion process. grow This indicates the weight that the growth characteristics of tea leaves account for in image segmentation.

[0131] S35: A segmentation model for tea shoots based on multi-source feature adaptive fusion is constructed, as shown in Equation (11). A learning mechanism is designed to predict the importance of each pixel in different feature dimensions. Specifically, the weight of each feature is predicted by calculating the features of local image regions and combining them with contextual information. First, the mapping function learned by the convolutional neural network can determine the feature weight based on local features. The sigmoid activation function can suppress the interference caused by excessive local differences, just like introducing gradient information. Next, after processing different feature maps, they are multiplied with parameters to highlight the changes in different features and enhance local contrast. Finally, different features are processed comprehensively, and the segmentation algorithm is used to balance the enhancement of local details and the overall fusion effect, thereby achieving optimized processing of the comprehensive features of the image.

[0132] f fusion =UNet(σ(φ) color (f color ))·P+σ(φ texture (f texture ))·P+σ(φ shape (f shape ))·P+σ(φ grow (f grow ))·P) (11)

[0133] Among them, f fusion (x,y) is the segmented feature map, σ represents the Sigmoid activation function, and φ color φ texture φ shape φ grow It is a mapping function learned through a convolutional neural network (CNN) that predicts the weight of each feature based on the local features of the input image. UNet represents the segmentation algorithm, and P represents the detection box of tea shoots obtained by S24.

[0134] S4 specifically includes:

[0135] S41: Extract key points from the tea leaf segmentation image obtained in S35. Use the HRNet key point detection method to extract the b-th (b∈N) key point. +The coordinates of the intersection points of the tea shoot leaves are used to determine the two-dimensional coordinates of the tea shoot picking point by taking 1-2 mm down from the intersection points. Then, the segmented image and the point cloud data from the LiDAR are fused to achieve data synchronization and registration. The LiDAR data is then projected onto the segmented image to obtain the depth value of the tea leaf portion.

[0136] S42: Since the picking point of the tea shoot is located at the stem and is small in size, the lidar cannot effectively scan this area, resulting in the inability to directly obtain the depth value of the picking point. Therefore, a depth data optimization technique was adopted, which combines the growth characteristics of tea leaves with the depth data of the point cloud of the tea shoot leaves for inference, as shown in Equation (12). Specifically, by modeling the growth angle of the tea leaves, the depth value of the leaves is adjusted to overcome the depth deviation caused by the growth of tea leaves and accurately infer the depth of the picking point. At the same time, considering that the leaf points closer to the centroid of the leaf are more likely to be on the same plane as the picking point, these points are given higher weights. This strategy effectively avoids the depth contradiction that may occur when the location of the picking point is unknown.

[0137]

[0138] in, This represents the depth value of the i-th point on the b-th tea shoot leaf. This represents the depth value of the b-th tea shoot picking point. This represents the growth angle of the leaf of the b-th tea shoot. This indicates that a model is established for the growth characteristics (angle) of the b-th tea shoot leaf to obtain the depth of the tea shoot picking point. c is a constant that controls the influence of distance on the weight, and n represents the total number of point clouds on the b-th tea shoot leaf.

[0139] S43: In tea picking, wind speed and direction may affect the depth measurement results, especially since dynamic changes in wind may cause fluctuations in sensor readings and changes in the location of tea shoot picking points, leading to errors (wind kinetic errors). Therefore, it is necessary to consider the correction of depth by wind speed and direction. As shown in Equation (13), by dynamically considering the real-time changes in wind speed, wind direction and environmental factors, and combining the correction function to accurately adjust the depth data, wind kinetic errors can be eliminated, and picking accuracy and stability can be improved.

[0140]

[0141] Where, ΔZ wind(t) represents the wind error correction value, indicating the amount of correction for depth by wind speed and direction; v(t) represents the wind speed at time t; θ(t) represents the wind direction angle at time t; w1 represents the initial influence coefficient of wind speed and direction on depth error; w2 is an exponential decay coefficient used to represent the weakening of the influence of wind speed over time, usually due to air resistance or other environmental factors; w3 is a time-dependent correction function used to control the influence of wind speed changes on depth over time. The time constant representing the wind dynamism error is typically the response time to wind speed fluctuations or environmental factors. f(E(t)) represents the environmental correction function. Considering the influence of environmental factors such as humidity and temperature on wind speed and direction corrections, the environmental correction function f(E(t)) is used to further adjust the wind dynamism error to ensure depth accuracy. E(t) represents the environmental variables at time t, such as humidity and temperature.

[0142] S44: After completing the depth correction, determine the location of the picking point for the tender tea shoots.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for identifying tea shoot picking points using intelligent robots based on multi-source data fusion, characterized in that: The method includes the following steps: S1: Acquire nighttime images of tea leaves and enhance the images based on an image decomposition model to address the issues of numerous noise points and insufficient brightness in nighttime images. S2: Use the DFOCC-YOLOv8 model to detect tea shoots in the enhanced image, obtain two-dimensional detection boxes for tea leaves, and filter the detection boxes to remove tea leaves that do not meet the requirements; the DFOCC-YOLOv8 model is a dual-feature winding model built on the basis of YOLOv8, which intertwines and compensates spatial features and channel features. S3: Construct a tea shoot segmentation model based on weighted fusion of heterogeneous multi-features. Process the two-dimensional detection box of tea obtained in step S2, remove the background unrelated to tea, and obtain the segmented image of tea shoots. The heterogeneous multi-features are the color characteristics, texture characteristics, shape characteristics, and growth characteristics of tea. S4: Introduce point cloud data from LiDAR to obtain depth information, and combine it with segmented images of tea shoots for data synchronization and registration; construct a picking point positioning system based on depth data optimization and wind speed and direction correction, optimize and correct the acquired depth data, and determine the three-dimensional coordinates of the tea shoot picking point.

2. The method for identifying tea shoot picking points using intelligent robots based on multi-source data fusion according to claim 1, characterized in that: S1 specifically includes the following steps: S11: Mount a camera on the arm of the tea-picking robot, tilt it downwards above the tea ridge, 30-50cm away from the tea leaves, at an angle of [30°, 60°], to obtain images of the tea leaves in a nighttime environment; install a lidar next to the camera to obtain point cloud data of the tea leaves in a nighttime environment. S12: A nighttime image enhancement method based on an image decomposition model is proposed, which decomposes the acquired nighttime image into a structure layer and a texture layer; a global brightness factor is constructed to enhance the brightness of the structure map, as shown in Equation (1): Among them, L g (x,y) represents the global brightness factor, I'(x,y) represents the brightness value of the input image, and μ image σ represents the global mean brightness of the image. image The standard deviation of image brightness is represented by ν, which is a small constant to avoid division by zero, η represents the intensity of brightness enhancement, γ represents the non-linear exponential adjustment parameter, and I' represents the standard deviation of image brightness. max σ represents the maximum brightness value of the image, which is 255; β represents the scaling factor; λ represents the parameter that adjusts the enhancement intensity of low-light areas; and σ represents the maximum brightness value of the image. max This represents the maximum standard deviation of the image's brightness. The pixel values ​​of the input image are standardized, and the local contrast is calculated by combining the global mean and standard deviation. By using a Gaussian distribution function to assign weights based on the distance between pixel values ​​and the maximum pixel value, the target brightness range can be highlighted. By combining local texture or brightness gradient information, the dynamic range is compressed through logarithmic transformation to optimize the details and edge representation of the image; S13: Construct a local brightness factor to enhance the details of the texture map; The brightness difference between a pixel and its neighborhood is used to measure local changes, while gradient information is introduced to suppress abrupt changes in order to reduce noise interference. By locally normalizing pixel values, details are highlighted and local contrast is enhanced. Balance local detail enhancement with overall smoothing effect using adjustable parameters; S14: The structure layer and texture layer processed by the night image enhancement method based on the image decomposition model are fused together for subsequent processing.

3. The method for identifying tea shoot picking points using an intelligent robot based on multi-source data fusion according to claim 2, characterized in that: S2 specifically includes: S21: YOLOv8 is used to extract features from the image to obtain the location information of the tea leaves; a DFOCC-YOLOv8 model is constructed to obtain the two-dimensional detection box of the tea leaves; a dual-feature wrapping model is constructed on the basis of YOLOv8 to wrap and compensate the spatial features and channel features; data augmentation is first performed on the dataset to improve the generalization ability of the model; then the three feature maps of different scales output by YOLOv8 are fed into the dual-feature wrapping module. S22: Perform channel feature extraction, as shown in equation (3): Where Conv represents one-dimensional convolution, Conv k The subscript k in F represents the kernel size. c (e,u) represents the feature map of the e-th row, u-th column, and c-th channel, H represents the height of the feature map, W represents the width of the feature map, δ represents the Sigmoid activation function, F1 represents the feature map after channel feature recalibration, Conv1 represents a one-dimensional convolution with a 1×1 kernel, and Conv3 represents a one-dimensional convolution with a 3×3 kernel. Global average pooling is performed on the input feature map to obtain a feature map with a scale of 1×1×c. Cross-channel interaction is used to improve the feature correlation between different channels. Features are extracted again using 1×1 convolution, and the corresponding weight coefficients are generated using the Sigmoid function for channel-weighted fusion of the input feature map. S23: Spatial feature extraction, as shown in equation (4); F2=σ(Conv21(ξ(Conv23(Tea ReLU(Conv1(F c (e,u))))))) (4) The input feature map is convolved to halve the number of channels; spatial information is extracted using 3×3 convolution, and ReLU is used for activation to enhance the nonlinearity of the model; then 1×1 convolution is used to reduce the dimensionality of the channels, and TeaReLU activation function is constructed to filter features, improve speed and reduce the number of parameters, as shown in Equation (5): Where Conv2 represents two-dimensional convolution, Conv... k In the diagram, the subscript k represents the kernel size, σ represents the sigmoid activation function, F2 represents the feature map after spatial feature recalibration, ξ represents the ReLU function, Conv21 represents a 2D convolution with a 1×1 kernel, Conv23 represents a 2D convolution with a 3×3 kernel, and x represents the input value of the neuron. It is a threshold parameter that controls the inflection point of nonlinearity. Finally, the Sigmoid function is used to generate the corresponding weight coefficients, which are used for spatial weighted fusion of the input feature maps. S24: The feature map obtained after being wrapped by channel features and spatial features is sent to the DFOCC-YOLOv8 model to detect the tea shoot bounding box and calculate the length of the tea shoot bounding box. If it does not meet the required length, it is discarded. In order to improve the accuracy, the DFOCC-YOLOv8 model constructs an adaptive occlusion loss function, as shown in Equation (6). The smaller IoU(a,b) is, the more severe the occlusion is, the greater the difficulty of sample learning, and the larger the corresponding weight. Where 1-IoU(a,b) is the occlusion weight, L occ Let χ denote the occlusion loss function, a represent the predicted bounding box, b represent the ground truth bounding box, and χ² represent the occlusion loss function. pos κ is the location loss weighting coefficient. cls These are the weight coefficients of the classification loss function. Represents the true label (0 or 1) of the nth class, logc n c represents the predicted probability of the nth class. n The logarithm of S is used to measure the uncertainty between the predicted probability and the true label. a S represents the area of ​​the prediction box. b N represents the area of ​​the true bounding box, and N represents the total number of categories.

4. The method for identifying tea shoot picking points using an intelligent robot based on multi-source data fusion according to claim 3, characterized in that: S3 specifically includes the following steps: S31: Extract the color feature information of tea leaves; first, introduce gradient information by calculating the rate of change of the intensity values ​​of the three color channels (red, green, and blue) in the horizontal and vertical directions in the image to detect areas of drastic color change; then, combine local pixel differences and statistical characteristics, highlight details and enhance contrast through local normalization, and use adjustable parameters to balance local detail enhancement and overall smoothing effect to optimize image details and texture, adapt to visual tasks in complex environments, as shown in Equation (7): Among them, I R (x,y), I G (x,y), I B (x, y) represent the red, green, and blue channel color values ​​of a pixel (x, y) in the image, respectively. These represent the color differences between the pixel and its neighboring pixels in the horizontal and vertical directions, respectively, f. color This indicates the weight of the color characteristics of tea leaves in image segmentation; S32: Extract the texture feature information of tea leaves; First, obtain image texture-related information by summing operations on different directions and scales; Next, calculate the sum of local pixel intensity in the denominator, subtract a specific value from it and take the absolute value to measure local differences; Then, process this difference through exponential functions and parameters to suppress abrupt changes and reduce noise; Finally, use the formula parts and adjustable parameters to balance local detail enhancement and overall smoothing effect, and optimize image texture, as shown in Equation (8): Among them, R d (x,y,s) represents the texture response, L s (x,y,d) represents local contrast or local texture intensity, n' represents the size of the complexity measurement region, ε represents the learning rate of the weighting factor, I(x+i',y+j') represents the pixel value at position (x+i',y+j') in the image, d represents the number of directions, referring to how many different directions are considered to capture texture variation information, s represents the number of scales, referring to the number of regions of different sizes or resolutions selected when extracting texture features, i' represents the offset in the x-direction relative to the current position, j' represents the offset in the y-direction relative to the current position, and f texture The weight of the texture characteristics of tea leaves in image segmentation is represented by D, which represents the total number of different directions selected when extracting texture features, and S represents the total number of different region sizes selected when extracting texture features. S33: To reflect the morphological changes of tea leaves, a method based on shape manifold and local symmetry analysis is proposed, as shown in equation (9): in, It is the second derivative, representing the curvature of the edge; y(q) represents the vertical coordinate value of the q-th point of the shape profile; L is the total length of the profile; f shape This indicates the weight of the shape characteristics of tea leaves in image segmentation; First, the second derivative part is similar to "gradient information," which measures the rate of shape change, including the difference in brightness between a pixel and its neighborhood to measure local changes. Next, the summation part combines "local pixel differences" and "statistical properties" to highlight the local changes of the function; Finally, the formula combines the two to balance the rate of shape change and local differences, thereby achieving an assessment of the overall shape characteristics. S34: A method for extracting the growth pattern features of tea is proposed, as shown in Equation (10), which focuses on the differences in growth direction, local spatial consistency and growth pattern correlation of tea; by quantifying the differences in growth direction between adjacent pixels, measuring the growth consistency of local areas and introducing angle difference measurement, the spatial structure changes of tea at different growth stages are captured. Where x and y represent the position of the current pixel in the image, i and j represent the coordinates of adjacent pixels, and Δθ ij ΔG represents the difference in growth direction between pixel i and pixel j, g represents the Gaussian-weighted standard deviation, used to control the influence range of the difference in growth direction within a local region. ij G represents the difference in growth pattern between pixel i and pixel j. L θ represents the standard deviation of the local spatial weighting, used to control the weighting range. ij This represents the growth direction angle between adjacent pixels i and j, describing the difference in the growth angle of tea leaves. g P This represents the standard deviation weighted by growth patterns, used to control the weighted range of correlation between growth patterns. The standard deviation of the fusion weights is used to control the influence range of each feature during the fusion process. grow This indicates the weight of tea plant growth characteristics in image segmentation; S35: Construct a tea shoot segmentation model based on multi-source feature adaptive fusion, as shown in Equation (11), and design a learning mechanism to predict the importance of each pixel in different feature dimensions. Specifically, the weight of each feature is predicted by calculating the features of local image regions and combining them with contextual information. First, the mapping function learned through the convolutional neural network can determine the feature weights based on local features, and the sigmoid activation function suppresses the interference caused by local differences. Next, the different feature maps are processed and multiplied with parameters to highlight the changes in different features and enhance local contrast. Finally, by comprehensively processing different features and using segmentation algorithms to balance the enhancement of local details with the overall fusion effect, the image's comprehensive features are optimized. f fusion =UNet(σ(φ color (f color ))·P+σ(φ texture (f texture ))·P+σ(φ shape (f shape ))·P+σ(φ grow (f grow ))·P) (11) Among them, f fusion This is the segmented feature map, where σ represents the Sigmoid activation function, and φ... color φ texture φ shape φ grow It is a mapping function learned through a convolutional neural network (CNN), which predicts the weight of each feature based on the local features of the input image. UNet represents the segmentation algorithm, and P represents the detection box of tea shoots obtained by S24.

5. The method for identifying tea shoot picking points using an intelligent robot based on multi-source data fusion according to claim 4, characterized in that: S4 specifically includes: S41: Extract key points from the tea leaf segmentation image obtained in S35; use the HRNet key point detection method to extract the coordinate information of the intersection point of the b-th tea shoot leaf, where b∈N. + Then, take 1-2 mm down from the intersection point to determine the two-dimensional coordinates of the tea shoot picking point. Then, the segmented image and the point cloud data from the LiDAR are fused to achieve data synchronization and registration; and the LiDAR data is projected onto the segmented image to obtain the depth value of the tea leaf portion. S42: Using depth data optimization technology, combining the growth characteristics of tea leaves with the depth data of the point cloud of tea shoots and leaves, as shown in Equation (12); Specifically, by modeling the growth angle of tea leaves, adjusting the depth value of the leaves, overcoming the depth deviation caused by tea growth, and inferring the depth of the picking point; At the same time, considering that the leaf points that are closer to the centroid of the leaf are on the same plane as the picking point, these points are given higher weights to avoid contradictions when the location of the picking point is unknown; in, This represents the depth value of the i-th point on the b-th tea shoot leaf. This represents the depth value of the b-th tea shoot picking point. This represents the growth angle of the leaf of the b-th tea shoot. This indicates that a model is established for the growth characteristics of the b-th tea shoot leaf to obtain the depth of the tea shoot picking point. c is a constant that controls the influence of distance on the weight, and n represents the total number of point clouds on the b-th tea shoot leaf. S43: In the tea picking task, the correction of depth by wind speed and wind direction is considered; as shown in Equation (13), by dynamically considering the real-time changes of wind speed, wind direction and environmental factors, and combining the correction coefficient to accurately adjust the depth data, wind error is eliminated, and picking accuracy and stability are improved. Where, ΔZ wind (t) represents the wind error correction value, indicating the amount of correction for depth by wind speed and direction; v(t) represents the wind speed at time t; θ(t) represents the wind direction angle at time t; w1 represents the initial influence coefficient of wind speed and direction on depth error; w2 is an exponential decay coefficient used to represent the weakening of the influence of wind speed over time; w3 is a time-dependent correction function used to control the influence of wind speed changes on depth over time. The time constant of the wind speed error is represented by f(E(t)), which represents the environmental correction function. Considering the influence of environmental factors such as humidity and temperature on the correction of wind speed and wind direction, the environmental correction function f(E(t)) is used to adjust the wind speed error, where E(t) represents the environmental variable at time t. S44: After completing the depth correction, determine the location of the picking point for the tender tea shoots.

6. A smart robot tea shoot picking point identification system based on multi-source data fusion, characterized in that: include: The image acquisition module is used to acquire nighttime images of tea leaves; The image enhancement module is used to enhance images based on an image decomposition model. The tea shoot detection module uses the DFOCC-YOLOv8 model to detect tea shoots in the enhanced image, obtains two-dimensional detection boxes for tea leaves, and filters the detection boxes to remove tea leaves that do not meet the requirements. The DFOCC-YOLOv8 model is a dual-feature winding model built on the basis of YOLOv8, which intertwines and compensates spatial features and channel features. The tea shoot segmentation module constructs a tea shoot segmentation model based on weighted fusion of heterogeneous multi-features. It processes the two-dimensional detection boxes of tea leaves obtained by the tea shoot detection module, removes backgrounds unrelated to tea leaves, and obtains segmented images of tea shoots. The heterogeneous multi-features are the color characteristics, texture characteristics, shape characteristics, and growth characteristics of tea leaves. The depth information acquisition module is used to acquire depth information by introducing point cloud data from LiDAR and to perform data synchronization and registration by combining it with segmented images of tea shoots. The picking point positioning module is used to build a picking point positioning system based on depth data optimization and wind speed and direction correction. It optimizes and corrects the depth data to determine the three-dimensional coordinates of the picking point of tea shoots.

Images