Method for estimating picking posture of pod pepper based on visual and tactile joint perception

Abstract

The invention provides a pod pepper picking attitude estimation method based on visual and tactile combined perception, relates to the technical field of agricultural product picking attitude estimation, and remarkably improves the precision and adaptability of pod pepper picking attitude estimation through dynamic fusion of visual and tactile data and six-degree-of-freedom pose collaborative calculation. Based on weighted harmony of a gradient magnitude mean value of a pepper body mask boundary and a depth map shielding rate, visual confidence is generated, a tactile pressure mean value, a variance and a friction force fluctuation coefficient are combined with tactile confidence, and a normalized exponential function is utilized to dynamically distribute weights, so that when a pepper body is shielded by branches and leaves and visual features are fuzzy, the visual feature is fuzzy. The touch confidence coefficient weight is improved, the visual weight is enhanced when a touch signal is interfered by mechanical vibration, self-adaptive complementation of multi-modal data is achieved, the sensitivity of vision to pepper body contour deformation is enhanced by calculating the six-degree-of-freedom poses of vision and touch, and the judgment capacity of touch to the contact state is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural product picking posture estimation, and in particular to a chili pepper picking posture estimation method based on visual and tactile joint perception. Background Art

[0002] With the development of agricultural automation, robotic picking technology has been widely used in the field of fruit and vegetable harvesting. However, the farmland environment is relatively complex and is prone to conditions such as lighting changes, occlusion by branches and leaves, and variable fruit postures. These conditions pose severe challenges to the accuracy and robustness of picking posture estimation. Small-sized, high-density fruits such as chili peppers are easily obstructed by branches and leaves and have fragile surfaces, requiring harvesting robots to have millimeter-level positioning capabilities and dynamic obstacle avoidance strategies.

[0003] Traditional harvesting robots mainly rely on single-modal sensors, such as vision or touch, for posture estimation. The vision system locates the fruit through RGB image segmentation and key point detection, but has a high risk of failure in occluded or reflective scenes; the tactile system determines the clamping state through pressure distribution, but lacks direct output of spatial posture, making it difficult to independently complete precise grasping.

[0004] Among existing technologies, visual perception methods often use RGB-D cameras to capture pepper images and estimate their pose through instance segmentation and keypoint detection. However, when obstructed by branches or uneven lighting, segmentation accuracy decreases, leading to pose errors. Tactile perception methods use flexible pressure sensors to obtain contact force information and utilize force-pose mapping models to infer the pepper's position. However, the pressure distribution on the contact surface is susceptible to vibration and slippage in the harvesting robot, making pose estimation based on single-modal tactile data insufficiently robust.

[0005] Current visual-tactile fusion methods mostly use fixed-weight superposition of visual and tactile pose data. When the pepper body is partially occluded or the tactile signal is partially ineffective, the sensor confidence cannot be dynamically adjusted, causing the fusion result to deviate from the true pose.

[0006] The above information disclosed in this Background section is only for enhancement of understanding of the background of the present disclosure and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention

[0007] The purpose of the present invention is to provide a method for estimating the picking posture of chili peppers based on combined visual and tactile perception to solve the problems raised in the above background technology.

[0008] To achieve the above object, the present invention provides the following technical solutions: A chili pepper picking posture estimation method based on visual and tactile joint perception includes the following steps: The researchers collected RGB images, depth maps, and tactile data between the picking site and the chili pepper. Based on the RGB images, they generated a mask of the chili pepper's main body and its mask boundary. The occlusion rate was calculated using the depth map, and the mean, variance, and friction fluctuation coefficient of the contact pressure between the picking site and the chili pepper were obtained based on the tactile data. The mean gradient amplitude of the pixel points at the mask boundary is calculated and weighted harmonic average is performed in combination with the occlusion rate to obtain the visual confidence. The pressure mean, variance, and friction fluctuation coefficient are integrated to obtain the tactile confidence. The visual confidence and tactile confidence are processed as a weighted sum to generate dynamic visual and tactile weights. Key points of the chili pepper are detected in the RGB image, and the key points are combined with the depth map to map the three-dimensional coordinates of the key feature points. The visual six-degree-of-freedom pose of the chili pepper relative to the RGB image acquisition point is calculated. The tactile data is used as the feature input of the pre-trained BP neural network to obtain the tactile six-degree-of-freedom pose of the chili pepper relative to the picking position. The six-degree-of-freedom pose includes a rotation matrix and a translation vector. Convert the visual and tactile rotation matrices into quaternions, interpolate and fuse them according to the dynamic weights of vision and touch to generate the final rotation data, and linearly weight the visual and tactile translation vectors according to the dynamic weights of vision and touch to obtain the final translation data; The final six-degree-of-freedom pose data of the chili pepper is obtained based on the final rotation data and the final translation data.

[0009] Furthermore, the RGB image and depth map of the chili pepper have the same resolution, and each pixel in the RGB image is mapped one-to-one to each pixel in the depth map, that is, each pixel in the RGB image has a corresponding depth value. The steps of generating the mask and mask boundary of the chili pepper body include: Obtain existing RGB images of chili peppers, perform pixel-level annotation on the chili pepper body in each image, and mark the binary mask of the pepper body outline; Build an instance segmentation model using Mask R-CNN as the baseline model. Use the existing RGB image of chili peppers as the feature input and the corresponding binary mask of the annotated pepper outline as the instance segmentation model label. Train the instance segmentation model. Use the pre-trained instance segmentation model to segment the pepper body of the collected RGB image, and obtain a mask of the pepper body with the pixel value of the pepper body area being 1 and the pixel value of the background part (non-pepper body part) being 0; Use edge detection algorithm to extract the boundary from the mask of the main body of the chili pepper to obtain the mask boundary of the main body of the chili pepper; When calculating the occlusion rate, the bottommost pixel row in the RGB image of the chili pepper is the first row, and the leftmost pixel column is the first column. The pixel positions in the mask of the chili pepper body and the mask boundary are mapped one by one using coordinates, so that each pixel has a unique coordinate value. Get the depth values of all pixels in the mask area in the depth map, calculate the mean depth of all pixels in the mask area, and set the depth difference threshold. The formula based on the depth difference threshold is: ; in, and Represent the depth mean and standard deviation of all pixels in the mask area, Represents the total number of pixels in the mask area. Indicates the first The depth value of each pixel, and Respectively represent the The horizontal and vertical coordinates of the pixel points, represents the depth threshold; like , then the horizontal and vertical coordinates in the marked mask area are and The pixels in the mask area are occluded pixels. The number of all occluded pixels is counted, and the ratio of the number of occluded pixels to the total number of pixels in the mask area is the occlusion rate.

[0010] Furthermore, the logic for obtaining visual confidence is: Convert the RGB image into a grayscale image, use the Sobel operator to calculate the directional gradient amplitude in the row and column directions for each pixel point on the mask boundary, calculate the gradient amplitude of each mask boundary pixel point and normalize it, and calculate the normalized gradient amplitude mean of all pixels on the mask boundary; The visual confidence is obtained by taking the weighted harmonic average of the normalized gradient amplitude mean and the occlusion rate. The formula is: ; in, represents the visual confidence, and represent the mean gradient amplitude and occlusion rate respectively, and denote the gradient weight and occlusion rate weight respectively, and , .

[0011] Furthermore, the logic for calculating the six-degree-of-freedom pose of the chili pepper relative to the visual position at the RGB image acquisition location is: Define at least four key feature points of chili peppers, obtain existing RGB images of chili peppers, annotate the key feature points in each image to form a feature training set, and build a key feature point prediction model based on the HRNet network. Use the RGB images in the feature training set as feature inputs and output the corresponding key feature points to train the key feature point prediction model. Input the collected RGB image into the trained key feature point prediction model to obtain the predicted coordinates of the key feature points in the collected RGB image; For the predicted positions of key feature points, the depth values of the corresponding positions are directly read from the depth map to obtain the 3D predicted coordinates of each key feature point in the chili pepper coordinate system. The intrinsic parameter matrix of the acquisition camera is used to convert the 3D predicted coordinates of each key feature point into the 3D camera coordinate system coordinates, and the center of mass of each key feature point in the chili pepper coordinate system and the center of mass of each key feature point in the 3D camera coordinate system are determined respectively. Obtain the decentralized points of each key feature point in the chili pepper coordinate system and the decentralized points of each key feature point in the chili pepper coordinate system, construct a covariance matrix, perform singular value decomposition on the covariance matrix, and calculate the visual rotation matrix and translation vector.

[0012] Furthermore, the tactile data is a 10×10 pressure matrix, and the method for obtaining the pressure mean, variance, and friction fluctuation coefficient of the contact between the picking part and the chili pepper is: There are 100 pressure data in the 10×10 pressure matrix. The sum of the 100 pressure data is calculated and the average is the mean pressure of the picking part in contact with the chili pepper. The variance is calculated based on the pressure mean. The formula is: ; in, represents the variance of the mean pressure, Indicates the first Rank The pressure value of the column, Indicates the mean pressure; The Coulomb friction model is constructed using the pressure matrix and the friction coefficient of the picking part in contact with the chili pepper to obtain a 10×10 friction matrix. The Sobel operator is used to calculate the spatial gradient of each element in the friction matrix in both the row and column directions. Row, No. The spatial gradients of the elements in the row and column directions are calibrated as and , calculate the friction fluctuation coefficient based on the formula: ; in, and represent the mean value of friction and the friction fluctuation coefficient respectively, represents the friction matrix Row, No. The elements of the column represent the friction force.

[0013] Furthermore, the logic for obtaining tactile confidence is: The pressure mean is normalized according to the following formula: ; in, represents the mean pressure after normalization, and Respectively represent the maximum and minimum pressure values in the pressure matrix, Indicates the mean pressure; The logic for calculating tactile confidence is: ; in, represents the tactile confidence, 、 and Represent the pressure mean weight, pressure variance weight and friction fluctuation weight respectively, ,and .

[0014] Furthermore, the logic for obtaining the six-degree-of-freedom pose of the chili pepper relative to the picking position is: A 10×10 pressure matrix between the chili pepper and the picking site is collected in advance, and the actual six-degree-of-freedom pose data of the chili pepper relative to the picking site is simultaneously recorded and used as a label. Expand the pre-collected 10×10 pressure matrix to form a 100-dimensional vector, build a BP network model, input the 100-dimensional vector into the BP network model to be trained, and use the corresponding six-degree-of-freedom posture data as a label to train the BP network model to be trained; The collected tactile data is expanded and a 100-dimensional vector is constructed and input into the trained BP network model to obtain the six-degree-of-freedom position of the chili pepper relative to the picking part, that is, the tactile rotation matrix and translation vector of the chili pepper relative to the picking part.

[0015] Furthermore, when generating the dynamic weights of vision and touch, a normalized exponential function is used to dynamically allocate the weights, based on the following formula: ; in, and Represent visual confidence and tactile confidence respectively, and Represent the dynamic weights of vision and touch respectively.

[0016] Furthermore, the rotation matrix is a 3×3 orthogonal matrix that describes the rotation transformation around a certain coordinate axis or an arbitrary axis in three-dimensional space. The rotation matrix is expressed as: ; in, Represents the rotation matrix, the internal elements express Middle Rank Elements of the column; The conversion formula for converting a rotation matrix to a quaternion is: ; in, Represents the rotation matrix The quaternion representation of and 、 、 Represent the real part and three imaginary parts of the quaternion respectively; When the final rotation data is generated by the dynamic weighted interpolation fusion of vision and touch, the quaternion converted from the rotation matrix of vision and touch is subtracted to generate the fused quaternion. The formula is: ; in, Represents the fused quaternion, and Represent the dynamic weights of vision and touch respectively, and They are the quaternions converted from the rotation matrices of vision and touch respectively. is the angle between two quaternions; The fused quaternion is converted into a rotation matrix according to the conversion formula of rotation matrix and quaternion, and calibrated as the final rotation data.

[0017] Furthermore, the visual and tactile translation vectors are linearly weighted according to the dynamic weights of vision and touch, and the formula for obtaining the final translation data is: ; in, and Represents the transpose of the visual and tactile translation vectors, and Represent the visual and tactile translation vectors respectively, Represents the final translation data, and Represent the dynamic weights of vision and touch respectively.

[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention significantly improves the accuracy and adaptability of chili pepper picking posture estimation through dynamic fusion of visual and tactile data and collaborative solution of six-degree-of-freedom posture. The visual confidence is generated based on the weighted harmonization of the gradient amplitude mean of the pepper mask boundary and the occlusion rate of the depth map. The tactile confidence is fused with the mean and variance of the tactile pressure and the friction fluctuation coefficient. The weight is dynamically allocated using a normalized exponential function. This ensures that the tactile confidence weight is increased when the pepper body is blocked by branches and leaves, causing the visual features to be blurred. The visual weight is enhanced when the tactile signal is disturbed by mechanical vibration, thereby realizing adaptive complementarity of multimodal data.

[0019] The visual and tactile six-degree-of-freedom poses are solved respectively through RGB image key point detection and tactile data-driven BP neural network. The rotation matrix is converted into quaternion for interpolation fusion and translation vector linear weighted fusion, which effectively suppresses the cumulative error of single-sensor pose solution. The introduction of the mean value of the gradient amplitude of the pepper mask boundary enhances the visual sensitivity to the deformation of the pepper contour, while the multi-dimensional fusion of the tactile pressure mean, variance and friction fluctuation coefficient improves the tactile ability to distinguish the contact state. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the overall method of the present invention. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to specific embodiments.

[0022] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative position relationships. When the absolute position of the object being described changes, the relative position relationship may also change accordingly.

[0023] Example: See also Figure 1 , the present invention provides a technical solution: A chili pepper picking posture estimation method based on visual and tactile joint perception includes the following steps: Step 1: Collect the RGB image, depth map, and tactile data between the picking part and the chili pepper. Generate the mask and mask boundary of the chili pepper body based on the RGB image, calculate the occlusion rate using the depth map, and obtain the mean, variance, and friction fluctuation coefficient of the contact pressure between the picking part and the chili pepper based on the tactile data.

[0024] In this embodiment, the resolution of the RGB image and the depth map of the chili pepper is the same, and each pixel in the RGB image is mapped one-to-one to each pixel in the depth map, that is, each pixel in the RGB image has a corresponding depth value. The steps of generating the mask and mask boundary of the chili pepper body include: Obtain existing RGB images of chili peppers, perform pixel-level annotation on the chili peppers in each image, and create a binary mask that represents the chili pepper outline. Use an RGB-D camera to capture chili pepper images from multiple angles in a real farmland or simulated environment, covering different lighting conditions (including occlusions) and maturity states. Perform pixel-level annotation on the chili peppers in each image, generating bounding boxes and instance masks. The bounding boxes mark the location of each chili pepper, while the instance masks accurately represent the binary masks of the chili pepper outlines. Semi-automatic annotation is performed using LabelMe, CVAT, or a professional agricultural annotation platform such as Roboflow. Color thresholding is used to assist in generating the initial mask, and edges are manually corrected.

[0025] An instance segmentation model was constructed using Mask R-CNN as the baseline model. The existing RGB image of chili peppers was used as the feature input. The binary mask corresponding to the annotated pepper outline was used as the label for the instance segmentation model. The instance segmentation model was trained. Mask R-CNN, as the baseline model, has a two-stage structure (first detecting the bounding box, then generating the mask), which is suitable for complex occlusion scenarios.

[0026] The pre-trained instance segmentation model is used to segment the pepper body of the collected RGB image, and a mask of the pepper body is obtained, in which the pixel value of the pepper body area is 1 and the pixel value of the background part (non-pepper body part) is 0.

[0027] An edge detection algorithm is used to extract the boundary from the mask of the main body of the chili pepper. The mask boundary of the main body of the chili pepper is obtained. The contour detection algorithm used is OpenCV's findContours. The pixel coordinates of the closed boundary are obtained to form a polygonal boundary. Subsequently, the RGB image is converted to a grayscale image and the mask area is retained.

[0028] When calculating the occlusion rate, the bottommost pixel row of the RGB image of the chili pepper is used as the first row, and the leftmost pixel column is used as the first column. The pixel positions in the mask of the chili pepper body and the mask boundary are mapped one by one using coordinates, so that each pixel has a unique coordinate value.

[0029] This example uses multimodal data fusion and high-precision instance segmentation to generate masks and boundaries for chili peppers. An RGB-D camera is used to capture chili pepper images from multiple angles in real farmland, covering different lighting, occlusion, and maturity states. A semi-automatic annotation tool is combined with a color threshold for the pepper body to generate an initial mask. Edges are manually corrected to ensure pixel-level annotation accuracy. An instance segmentation model is constructed based on Mask R-CNN, taking RGB images as input and binary masks as labels. Mask R-CNN's two-stage structure (first detecting the bounding box, then generating the mask) preserves spatial details through ROI Align, enhancing segmentation robustness in complex occlusion scenarios. The trained model is used to segment the RGB images captured in real time, outputting binary masks of the pepper body region (pixel value 1) and background (pixel value 0). OpenCV's findContours algorithm is then used to extract closed boundaries, and the occlusion rate is calculated in combination with the depth map.

[0030] Strict pixel alignment between RGB and depth maps ensures a one-to-one mapping between masked areas and depth values, ensuring spatial consistency for subsequent occlusion rate calculations. Training data is enhanced through synthetic occlusion (e.g., virtual leaf overlays) and analyzed in conjunction with depth maps to effectively distinguish between true pepper areas and occlusion noise. Semi-automatic annotation tools reduce manual workload and improve annotation efficiency by combining pre-screening of pepper color thresholds. Training with multi-angle and multi-illumination data adapts the model to complex field environments. While traditional methods rely on single RGB data and are time-consuming, this solution integrates RGB-D information and verifies mask spatial consistency with depth maps, improving segmentation accuracy.

[0031] Accurate masks and boundaries are the core inputs for calculating visual confidence, directly affecting the dynamic allocation of visual-tactile weights. The gradient of the mask boundary reflects the clarity of the pepper's outline. Combined with the occlusion rate of the depth map, the weighted harmonic average visual confidence can dynamically suppress errors in occluded or blurred scenes. The lightweight deployment of Mask R-CNN meets real-time requirements.

[0032] Get the depth values of all pixels in the mask area in the depth map, calculate the mean depth of all pixels in the mask area, and set the depth difference threshold. The formula based on the depth difference threshold is: ; in, and Represent the depth mean and standard deviation of all pixels in the mask area, Represents the total number of pixels in the mask area. Indicates the first The depth value of each pixel, and Respectively represent the The horizontal and vertical coordinates of the pixel points, represents the depth threshold; like , then the horizontal and vertical coordinates in the marked mask area are and The pixels in the mask area are occluded pixels. The number of all occluded pixels is counted, and the ratio of the number of occluded pixels to the total number of pixels in the mask area is the occlusion rate.

[0033] The standard deviation reflects the degree of discreteness of the depth value, that is, the change in the depth of the pepper surface. Setting the depth threshold to 2 times the standard deviation is based on the empirical rule in statistics, which believes that approximately 95% of the data falls within the range of two standard deviations of the mean. Therefore, pixels below this threshold may be outliers and may be occluded. Through dynamic thresholds, the system can adapt to the depth changes of different scenes and avoid the shortcomings of fixed thresholds in complex environments. For example, when the pepper body is partially obscured, the depth value fluctuates greatly, and the standard deviation will increase accordingly, thereby adjusting the threshold to more accurately distinguish the obscured area.

[0034] This embodiment dynamically sets a depth difference threshold to identify occluded pixels by calculating the mean and standard deviation of the depth distribution within the masked area. Its core logic is based on two key points: the depth mean reflects the average depth position of the pepper surface and is used to locate the target subject's reference position in three-dimensional space. The standard deviation quantifies the degree of dispersion in the depth distribution. The threshold is set at twice the standard deviation (covering approximately 95% of normally distributed data), effectively distinguishing the pepper body from abnormal depth regions (such as occluders or noise).

[0035] By statistically analyzing depth distribution, we can accurately distinguish the main pepper body from occlusions (such as leaves and branches), avoiding missegmentation caused by color confusion in RGB images. Existing techniques often rely on fixed thresholds or RGB texture analysis, which are susceptible to lighting changes and color interference. This method, combined with depth statistics, can improve the accuracy of occlusion detection.

[0036] In this embodiment, the tactile data is a 10×10 pressure matrix. The method for obtaining the pressure mean, variance, and friction fluctuation coefficient of the contact between the picking part and the chili pepper is as follows: There are a total of 100 pressure data in the 10×10 pressure matrix. The sum of the 100 pressure data is calculated and the average is the mean pressure of the contact between the picking part and the chili pepper. The mean pressure represents the average pressure intensity of the contact between the picking part and the chili pepper, reflecting whether the grasping force is moderate. The variance is calculated based on the mean pressure, based on the formula: ; in, represents the variance of the mean pressure, Indicates the first Rank The pressure value of the column, Represents the mean pressure. The larger the variance, the more uneven the pressure distribution, which may lead to sliding risk.

[0037] The pressure mean is a simple average, while the variance is the average of the squared differences between each pressure value and the mean. The friction force fluctuation coefficient involves gradient calculation. The Sobel operator is used to calculate the spatial gradient, and then the gradient amplitude is statistically analyzed. The force mean indicates whether the overall grasping force is moderate, and the variance reflects the uniformity of the pressure distribution. The friction force fluctuation coefficient captures the dynamic changes of the contact surface through gradient calculation. Traditional methods may only use a single pressure value or simple statistics. However, the use of variance and gradient analysis here can more comprehensively capture the contact state, improving detection accuracy and robustness.

[0038] This embodiment achieves a refined quantitative assessment of the grasping state by constructing a tactile parameter system of pressure mean, variance, and friction fluctuation coefficient. The formula for calculating the pressure mean directly reflects the overall pressure intensity of the contact between the clamping part and the chili pepper. Its essence is to calculate the average force of 100 pressure sensors. This parameter can quickly determine whether the grasping force is within a safe range, avoiding slippage caused by too little clamping force or damage to the fruit caused by too much clamping force. The pressure variance reveals the uniformity of the pressure distribution by calculating the degree of deviation of each pressure value from the mean. The larger the variance, the more abnormal the local pressure (such as overpressure at a single point or uneven contact surface), which may lead to reduced grip stability. The key to this embodiment is to expand the tactile data from a single intensity indicator to a joint "mean-variance" feature, which not only describes the overall grasping force level but also captures spatial distribution anomalies, providing multi-dimensional information input for subsequent tactile confidence calculations.

[0039] This invention significantly improves the robustness of tactile perception by integrating the mean and variance of pressure. Traditional tactile analysis typically focuses only on the maximum or total pressure value, failing to identify hidden risks caused by abnormal pressure distribution. In this invention, the mean pressure is used to determine whether the clamping force is appropriate, while the variance serves as a sensitive indicator of the uniformity of pressure distribution.

[0040] The Coulomb friction model is constructed using the pressure matrix and the friction coefficient of the picking part in contact with the chili pepper to obtain a 10×10 friction matrix. The Sobel operator is used to calculate the spatial gradient of each element in the friction matrix in both the row and column directions. Row, No. The spatial gradients of the elements in the row and column directions are calibrated as and .

[0041] When calculating the row direction gradient value and vertical direction gradient value of each element in the friction matrix, the friction value of the element and the friction value of its adjacent elements are multiplied with the row direction gradient template of the Sobel operator, and the results of all products are added to obtain the row direction gradient value of the element. The friction value of the element and the friction value of its adjacent elements are multiplied with the column direction gradient template of the Sobel operator, and the results of all products are added to obtain the column direction gradient value of the element. If the friction value of a certain adjacent element does not exist, it is set to 0 during the calculation. The formula for calculating the row direction gradient value and the column direction gradient value is as follows: ; ; in, is the friction matrix Row, No. The friction value of the column elements.

[0042] This example uses the Sobel operator to perform spatial gradient analysis on the friction matrix and constructs a dynamic slip risk perception model. The Sobel operator's row and column gradient templates are essentially discrete differential operators. Their design principle is to enhance the response of edges or sudden changes by weighting the differences between neighboring pixels. In the friction matrix, the row gradient reflects the rate of change of horizontal friction, while the column gradient captures the severity of vertical changes.

[0043] This method overcomes the limitations of traditional tactile perception through local gradient analysis. Traditional methods typically rely on the average or extreme friction values to determine slip risk, but are unable to identify early signs of local stress concentration or progressive slip. In this embodiment, the convolution operation of the Sobel operator uses a weighted calculation over a 3×3 neighborhood, suppressing sensor noise while amplifying the edge features of continuous friction changes.

[0044] The calculation of the friction fluctuation coefficient is based on the formula: ; in, and represent the mean value of friction and the friction fluctuation coefficient respectively, represents the friction matrix Row, No. The elements of the column represent the friction force.

[0045] The friction force fluctuation coefficient constructed in this embodiment achieves a multi-dimensional quantitative assessment of the mechanical state of the contact surface by fusing the global mean and local gradient characteristics. The friction force mean is used as the denominator for normalization processing, aiming to eliminate the scale effect of different clamping force levels on the gradient amplitude, so that the fluctuation coefficient can purely reflect the relative change intensity of the friction force distribution. The numerator calculates the root mean square of the gradient amplitude at all positions, which is essentially the integration of the spatial gradient energy on the contact surface, which both amplifies the contribution of the local mutation area and suppresses random noise through the averaging operation.

[0046] This embodiment transcends the traditional tactile system's reliance on a single dimension of frictional "intensity" by incorporating "spatial distribution uniformity" into its stability assessment system for the first time. Traditional methods typically monitor only extreme or mean frictional values, but fail to identify latent instability patterns such as "normal center pressure with sudden gradient changes at the edges." In this embodiment, the fluctuation coefficient encodes the contact surface mechanical state as a scalar metric through the statistical analysis of gradient energy.

[0047] This example establishes a strong correlation between the spatiotemporal variation of friction and gripping stability. Traditional tactile features (such as pressure mean and range) primarily reflect static mechanical states. However, the friction fluctuation coefficient, using the Sobel gradient operator to extract the spatial differential characteristics of the friction field, can sensitively capture changes in mechanical distribution caused by geometric deformation of the contact surface.

[0048] Step 2: Calculate the mean gradient amplitude of the pixel points at the mask boundary and perform weighted harmonic averaging with the occlusion rate to obtain the visual confidence. The pressure mean, variance, and friction fluctuation coefficient are integrated to obtain the tactile confidence. The visual confidence and tactile confidence are processed as a weighted sum to generate dynamic visual and tactile weights.

[0049] In this embodiment, the logic for obtaining visual confidence is: Convert the RGB image into a grayscale image, and use the Sobel operator to calculate the directional gradient amplitude and directional gradient in the row and column directions for each pixel point on the mask boundary. When calculating the row and column gradient values of each pixel point, multiply the grayscale value of the pixel point and the grayscale values of its adjacent pixels with the row gradient template of the Sobel operator, and add the results of all the products to obtain the row gradient value of the pixel. Multiply the grayscale value of the pixel point and the grayscale value of its adjacent pixels with the column gradient template of the Sobel operator, and add the results of all the products to obtain the column gradient value of the pixel. The formula for calculating the row and column gradient values is: ; ; in, 、 They are the first pixels in the mask boundary Row, No. The row direction gradient value and column direction gradient value of the column pixel point, is the pixel point at the membrane boundary Row, No. The grayscale value of a column pixel. If the adjacent pixel at a certain location does not exist, the grayscale value is set to 0 during calculation.

[0050] The formula for generating the gradient magnitude of a pixel is: ; in, For the Row, No. Column mask gradient magnitude at boundary pixels.

[0051] This embodiment calculates the gradient amplitude of the mask boundary through the Sobel operator, and constructs the visual confidence in combination with the occlusion rate, thereby realizing a dynamic quantitative evaluation of the reliability of visual information. The row-direction template and column-direction template of the Sobel operator are essentially discrete differential operators, and their convolution operation strengthens the boundary response by weighting the difference between neighboring pixels. The gradient amplitude synthesizes the row and column gradient vectors into a scalar, reflecting the edge strength of the pixel point - the clearer the outline of the pepper, the higher the gradient amplitude and the stronger the visual confidence; while blurred or broken boundaries result in a decrease in the gradient amplitude. At the same time, the occlusion rate quantifies the degree to which the pepper is covered by external objects by counting the proportion of abnormal pixels in the depth map.

[0052] Calculate the gradient amplitude of each mask boundary pixel and perform normalization processing, calculate the normalized gradient amplitude mean of all pixels on the mask boundary, and normalize the gradient amplitude of all mask boundary pixels to Within the range of , the normalized gradient amplitude mean and occlusion rate are weighted and harmonized to obtain the visual confidence, based on the formula: ; in, represents the visual confidence, and represent the mean gradient amplitude and occlusion rate respectively, and denote the gradient weight and occlusion rate weight respectively, and , .

[0053] The mean gradient reflects the clarity of the mask boundary, which is an important indicator in image processing for determining the completeness of the target object's outline. The occlusion rate, on the other hand, is the proportion of the occluded portion calculated based on depth information. In visual perception, a clear boundary generally indicates that the target object is not severely occluded or blurred, so the mean gradient carries a higher weight.

[0054] The occlusion rate is handled by an exponential decay function. The harmonic mean is sensitive to low values and its effect is amplified when the occlusion rate is high. Therefore, a higher This ensures that the dominant role of the gradient mean is more obvious when the occlusion rate is low, and when the occlusion rate increases, the visual confidence is rapidly reduced through exponential decay.

[0055] This embodiment constructs a nonlinear evaluation model of visual confidence by fusing the gradient mean and occlusion rate through weighted harmonic mean. This embodiment uses the characteristic of harmonic mean being sensitive to low values to dynamically balance the dual effects of gradient clarity and occlusion interference. The occlusion rate directly reflects the overall sharpness of the mask boundary, while the occlusion rate is nonlinearly mapped through the exponential decay term - when the occlusion rate increases, the It decreases sharply, significantly amplifying the inhibitory effect of the occlusion rate on the confidence.

[0056] The setting of gradient weight and occlusion rate weight is based on the statistical optimization of farmland scenes. Generally, the contribution of pepper outline clarity to visual pose estimation accounts for about 70%, while occlusion interference accounts for 30%. Through the exponential decay function and weight bias design, accurate quantification under the influence of multiple factors is achieved. Traditional methods mostly use linear weighting, but cannot distinguish the difference between "high occlusion-high clarity" and "low occlusion-low clarity" scenes. The strong attenuation of , more realistically reflects the destructive effect of occlusion on pose estimation.

[0057] This embodiment provides an adaptive arbitration mechanism for multimodal fusion, which is the core guarantee for the robustness of the overall solution. The refined modeling of visual confidence enables the system to significantly improve performance in the following scenarios: When the sudden change in illumination causes the gradient mean to fluctuate, Continuous monitoring maintains a smooth transition of weights to avoid jitter of the robotic arm; when the depth sensor falsely detects occlusion (if real reflections are mistakenly judged as occlusion), The high value of Abnormal increase in the visual weight to prevent excessive suppression.

[0058] In this embodiment, the logic for obtaining the tactile confidence is: The pressure mean is normalized according to the following formula: ; in, represents the mean pressure after normalization, and Respectively represent the maximum and minimum pressure values in the pressure matrix, Indicates the mean pressure; The logic for calculating tactile confidence is: ; in, represents the tactile confidence, 、 and Represent the pressure mean weight, pressure variance weight and friction fluctuation weight respectively, ,and .

[0059] This embodiment constructs tactile confidence through the fusion of multi-dimensional tactile features, achieving a refined assessment of clamping stability. The core logic of tactile confidence is to couple the three key parameters of pressure mean, pressure variance and friction fluctuation coefficient through exponential functions and weight distribution to form a comprehensive scoring mechanism. The normalization of the pressure mean eliminates the dimensional differences in the absolute values of the clamping force in different scenarios, enabling horizontal comparison across tasks, reflecting both moderate clamping strength and avoiding numerical deviations caused by different sensor ranges. The exponential decay term of the pressure variance nonlinearly amplifies the negative impact of the increase in variance on the confidence. When the variance exceeds the threshold, the value of this term rapidly approaches zero, significantly lowering the overall confidence, thereby accurately identifying the hidden risks caused by uneven pressure distribution. The friction fluctuation coefficient quantifies the slip risk through exponential decay after gradient energy normalization. For every 0.1 increase in the fluctuation coefficient, the contribution of this term decreases by about 25%, ensuring the system's sensitivity to progressive slip.

[0060] Through the synergistic effect of weight distribution and nonlinear transformation, the system breaks through the traditional tactile system's reliance on a single parameter. The weighting is set in a hierarchical structure, with the mean pressure weighting greater than the variance weighting greater than the friction fluctuation weighting (ω_C). This distribution is based on the physical characteristics of field harvesting: the mean pressure directly determines whether the gripping force is sufficient, the variance reflects the uniformity of the contact surface, and the friction fluctuation serves as an auxiliary indicator to capture sudden slip.

[0061] This embodiment achieves multi-parameter coupling through geometric averaging. Traditional tactile confidence calculations often use arithmetic averaging or independent thresholds, which fail to capture nonlinear correlations between parameters. This embodiment, due to the exponential decay and weighted power superposition effects, truly reflects the dual risks of "insufficient and unevenly distributed clamping force." The actual application of weight setting and parameter calculation requires calibration and optimization through extensive field experiments.

[0062] Furthermore, when generating the dynamic weights of vision and touch, a normalized exponential function is used to dynamically allocate the weights, based on the following formula: ; in, and Represent visual confidence and tactile confidence respectively, and Represent the dynamic weights of vision and touch respectively.

[0063] This embodiment dynamically assigns visual and tactile weights through a normalized exponential function (Softmax). Essentially, this maps multimodal confidence levels to a probability distribution, enabling adaptive modal-dominant switching. The exponential function nonlinearly amplifies the contribution of high-confidence modalities. When the confidence of one modality is significantly higher than that of another, its weight rapidly approaches 1, forcing the system to rely on that modal decision. This nonlinear response mechanism is more discriminatory than traditional linear weighting—when visual confidence drops from 0.6 to 0.4, the linear weight only drops from 60% to 40%, while the Softmax weight drops sharply from 73% to 43%, accelerating modal switching to respond to sudden environmental changes.

[0064] This embodiment solves the competition and complementarity issues of multimodal data in dynamic environments through the synergistic effect of exponential amplification and probabilistic normalization. Traditional fixed weight or threshold switching methods are prone to weight lag or oscillation in scenarios with sudden changes in lighting, occlusion, or tactile noise. Softmax's dynamic response allows the system to use tactile perception to take over decision-making when vision is obstructed by branches and leaves, avoiding grasping failures caused by visual failure.

[0065] Visual confidence is calculated by processing the RGB image, combining the mean gradient of the mask boundary and the occlusion ratio. The mean gradient reflects the sharpness of the pepper's outline, while the occlusion ratio is the proportion of occlusion calculated from the depth map. A high visual confidence value indicates that the target object in the image has a clear outline and minimal occlusion, indicating that the visual data is reliable. Conversely, a low mean gradient or a high occlusion ratio results in a low visual confidence value, indicating that the visual information may be unreliable.

[0066] Tactile confidence is based on tactile sensor data, including pressure mean, pressure variance, and friction fluctuation coefficient. The pressure mean indicates the overall strength of the clamping force, the pressure variance reflects the uniformity of the pressure distribution, and the friction fluctuation coefficient captures the dynamic changes in the contact surface. A high tactile confidence indicates stable clamping, uniform pressure distribution, and minimal friction variation. Uneven pressure or large friction fluctuations will reduce tactile confidence, indicating possible slippage or unstable clamping.

[0067] Dynamic weights are dynamically assigned based on these two confidence levels using a softmax function. These weights reflect which sensor data is more reliable in the current environment. For example, when visual confidence is high, vision is given a higher weight, relying more on visual data for pose estimation; when tactile confidence is high, tactile data is given a higher weight, relying more on tactile data. This dynamic adjustment ensures flexible switching of the dominant sensor in complex environments, improving robustness.

[0068] Step 3: Detect the key points of the chili pepper in the RGB image, and map the key points into the three-dimensional coordinates of the key feature points in combination with the depth map. Calculate the visual six-degree-of-freedom pose of the chili pepper relative to the RGB image acquisition point, and use the tactile data as the feature input of the pre-trained BP neural network to obtain the tactile six-degree-of-freedom pose of the chili pepper relative to the picking position. The six-degree-of-freedom pose includes a rotation matrix and a translation vector.

[0069] In this embodiment, the logic for calculating the six-degree-of-freedom position of the chili pepper relative to the visual position at the RGB image acquisition location is as follows: Define at least four key feature points of the chili pepper, including the tip of the pepper, the connection point of the pepper stalk, the widest points on the left and right sides of the pepper body, the center of the pepper body, and at least four of the end of the pepper body. The key feature point is essentially a pixel point in the RGB image. Obtain the existing RGB image of the chili pepper, annotate the key feature points in each image to form a feature training set, and construct a key feature point prediction model based on the HRNet network. Use the RGB image in the feature training set as the feature input, output the corresponding key feature points, and train the key feature point prediction model.

[0070] The collected RGB image is input into the trained key feature point prediction model to obtain the predicted coordinates of the key feature points in the collected RGB image.

[0071] This embodiment uses the HRNet network to achieve high-precision detection of key points of chili peppers, and combines depth map mapping with tactile data fusion to construct a multimodal collaborative six-degree-of-freedom pose estimation framework. First, key feature points such as the pepper tip and the pepper stalk connection point are defined, and a training set is constructed through manual annotation. HRNet uses its multi-resolution parallel subnetwork structure to retain high, medium, and low-resolution features simultaneously during the training process, significantly improving the detection robustness in small targets and occlusion scenarios. For example, when the pepper body is 30% obscured by branches and leaves, the key point positioning error of the traditional CNN model reaches 8 pixels, while HRNet can control the error to within 3 pixels through cross-resolution feature exchange. The detected two-dimensional key point coordinates are strictly aligned with the depth map, and the camera intrinsic parameter matrix is used to convert the pixel coordinates into three-dimensional camera coordinate system coordinates, providing a spatial reference for subsequent pose solution.

[0072] Compared to traditional monocular vision methods, the innovation of this step lies in the deep integration of tactile data-driven pose correction. Tactile data is used to directly predict the six-degree-of-freedom pose of the gripper relative to the pepper body through a pre-trained BP neural network (with input as the flattened vector of a 100-dimensional pressure matrix). The physical significance of this is that when visual key point detection drifts due to occlusion or light interference, the tactile network uses features such as pressure mean, variance, and friction gradient to capture abnormal gripping force distribution (such as unilateral overpressure) or slip trends (such as friction fluctuations), generating compensatory pose parameters. The visual pose and tactile pose are fused in a unified coordinate system through dynamic weighting, with the weights jointly driven by visual confidence and tactile confidence.

[0073] For the predicted positions of the key feature points, the depth values of the corresponding positions are directly read from the depth map to obtain the three-dimensional predicted coordinates of each key feature point in the chili pepper coordinate system. The intrinsic parameter matrix of the acquisition camera is used to convert the three-dimensional predicted coordinates of each key feature point into the three-dimensional camera coordinate system coordinates, and the center of mass of each key feature point in the chili pepper coordinate system and the center of mass of each key feature point in the three-dimensional camera coordinate system are determined respectively.

[0074] Obtain the decentralized points of each key feature point in the chili pepper coordinate system and the decentralized points of each key feature point in the chili pepper coordinate system, construct a covariance matrix, perform singular value decomposition on the covariance matrix, and calculate the visual rotation matrix and translation vector.

[0075] This embodiment achieves accurate calculation of the visual six-degree-of-freedom pose through three-dimensional coordinate transformation of key points and covariance matrix decomposition. First, the two-dimensional key points detected in the RGB image (such as the tip of the pepper and the stalk of the pepper) are combined with the depth values of the corresponding positions in the depth map, and the pixel coordinate system is converted to the three-dimensional camera coordinate system through the camera intrinsic parameter matrix. This step maps the discrete two-dimensional image features into continuous three-dimensional points in the physical space, providing a geometric basis for subsequent pose estimation. Next, the center of mass of the key points in the local coordinate system of the chili pepper (with the center of mass of the pepper body as the origin) and the camera coordinate system are calculated respectively. By subtracting the center of mass coordinates, a decentralized point set is obtained, which eliminates the interference of the translation component and allows subsequent analysis to focus on rotational alignment.

[0076] In terms of technical effects, the construction of the covariance matrix and the singular value decomposition achieve optimal rotation estimation through mathematical optimization, quantify the spatial correlation of the key points distributed in the two coordinate systems, and its SVD decomposition\extracts the principal component directions. The rotation matrix obtains the optimal rotation in the least squares sense by multiplying the orthogonal matrix, and the translation vector is directly calculated by the center of mass difference.

[0077] Traditional methods rely on feature matching or iterative optimization and are susceptible to depth map noise or keypoint positioning errors. The covariance matrix, on the other hand, integrates all keypoint information through statistical distribution, diluting local errors with the global distribution. For example, when individual keypoints have abnormal depth values due to occlusion, decentralization and matrix summation operations can effectively suppress the influence of outliers. At the same time, the closed-form solution of SVD avoids the local optimal trap of iterative algorithms, ensuring the uniqueness and stability of pose estimation. For the overall solution, accurate visual pose provides a high-precision spatial reference for multimodal fusion. When tactile data experiences pose offsets due to uneven distribution of gripping force, visual pose can be used to perform direction corrections through weighted fusion.

[0078] Furthermore, the three-dimensional predicted coordinates of the key feature points in the chili pepper coordinate system are calibrated as , and Respectively The row and column coordinates of the key feature points in the RGB image, For the The depth value of the key feature point in the depth map, the intrinsic parameter matrix of the camera Expressed as: ; in, and Respectively represent the horizontal and vertical focal lengths of the acquisition camera, 、 Indicates the coordinates of the principal point of the acquisition camera.

[0079] The converted three-dimensional camera coordinate system coordinates are , the conversion formula is: ; When determining the center of mass of each key feature point in the chili pepper coordinate system and the center of mass in the 3D camera coordinate system, the center of mass in the chili pepper coordinate system is expressed as: ; in, Indicates the centroid coordinates of the key feature points in the Chaotianjiao coordinate system, Represents the number of selected key feature points, and the center of mass in the 3D camera coordinate system is: ; in, Indicates the centroid coordinates of the key feature points in the chili pepper coordinate system, with the first Taking a key feature point as an example, the decentralized point of the key feature point in the chili pepper coordinate system is expressed as: , the decentralized point in the 3D camera coordinate system is expressed as: .

[0080] Before constructing the covariance matrix, the decentralized coordinates will be organized into a matrix form for calculation, that is, the decentralized points of each key feature point in the two coordinate systems are first constructed into a matrix and then the covariance is constructed. For example, the decentralized point representation matrix in the chili pepper coordinate system is for: ; Correspondingly, the decentralized point representation matrix in the 3D camera coordinate system is for: ; Constructed covariance matrix for: ; The formula for singular value decomposition of the covariance matrix is: ; in, and is an orthogonal matrix, corresponding to the rotation direction of the object coordinate system and the camera coordinate system, is a singular value matrix representing the scaling factor.

[0081] Visual rotation matrix The calculation method is: ; Visual translation vector The calculation formula is: ; In this embodiment, the logic for obtaining the six-degree-of-freedom position of the chili pepper relative to the picking position is: A 10×10 pressure matrix between the chili pepper and the picking site is collected in advance, and the actual six-degree-of-freedom pose data of the chili pepper relative to the picking site is simultaneously recorded and used as a label. Expand the pre-collected 10×10 pressure matrix to form a 100-dimensional vector, build a BP network model, input the 100-dimensional vector into the BP network model to be trained, and use the corresponding six-degree-of-freedom posture data as a label to train the BP network model to be trained; The collected tactile data is expanded and a 100-dimensional vector is constructed and input into the trained BP network model to obtain the six-degree-of-freedom position of the chili pepper relative to the picking part, that is, the tactile rotation matrix and translation vector of the chili pepper relative to the picking part.

[0082] The six-degree-of-freedom pose calculated using the RGB image and depth map is relative to the camera coordinate system, that is, the optical center position of the acquisition camera during image acquisition. The six-degree-of-freedom pose of the chili pepper relative to the picking part is relative to the tactile data coordinate system, that is, relative to the part of the chili pepper that is in contact with the picking part when the chili pepper is collected.

[0083] This embodiment uses tactile data to drive a BP neural network to predict six-degree-of-freedom pose, establishing a vision-independent physical interaction perception channel. The tactile pose acquisition logic is based on the strong correlation between the tactile sensor (10×10 pressure matrix) and the gripping state. The spatial pattern of pressure distribution (such as pressure gradient and symmetry) implicitly encodes the geometric relationship between the gripper and the contact surface of the chili pepper (such as tilt angle and contact depth). By pre-collecting a large amount of gripping experimental data (e.g., 300 gripping motions), the pressure matrix is flattened into a 100-dimensional vector as input, and the six-degree-of-freedom pose (rotation matrix and translation vector) is used as a label. The BP neural network is trained to learn the nonlinear mapping from tactile features to pose. For example, when the gripper tilts to the right, the pressure value in the right region of the pressure matrix increases significantly. The neural network extracts these spatial features through fully connected layers. This design enables tactile pose estimation to rely entirely on mechanical feedback, without visual assistance, and can still function independently even when vision is completely lost (such as full occlusion).

[0084] The introduction of tactile pose sensing overcomes the scenario limitations of traditional vision-dependent solutions. Traditional methods experience a sharp increase in pose estimation error when visually obstructed or illuminated. Tactile pose, however, directly reflects the gripping state through the physical properties of pressure distribution. For example, when a pepper is completely obscured by leaves and branches, the tactile network identifies the gripper's leftward tilt based on the leftward deviation of the mean pressure, automatically returning the gripper to its original position and avoiding a collision.

[0085] This embodiment upgrades tactile data from simple threshold alarms to high-dimensional pose estimation. Traditional tactile systems usually only set pressure extreme alarms and cannot provide spatial pose information. This embodiment uses a neural network to learn the relationship between complex pressure patterns and poses to achieve precise positioning. The heterogeneous fusion of tactile pose and visual pose significantly improves the system's fault tolerance and scene adaptability. In terms of coordinate system alignment, the tactile pose is based on the coordinate system of the gripper body (directly related to the motion control of the robotic arm), and the visual pose is based on the camera coordinate system. After the two are unified to the global coordinate system through the hand-eye calibration matrix, they can be interpolated and fused based on dynamic weights.

[0086] Step 4: Convert the visual and tactile rotation matrices into quaternions, interpolate and fuse them according to the dynamic weights of vision and touch to generate the final rotation data, and linearly weight the visual and tactile translation vectors according to the dynamic weights of vision and touch to obtain the final translation data.

[0087] In this embodiment, the rotation matrix is a 3×3 orthogonal matrix that describes the rotation transformation around a certain coordinate axis or an arbitrary axis in a three-dimensional space. The rotation matrix is expressed as: ; in, Represents the rotation matrix, the internal elements express Middle Rank Elements of the column; The conversion formula for converting a rotation matrix to a quaternion is: ; in, Represents the rotation matrix The quaternion representation of and 、 、 Represent the real part and three imaginary parts of the quaternion respectively; When the final rotation data is generated by the dynamic weighted interpolation fusion of vision and touch, the quaternion converted from the rotation matrix of vision and touch is subtracted to generate the fused quaternion. The formula is: ; in, Represents the fused quaternion, and Represent the dynamic weights of vision and touch respectively, and They are the quaternions converted from the rotation matrices of vision and touch respectively. is the angle between two quaternions; The fused quaternion is converted into a rotation matrix according to the conversion formula of rotation matrix and quaternion, and calibrated as the final rotation data.

[0088] This embodiment realizes the adaptive fusion of visual and tactile rotation data by combining spherical linear interpolation with dynamic weights, which solves the problems of rotation smoothness and weight sensitivity in multimodal pose estimation. The quaternion interpolation weight is dynamically allocated by visual confidence and tactile confidence, and the interpolation angle is the angle between the two quaternions. and The coefficient distribution ensures the shortest path interpolation on the four-dimensional hypersphere.

[0089] This embodiment significantly improves the robustness of rotational fusion through the nonlinear coupling of geometric interpolation paths and dynamic weights. Traditional linear interpolation in quaternion space can lead to uneven angular velocity, while spherical linear interpolation maintains a constant angular velocity, avoiding robotic arm jitter during the fusion process. This embodiment introduces dynamic weights into classic spherical linear interpolation, breaking through the environmental adaptability limitations of traditional fixed-weight interpolation. Traditional methods typically use fixed ratios or threshold-based switching, which cannot cope with dynamic changes in sensor confidence.

[0090] Quaternion interpolation fusion provides a mathematically rigorous rotational representation and fusion interface for multimodal pose estimation. When there are spatial inconsistencies between visual and tactile poses, spherical linear interpolation unifies intersecting rotations into a composite rotation through quaternion multiplication, avoiding singularities caused by the order dependency of Euler angles.

[0091] In this embodiment, the visual and tactile translation vectors are linearly weighted according to the dynamic weights of vision and touch, and the formula for obtaining the final translation data is: ; in, and Represents the transpose of the visual and tactile translation vectors, and Represent the visual and tactile translation vectors respectively, 、 and are the internal elements of the visual translation vector, 、 and represent the internal elements of the tactile translation vector, and , Represents the final translation data, and Represent the dynamic weights of vision and touch respectively.

[0092] This embodiment constructs a lightweight and efficient multimodal data fusion mechanism by linearly weighting the fusion of visual and tactile translation vectors with dynamic weights. Its core logic is to regard the visual and tactile translation vectors as complementary observations in the same physical space, and achieve adaptive fusion through confidence-driven weight distribution. For example, when the visual pose estimation error increases due to occlusion of branches and leaves, and the tactile data maintains a high confidence level due to the stable pressure distribution, the tactile translation vector dominates the final output, effectively suppressing the impact of visual noise on the pose, and ensuring that the fusion process is extremely computationally intensive through linear superposition rather than complex transformations.

[0093] Linear weighting significantly improves system response speed while maintaining accuracy. Traditional methods use Kalman filtering or multi-sensor calibration matrix fusion, which requires iterative calculations or matrix inversion. This embodiment achieves lossless coupling of multimodal data through dynamic weight allocation. Traditional methods often use fixed weights or hard switching strategies, which can easily cause fusion jumps when sensor confidence levels suddenly change.

[0094] Dynamically weighted fusion of translation vectors ensures both spatial consistency and operational safety for multimodal pose estimation. When visual and tactile poses conflict, weight allocation is automatically arbitrated based on confidence—the high-confidence modality dominates the correction direction, preventing oscillation of the robotic arm due to sensor conflict.

[0095] Step 5: Obtain the final six-degree-of-freedom pose data of the chili pepper based on the final rotation data and the final translation data.

[0096] Six degrees of freedom (6DOF) poses include three rotational degrees of freedom (3DOF) and three translational degrees of freedom (3DOF). In three-dimensional space, these can be represented using a rotation matrix and a translation vector. This example combines these three rotational degrees of freedom (3DOF) and three translational degrees of freedom (3DOF) into a complete pose transformation matrix. The rotation matrix and translation vector together form a 4x4 homogeneous transformation matrix, a standard method for representing poses in robotics. This matrix can be used for motion planning of the robotic arm, enabling it to accurately grasp objects.

[0097] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0098] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software depends on the specific application and design constraints of the technical solution.

[0099] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, and may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment as needed.

[0100] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for estimating the picking posture of chili peppers based on joint visual and tactile perception, characterized in that: The specific steps include: The researchers collected RGB images, depth maps, and tactile data between the picking site and the chili pepper. Based on the RGB images, they generated a mask of the chili pepper's main body and its mask boundary. The occlusion rate was calculated using the depth map, and the mean, variance, and friction fluctuation coefficient of the contact pressure between the picking site and the chili pepper were obtained based on the tactile data. The mean gradient amplitude of the pixel points at the mask boundary is calculated and weighted harmonic average is performed in combination with the occlusion rate to obtain the visual confidence. The pressure mean, variance, and friction fluctuation coefficient are integrated to obtain the tactile confidence. The visual confidence and tactile confidence are processed as a weighted sum to generate dynamic visual and tactile weights. Key points of the chili pepper are detected in the RGB image, and the key points are combined with the depth map to map the three-dimensional coordinates of the key feature points. The visual six-degree-of-freedom pose of the chili pepper relative to the RGB image acquisition point is calculated. The tactile data is used as the feature input of the pre-trained BP neural network to obtain the tactile six-degree-of-freedom pose of the chili pepper relative to the picking position. The six-degree-of-freedom pose includes a rotation matrix and a translation vector. Convert the visual and tactile rotation matrices into quaternions, interpolate and fuse them according to the dynamic weights of vision and touch to generate the final rotation data, and linearly weight the visual and tactile translation vectors according to the dynamic weights of vision and touch to obtain the final translation data; The final six-degree-of-freedom pose data of the chili pepper is obtained based on the final rotation data and the final translation data.

2. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 1, characterized in that: The RGB image and depth map of the chili pepper have the same resolution, and each pixel in the RGB image is mapped one-to-one to each pixel in the depth map. That is, each pixel in the RGB image has a corresponding depth value. The steps for generating the mask and mask boundary of the chili pepper body include: Obtain existing RGB images of chili peppers, perform pixel-level annotation on the chili pepper body in each image, and mark the binary mask of the pepper body outline; Build an instance segmentation model using Mask R-CNN as the baseline model. Use the existing RGB image of chili peppers as the feature input and the corresponding binary mask of the annotated pepper outline as the instance segmentation model label. Train the instance segmentation model. Use the pre-trained instance segmentation model to segment the pepper body of the collected RGB image, and obtain a mask of the pepper body with the pixel value of the pepper body area being 1 and the pixel value of the background part (non-pepper body part) being 0; Use edge detection algorithm to extract the boundary from the mask of the main body of the chili pepper to obtain the mask boundary of the main body of the chili pepper; When calculating the occlusion rate, the bottommost pixel row in the RGB image of the chili pepper is the first row, and the leftmost pixel column is the first column. The pixel positions in the mask of the chili pepper body and the mask boundary are mapped one by one using coordinates, so that each pixel has a unique coordinate value. Get the depth values of all pixels in the mask area in the depth map, calculate the mean depth of all pixels in the mask area, and set the depth difference threshold. The formula based on the depth difference threshold is: ； in, and Represent the depth mean and standard deviation of all pixels in the mask area, Represents the total number of pixels in the mask area. Indicates the first The depth value of each pixel, and Respectively represent the The horizontal and vertical coordinates of the pixel points, represents the depth threshold; like , then the horizontal and vertical coordinates in the marked mask area are and The pixels in the mask area are occluded pixels. The number of all occluded pixels is counted, and the ratio of the number of occluded pixels to the total number of pixels in the mask area is the occlusion rate.

3. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 2, wherein: The logic for obtaining visual confidence is: Convert the RGB image into a grayscale image, use the Sobel operator to calculate the directional gradient amplitude in the row and column directions for each pixel point on the mask boundary, calculate the gradient amplitude of each mask boundary pixel point and normalize it, and calculate the normalized gradient amplitude mean of all pixels on the mask boundary; The visual confidence is obtained by taking the weighted harmonic average of the normalized gradient amplitude mean and the occlusion rate. The formula is: ； in, represents the visual confidence, and represent the mean gradient amplitude and occlusion rate respectively, and denote the gradient weight and occlusion rate weight respectively, and , .

4. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 3, wherein: The logic for calculating the six-degree-of-freedom pose of the chili pepper relative to the visual position of the RGB image acquisition point is: Define at least four key feature points of chili peppers, obtain existing RGB images of chili peppers, annotate the key feature points in each image to form a feature training set, and build a key feature point prediction model based on the HRNet network. Use the RGB images in the feature training set as feature inputs and output the corresponding key feature points to train the key feature point prediction model. Input the collected RGB image into the trained key feature point prediction model to obtain the predicted coordinates of the key feature points in the collected RGB image; For the predicted positions of key feature points, the depth values of the corresponding positions are directly read from the depth map to obtain the 3D predicted coordinates of each key feature point in the chili pepper coordinate system. The intrinsic parameter matrix of the acquisition camera is used to convert the 3D predicted coordinates of each key feature point into the 3D camera coordinate system coordinates, and the center of mass of each key feature point in the chili pepper coordinate system and the center of mass of each key feature point in the 3D camera coordinate system are determined respectively. Obtain the decentralized points of each key feature point in the chili pepper coordinate system and the decentralized points of each key feature point in the chili pepper coordinate system, construct a covariance matrix, perform singular value decomposition on the covariance matrix, and calculate the visual rotation matrix and translation vector.

5. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 1, characterized in that: The tactile data is a 10×10 pressure matrix. The method for obtaining the pressure mean, variance, and friction fluctuation coefficient of the contact between the picking part and the chili pepper is as follows: There are 100 pressure data in the 10×10 pressure matrix. The sum of the 100 pressure data is calculated and the average is the mean pressure of the picking part in contact with the chili pepper. The variance is calculated based on the pressure mean. The formula is: ； in, represents the variance of the mean pressure, Indicates the first Rank The pressure value of the column, Indicates the mean pressure; The Coulomb friction model is constructed using the pressure matrix and the friction coefficient of the picking part in contact with the chili pepper to obtain a 10×10 friction matrix. The Sobel operator is used to calculate the spatial gradient of each element in the friction matrix in both the row and column directions. Row, No. The spatial gradients of the elements in the row and column directions are calibrated as and , calculate the friction fluctuation coefficient based on the formula: ； in, and represent the mean value of friction and the friction fluctuation coefficient respectively, represents the friction matrix Row, No. The elements of the column represent the friction force.

6. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 5, characterized in that: The logic for obtaining tactile confidence is: The pressure mean is normalized according to the following formula: ； in, represents the mean pressure after normalization, and Respectively represent the maximum and minimum pressure values in the pressure matrix, Indicates the mean pressure; The logic for calculating tactile confidence is: ； in, represents the tactile confidence, 、 and Represent the pressure mean weight, pressure variance weight and friction fluctuation weight respectively, ,and .

7. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 6, characterized in that: The logic for obtaining the six-degree-of-freedom pose of the chili pepper relative to the picking position is: A 10×10 pressure matrix between the chili pepper and the picking site is collected in advance, and the actual six-degree-of-freedom pose data of the chili pepper relative to the picking site is simultaneously recorded and used as a label. Expand the pre-collected 10×10 pressure matrix to form a 100-dimensional vector, build a BP network model, input the 100-dimensional vector into the BP network model to be trained, and use the corresponding six-degree-of-freedom posture data as a label to train the BP network model to be trained; The collected tactile data is expanded and a 100-dimensional vector is constructed and input into the trained BP network model to obtain the six-degree-of-freedom position of the chili pepper relative to the picking part, that is, the tactile rotation matrix and translation vector of the chili pepper relative to the picking part.

8. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 1, characterized in that: When generating dynamic weights for vision and touch, a normalized exponential function is used to dynamically allocate weights, based on the following formula: ； in, and Represent visual confidence and tactile confidence respectively, and Represent the dynamic weights of vision and touch respectively.

9. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 1, characterized in that: The rotation matrix is a 3×3 orthogonal matrix that describes the rotation transformation around a certain coordinate axis or an arbitrary axis in three-dimensional space. The rotation matrix is expressed as: ； in, Represents the rotation matrix, the internal elements express Middle Rank Elements of the column; The conversion formula for converting a rotation matrix to a quaternion is: ； in, Represents the rotation matrix The quaternion representation of and 、 、 Represent the real part and three imaginary parts of the quaternion respectively; When the final rotation data is generated by the dynamic weighted interpolation fusion of vision and touch, the quaternion converted from the rotation matrix of vision and touch is subtracted to generate the fused quaternion. The formula is: ； in, Represents the fused quaternion, and Represent the dynamic weights of vision and touch respectively, and They are the quaternions converted from the rotation matrices of vision and touch respectively. is the angle between two quaternions; The fused quaternion is converted into a rotation matrix according to the conversion formula of rotation matrix and quaternion, and calibrated as the final rotation data.

10. The method for estimating the picking posture of chili peppers based on visual and tactile combined perception according to claim 1, characterized in that: The visual and tactile translation vectors are linearly weighted according to the dynamic weights of vision and touch, and the formula for obtaining the final translation data is: ； in, and Represents the transpose of the visual and tactile translation vectors, and Represent the visual and tactile translation vectors respectively, Represents the final translation data, and Represent the dynamic weights of vision and touch respectively.

Images