Robot tactile and visual fusion double ring coupling adaptive grasping control method

Abstract

The application provides a robot tactile and visual fusion double-ring coupling adaptive grasping control method, relates to the technical field of robot grasping control, and comprises the following steps: acquiring surface characteristic data through a light field camera array, collecting dynamic contact information and extracting stability features by a sensor array, constructing cross-modal coupling and fusing tactile and visual information based on Tucker decomposition, establishing mapping parameters and constructing an adaptive time window to optimize grasping parameters, and adjusting grasping posture and clamping force in combination with Lyapunov stability criterion and a dynamic compensation mechanism. The method can improve the stability and robustness of robot grasping in a complex environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot grasping control, and in particular to a robot tactile and visual fusion double-ring coupling adaptive grasping control method. BACKGROUND

[0002] Robot grasping technology is an important research direction in the fields of intelligent manufacturing, service robots and human-computer interaction, and its performance directly affects the operation ability of robots in unstructured environments. Traditional robot grasping mainly relies on single visual or tactile information for control, which is difficult to cope with uncertain factors in complex environments. In recent years, with the expansion of intelligent manufacturing and service robot application scenarios, higher requirements have been put forward for the adaptability and robustness of robot grasping, prompting researchers to explore multi-modal perception fusion robot grasping methods.

[0003] Existing technologies mainly use simple feature-level fusion or decision-level fusion methods, lack effective cross-modal information coupling mechanisms, and the complementarity of visual and tactile information cannot be fully utilized, especially in dynamic changing environments, making it difficult to achieve efficient integration and consistent expression of information. The traditional grasping control method lacks adaptability to environmental changes and object property changes, lacks adaptive adjustment mechanism, and cannot dynamically optimize grasping parameters according to real-time perception information, resulting in a significant decrease in grasping success rate when facing unknown objects or interference factors. Existing technologies have deficiencies in handling the timing characteristics of tactile and visual information, and the distinction and utilization of transient and steady-state responses are not sufficient, which cannot effectively cope with dynamic changes and environmental disturbances during grasping, limiting the application ability of robots in complex tasks. SUMMARY

[0004] The robot tactile and visual fusion double-ring coupling adaptive grasping control method provided by the embodiments of the present application can solve the problems in the prior art.

[0005] In a first aspect of the embodiments of the present application, a robot tactile and visual fusion double-ring coupling adaptive grasping control method is provided, comprising:

[0006] During the movement of the robot, adjacent view point images are collected by a light field camera array, a depth map and a normal map of the surface of the grasping target are calculated according to the adjacent view point images, and surface property data is obtained;

[0007] When the robot reaches the preset grasping position, a sensor array arranged on the grasping surface acquires dynamic contact information of the grasping target through piezoelectric and piezoresistive dual-mode responses, and stable features are extracted from the dynamic contact information based on wavelet transform;

[0008] The cross-modal coupling is constructed based on Tucker decomposition, the transient response channel and the steady-state response channel are constructed based on the surface characteristic data and the stability feature, the transient response channel and the steady-state response channel are synchronized through a recursive optimization algorithm combined with a time sequence consistency constraint and are adaptively fused based on an attention mechanism, and the double-loop coupling of the haptics and the vision is realized.

[0009] The mapping parameters between the double-loop coupling output and the grasping parameters are established, the adaptive sliding time window is constructed based on the change rate of the grasping parameters obtained through the mapping, the time-varying characteristics of the grasping parameters are calculated in each time window, the grasping parameters are optimized using the time-varying characteristics combined with the Lyapunov stability criterion, the grasping posture and the clamping force of the robot are adjusted according to the optimized grasping parameters, and the influence of environmental disturbances and modeling errors is eliminated through a dynamic compensation mechanism, so that adaptive grasping control is realized.

[0010] The adjacent view images are collected by the light field camera array, the depth map and the normal map of the surface of the grasping target are calculated according to the adjacent view images, and the surface characteristic data is obtained.

[0011] The epipolar pairs of the adjacent view images collected by the light field camera array are extracted, the gray difference values of the corresponding pixel points on the epipolar pairs are calculated, the disparity value corresponding to the smallest gray difference value is selected, and the disparity map is obtained.

[0012] The surface depth value is calculated using the disparity map, the first-order difference and the second-order difference are calculated in the pixel coordinate system according to the surface depth value, the local normal vector is calculated according to the depth difference of adjacent pixel points, the direction projection of the local normal vector in the pixel coordinate system is taken as the surface orientation parameter, the deviation of the first-order difference from the preset smoothness threshold is taken as the depth smoothness constraint, and the change rate of the second-order difference and the surface orientation parameter is taken as the normal consistency constraint, the surface depth value and the local normal vector are iteratively adjusted to minimize the deviation of the depth smoothness constraint and the normal consistency constraint, and the depth map and the normal map are obtained.

[0013] The depth map and the normal map are taken as the surface characteristic data of the grasping target.

[0014] The cross-modal coupling is constructed based on Tucker decomposition, the transient response channel and the steady-state response channel are constructed based on the surface characteristic data and the stability feature, the transient response channel and the steady-state response channel are synchronized through a recursive optimization algorithm combined with a time sequence consistency constraint and are adaptively fused based on an attention mechanism, and the double-loop coupling of the haptics and the vision is realized.

[0015] decoupling the surface property data in a spatial dimension to obtain a deformation component and a force component, and decoupling the stability feature in a time dimension to obtain a transient response component and a steady-state response component; aligning and combining the deformation component and the transient response component in the spatial dimension and the time dimension respectively to obtain a transient feature tensor; aligning and combining the force component and the steady-state response component in the spatial dimension and the time dimension respectively to obtain a steady-state feature tensor; obtaining a transient core tensor and a steady-state core tensor through Tucker decomposition of the transient feature tensor and the steady-state feature tensor respectively;

[0016] constructing a coupling metric matrix based on tensor outer product operation of the transient core tensor and the steady-state core tensor, taking a product operation result of the transient feature tensor and the coupling metric matrix as a transient response channel, and taking a product operation result of the steady-state feature tensor and a transpose of the coupling metric matrix as a steady-state response channel.

[0017] establishing a mapping parameter between the double-loop coupled output and a grasping parameter, constructing an adaptive sliding time window based on a variation rate of the grasping parameter obtained through the mapping, and calculating a time-varying characteristic of the grasping parameter in each time window, including:

[0018] calculating a difference value of the grasping parameter at adjacent sampling time points, dividing the difference value by a sampling time interval to obtain a variation rate of the grasping parameter, constructing an adaptive sliding time window based on the variation rate, and obtaining a length of the adaptive sliding time window through negative exponential operation of a preset basic window length and the variation rate;

[0019] establishing a state transition matrix of the grasping parameter in the adaptive sliding time window, calculating a state estimation gain of the grasping parameter based on a Kalman filter, iteratively updating the state transition matrix using the state estimation gain, calculating eigenvalues of the state transition matrix, and statistically calculating a norm difference value of the state transition matrix in two adjacent iterations to obtain the time-varying characteristic of the grasping parameter according to a distribution of the eigenvalues and a variation trend of the norm difference value.

[0020] optimizing the grasping parameter using the time-varying characteristic in combination with Lyapunov stability criterion, adjusting a grasping posture and a clamping force of the robot according to the optimized grasping parameter, and eliminating influences of environmental disturbance and modeling error through a dynamic compensation mechanism, including:

[0021] composing a state vector by using state transition eigenvalues and a norm difference value in the time-varying characteristic, constructing a positive definite matrix based on the state transition eigenvalues, multiplying the state vector by the positive definite matrix, adding an integral term of the state vector, and calculating a time derivative, and taking a product of the time derivative and a preset convergence coefficient being less than zero as a stability constraint condition;

[0022] Under the premise of meeting the stability constraint condition, a weighted square error between the grasping parameter and a preset expected parameter value is calculated, a gradient of the weighted square error with respect to the grasping parameter is calculated, and the grasping parameter is obtained by subtracting the product of the gradient and a preset learning rate from the grasping parameter, to obtain an optimized grasping parameter;

[0023] A basic control quantity is generated through the optimized grasping parameter, and the grasping posture and the clamping force of the robot are adjusted using the basic control quantity;

[0024] The environmental disturbance is decomposed into multi-dimensional disturbance features through wavelet transform, and a disturbance compensation control quantity is obtained based on the multi-dimensional disturbance features by using fuzzy rule reasoning, a modeling error signal is calculated, and an error compensation control quantity is generated by combining a robust adaptive control strategy; and the disturbance compensation control quantity and the error compensation control quantity are superimposed on the basic control quantity.

[0025] The environmental disturbance is decomposed into multi-dimensional disturbance features through wavelet transform, and a disturbance compensation control quantity is obtained based on the multi-dimensional disturbance features by using fuzzy rule reasoning, a modeling error signal is calculated, and an error compensation control quantity is generated by combining a robust adaptive control strategy, including:

[0026] The environmental disturbance is decomposed into multi-scale coefficients through wavelet transform, and a multi-dimensional disturbance feature vector is obtained by calculating the energy distribution of the multi-scale coefficients in different frequency bands; a membership value of the multi-dimensional disturbance feature vector is calculated, the membership value is matched with a preset fuzzy rule set to obtain a rule activation strength, and the rule activation strength is weighted and averaged to obtain a disturbance compensation control quantity;

[0027] A modeling error signal is calculated by calculating the deviation of the state transition matrix and the actual system response, a two-norm of the modeling error signal is calculated, and a projection boundary value is set according to the two-norm, an adaptive dynamic parameter of the modeling error signal is calculated, the adaptive dynamic parameter is projected into the projection boundary value and integrated to obtain an adaptive gain matrix, and the modeling error signal is multiplied by the adaptive gain matrix to obtain an error compensation control quantity.

[0028] In a second aspect of the embodiment of the application, a robot tactile and visual fusion double-loop coupled adaptive grasping control system is provided, comprising:

[0029] A first unit is configured to acquire adjacent view point images by using a light field camera array during movement of the robot, calculate a depth map and a normal map of a grasping target surface according to the adjacent view point images, and obtain surface characteristic data.

[0030] The second unit is configured to, when the robot reaches a preset grasping position, acquire dynamic contact information of a grasping target by a piezoelectric and piezoresistive dual-mode response of a sensing array arranged on a grasping surface, and extract stability features from the dynamic contact information based on wavelet transform;

[0031] The third unit is configured to construct a cross-modal coupling based on Tucker decomposition, construct a transient response channel and a steady-state response channel based on the surface feature data and the stability features, combine a time sequence consistency constraint, synchronize the transient response channel and the steady-state response channel by a recursive optimization algorithm, and realize dual-loop coupling of tactile and visual based on an attention mechanism adaptive fusion.

[0032] The fourth unit is configured to establish a mapping parameter between a dual-loop coupling output and a grasping parameter, construct an adaptive sliding time window based on a change rate of the grasping parameter obtained by mapping, calculate a time-varying characteristic of the grasping parameter in each time window, use the time-varying characteristic to combine Lyapunov stability criteria to optimize the grasping parameter, adjust a grasping posture and a clamping force of the robot according to the optimized grasping parameter, and eliminate the influence of environmental disturbance and modeling error through a dynamic compensation mechanism to realize adaptive grasping control.

[0033] The third aspect of the embodiment of the application,

[0034] An electronic device is provided, comprising:

[0035] A processor;

[0036] A memory for storing processor-executable instructions;

[0037] The processor is configured to invoke the instructions stored in the memory to execute the method described above.

[0038] The fourth aspect of the embodiment of the application,

[0039] A computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the method described above.

[0040] The beneficial effects of the present application are as follows:

[0041] The present application acquires target surface features and dynamic contact information through a light field camera array and a piezoelectric and piezoresistive dual-mode sensing system, constructs a dual-loop coupling fusion mechanism of tactile and visual, realizes the coordinated cooperation of multiple sensing channels in the grasping process, and improves the sensing accuracy and grasping adaptability of the robot to complex objects.

[0042] The application establishes transient and steady response channels based on a Tucker decomposition cross-modal coupling and recursive optimization algorithm, and effectively solves the difference problem of visual perception and tactile feedback in the time and space scales through adaptive fusion of an attention mechanism, thereby enhancing the grasping robustness of the robot in an uncertain environment.

[0043] The application adopts an adaptive sliding time window and a Lyapunov stability optimization strategy, and combines a dynamic compensation mechanism, so that the grasping posture and clamping force can be adjusted in real time, the influence of environmental disturbance and modeling error is effectively inhibited, and the grasping success rate and operation precision of the robot in a dynamically changing environment are improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A flowchart of a robot tactile and visual fusion double-loop coupled adaptive grasping control method according to an embodiment of the application is shown in

[0045] Figure 2 A flowchart of a Tucker decomposition cross-modal coupling construction is shown in DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described below in detail with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0047] The technical solutions of the application will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described in some embodiments.

[0048] Figure 1 A flowchart of a robot tactile and visual fusion double-loop coupled adaptive grasping control method according to an embodiment of the application is shown in Figure 1 The method comprises the following steps:

[0049] During movement of the robot, adjacent viewpoint images are collected by a light field camera array, a depth map and a normal map of a grasping target surface are calculated according to the adjacent viewpoint images, and surface characteristic data is obtained;

[0050] When the robot reaches a preset grasping position, a sensing array arranged on a grasping surface acquires dynamic contact information of the grasping target through piezoelectric and piezoresistive dual-mode responses, and stable features are extracted from the dynamic contact information based on wavelet transform;

[0051] The cross-modal coupling is constructed based on Tucker decomposition, and the transient response channel and the steady-state response channel are constructed based on the surface characteristic data and the stability feature; the transient response channel and the steady-state response channel are synchronized through a recursive optimization algorithm combined with a time sequence consistency constraint and are adaptively fused based on an attention mechanism, so that the dual-loop coupling of haptics and vision is realized.

[0052] The mapping parameters between the dual-loop coupling output and the grasping parameters are established, the change rate of the grasping parameters obtained based on the mapping is used to construct an adaptive sliding time window, the time-varying characteristics of the grasping parameters are calculated in each time window, the grasping parameters are optimized using the time-varying characteristics combined with the Lyapunov stability criterion, the grasping posture and the clamping force of the robot are adjusted according to the optimized grasping parameters, and the influence of environmental disturbance and modeling error is eliminated through a dynamic compensation mechanism, so that adaptive grasping control is realized.

[0053] The light field camera array is composed of 8 high-definition cameras arranged on a circumference with a radius of 25 centimeters, and the adjacent cameras are spaced 45 degrees apart. The resolution of each camera is 4096x3072 pixels, and the frame rate is 60 frames per second. The camera array is triggered synchronously to ensure synchronization of the acquisition time, and the time error is less than 0.5 milliseconds. During the movement of the robot, the light field camera array continuously acquires adjacent viewpoint images with a sampling frequency of 30 hertz.

[0054] The adjacent viewpoint images are subjected to feature point extraction, and the target surface depth map is obtained by point cloud fitting. The local plane fitting algorithm is applied to the depth map to obtain the normal vector of each point, and the normal map is constructed.

[0055] The sensor array is arranged on the grasping surface of the robot and is composed of 4x4 sensor units, each unit having a size of 5mmx5mm and containing piezoelectric and piezoresistive dual-mode sensing elements. The piezoelectric element has a response frequency range of 0-1000 hertz and can detect high-frequency vibration signals; the piezoresistive element has a response frequency range of 0-100 hertz and is used to detect low-frequency pressure changes. The sampling frequency of the sensor array is 1000 hertz, the dynamic range is 0-10 newtons, and the resolution is 0.01 newtons.

[0056] When the robot reaches the preset grasping position, the sensor array contacts the target surface and acquires dynamic contact information. The contact information includes pressure distribution and material characteristic data. The acquired contact signals are subjected to multi-scale analysis using wavelet transform, and the Daubechies wavelet basis is used with 5 layers of decomposition. By analyzing the energy distribution of the wavelet coefficients, features including energy entropy, maximum coefficient value, and spectral center are extracted. For hard materials such as metal, the high-frequency coefficients of the wavelet are significant; for soft materials such as rubber, the low-frequency coefficients dominate. The stability features extracted include contact area, pressure distribution uniformity, sliding detection indicators, and vibration spectrum features.

[0057] When constructing the cross-modal coupling based on Tucker decomposition, the surface property data and the stability feature are respectively represented as third-order tensors, and the information of the two modalities is projected to the transient response channel and the steady-state response channel through Tucker decomposition. The transient response channel processes fast-changing signals, and the steady-state response channel processes slow-changing signals. The time consistency constraint is applied to the output of the two channels to ensure that the feature changes of adjacent time windows are smooth, and the constraint threshold is set to 0.2. The sliding time window method is used in the recursive optimization algorithm, and the window size is initially set to 200 milliseconds and is adaptively adjusted according to the change rate of the grasping parameters. The iteration number of the algorithm is 20 times, and the convergence threshold is 0.01. The attention mechanism allocates weights to the two channels, and the initial weight ratio is 1:1, which is dynamically adjusted according to the characteristics of the grasping task. For example, when it is detected that the target surface is smooth, the weight of the tactile channel is increased to 0.7 and the weight of the visual channel is reduced to 0.3; when it is detected that the ambient light is insufficient, the weight of the visual channel is reduced to 0.2 and the weight of the tactile channel is increased to 0.8.

[0058] In an alternative embodiment, adjacent view images are captured by a light field camera array, and a depth map and a normal map of the surface of the grasping target are calculated based on the adjacent view images to obtain the surface property data, including:

[0059] The epipolar pairs of the adjacent view images captured by the light field camera array are extracted, the gray difference values of the corresponding pixel points on the epipolar pairs are calculated, the disparity value corresponding to the smallest gray difference value is selected, and a disparity map is obtained;

[0060] The surface depth value is calculated using the disparity map, the first-order difference and the second-order difference are calculated in the pixel coordinate system based on the surface depth value, and the local normal vector is calculated based on the depth difference of adjacent pixel points. The direction projection of the local normal vector in the pixel coordinate system is taken as the surface orientation parameter, the deviation of the first-order difference from the preset smoothness threshold is taken as the depth smoothness constraint, and the change rate of the second-order difference and the surface orientation parameter is taken as the normal consistency constraint. The surface depth value and the local normal vector are iteratively adjusted to minimize the deviation of the depth smoothness constraint and the normal consistency constraint, and a depth map and a normal map are obtained;

[0061] The depth map and the normal map are taken as the surface property data of the grasping target.

[0062] The light field camera array consists of eight 4K resolution cameras arranged on a circular support with a radius of 30 cm, spaced 45 degrees apart, with a focal length of 35 mm. The camera intrinsic parameter matrix was obtained through calibration using a calibration board, with a calibration error of less than 0.1 pixels. The camera extrinsic parameters were obtained using a dual-target calibration method, with camera pose estimation accuracy better than 0.5 mm. The light field camera array synchronously acquires images at a frequency of 30 frames per second, with a resolution of 3840×2160 pixels and a color depth of 10 bits. The camera array is connected to a high-performance image processor equipped with a 12-core CPU and 16GB of memory, ensuring real-time image processing capabilities.

[0063] Epipolar geometric constraint methods are used to extract epipolar pairs from adjacent viewpoint images acquired by a light field camera array. For any two adjacent viewpoints, a fundamental matrix is ​​used to establish the correspondence between pixels. The fundamental matrix is ​​pre-calibrated with an accuracy better than 0.1 pixels. For each pixel in the image, a corresponding point is searched along the epipolar line in the other viewpoint image. Epipolar line calculation uses projection transformation, and the search range is limited to a region of ±50 pixels around the pixel. 256 epipolar pairs are extracted for each pair of viewpoint images. The epipolar line density is higher in the image center and lower in the edge region to ensure the processing accuracy of the target area. After the epipolar pairs are extracted, epipolar line correction is performed to ensure that corresponding points are on the same horizontal line, with a correction error of less than 0.2 pixels.

[0064] The grayscale difference between corresponding pixels on the epipolar pair is calculated using a normalized cross-correlation method. When searching for corresponding points, a 9×9 pixel window is taken centered on the target point, and the correlation coefficient with the candidate point window on the epipolar line is calculated. The grayscale difference is defined as 1 minus the correlation coefficient, ranging from 0 to 2; a smaller difference indicates a higher matching degree. Small windows (5×5 pixels) are used for texture-rich areas, and large windows (13×13 pixels) are used for texture-sparse areas, adaptively adjusting to improve matching accuracy. To reduce the impact of lighting variations, pixel values ​​within the window are normalized to zero mean. Occluded or unmatchable areas are marked as invalid matching points and handled specially in subsequent processing.

[0065] A dynamic programming algorithm is used to optimize the global matching results and select the disparity value corresponding to the minimum gray-level difference. For each pixel on the epipolar line, 20 candidate disparity values ​​are considered, ranging from 0 to 60 pixels, with a step size of 3 pixels. The gray-level difference value corresponding to each candidate disparity value is calculated, and a disparity-cost matrix is ​​constructed. A smoothing constraint is used to ensure the continuity of disparity changes between adjacent pixels, with a smoothing coefficient set to 0.8. For matching points with a gray-level difference value less than 0.1, sub-pixel accuracy optimization is performed, and the disparity accuracy is improved to 0.1 pixels using a quadratic curve fitting method. Through multi-view consistency checks, erroneous matches are eliminated, and matching results that are consistent in at least 3 viewpoints are retained. The final disparity map has the same resolution as the original image, with a disparity accuracy of 0.1 pixels.

[0066] The parallax map is converted to surface depth values based on the principle of triangulation. For each pixel in the parallax map, given its parallax value d, focal length f and baseline length b, the depth value z is calculated by the basic relation. In practical implementation, a parallax-depth mapping table is pre-computed, containing 1000 mapping points, and the depth value is obtained quickly by table lookup and linear interpolation at runtime. For distant points with parallax value less than 5 pixels, a non-linear mapping relation is adopted to improve the depth precision. The depth calculation precision decreases with distance, and the precision is 1 mm at 1 m distance and decreases to 8 mm at 3 m distance.

[0067] First-order and second-order differences are calculated in the pixel coordinate system according to the surface depth values using the Sobel operator. The first-order difference calculation uses a 3x3 window to calculate the gradients in the x and y directions, respectively, and the gradient amplitude is taken as the first-order difference value. The second-order difference is calculated simultaneously using a Laplacian operator, using a 5x5 window to reduce noise effects. For edge regions, a reflection padding is used to handle the boundary conditions. To reduce noise effects, a Gaussian filter is applied to the depth map, with a kernel size of 5x5 pixels and a standard deviation of 1.2 pixels. The first-order and second-order difference calculation results are stored in matrices with the same resolution as the depth map.

[0068] Least squares plane fitting is performed on the depth values in the 3x3 pixel window around each pixel point to calculate the local normal vector. The normal vector of the fitted plane is the local normal vector of the point. The normal vector is normalized to a unit vector, containing x, y, and z components. For regions with large depth changes, the window size is increased to 5x5 pixels to improve the stability of the normal estimation. For depth discontinuous regions, a depth difference threshold (threshold set to 5 mm) is used to separate different planes and avoid normal errors caused by cross-border plane fitting. The calculated normal map has the same resolution as the depth map, and each pixel point contains 3 channels representing the x, y, and z components of the normal vector.

[0069] The three-dimensional normal vector is projected onto the image plane to obtain a two-dimensional direction vector as the surface orientation parameter. The direction vector is normalized and represented by an angle, with a value range of 0 to 360 degrees. The surface orientation parameter reflects the local tilt direction of the surface and plays an important role in identifying the geometric features of the object. The surface orientation parameter is quantized into 36 direction intervals, each interval being 10 degrees, to construct an orientation histogram feature. For flat regions, the orientation parameter changes by less than 5 degrees; for edge regions, the orientation parameter can change by up to 45 degrees.

[0070] The calculation of the depth smoothness constraint involves the deviation of the first-order difference from a preset smoothness threshold. Different smoothness thresholds are set for different materials: 0.05 for metal surfaces, 0.1 for plastic surfaces, and 0.2 for fabric surfaces. The deviation is calculated as the absolute difference between the first-order difference and the corresponding threshold. For areas that exceed the threshold, they are marked as areas that require smoothing. To accommodate surface changes of different scales, a multi-scale pyramid is constructed, containing 3 levels with a downsampling factor of 2 for each level. The depth smoothness constraint is calculated at different scales.

[0071] The calculation of the normal consistency constraint involves the correlation between the second-order difference and the rate of change of the surface orientation parameter. The rate of change of the surface orientation parameter is calculated by the difference between the orientation parameters of adjacent pixels, with a unit of degrees per pixel. Assuming that the second-order depth difference of a continuous surface region is proportional to the rate of change of the orientation, the proportionality coefficient is adjusted according to the surface material characteristics: 0.8 for metal surfaces, 1.0 for plastic surfaces, and 1.2 for fabric surfaces. The normal consistency constraint is defined as the difference between the second-order difference and the rate of change of the orientation multiplied by the proportionality coefficient. The smaller the constraint value, the better the consistency of the depth value and the normal vector.

[0072] An alternating optimization strategy is used to iteratively adjust the surface depth value and the local normal vector. The maximum number of iterations is set to 50, and the convergence threshold is set to 0.001. In each iteration, the depth value is optimized first by fixing the normal vector, and then the normal vector is optimized by fixing the depth value. The depth value optimization uses the weighted least squares method, with the weight being inversely proportional to the depth smoothness constraint. The normal vector optimization uses the gradient descent method, with the step size initially set to 0.1 and gradually reduced to 0.01 as the number of iterations increases. To speed up convergence, a multi-resolution strategy is used, which first optimizes in low-resolution images and then gradually increases to the original resolution. In typical scenarios, the algorithm converges after 8-12 iterations, with a calculation time of less than 200 milliseconds.

[0073] An image sequence of a metal part is processed as an actual application case. The light field camera array captures 8 view images with a resolution of 3840x2160 pixels. After epipolar line pair extraction, about 2.4 million valid matching points are obtained, with a matching accuracy of 95%. In the calculated disparity map, the part region has a disparity range of 15-40 pixels, and the background region has a disparity of less than 5 pixels. After converting to depth values, the part is about 85 cm away from the camera center, with a depth variation range of 5 cm. The first-order difference calculation result shows that the edge gradient value of the part is greater than 0.2, and the plane area gradient value is less than 0.05. The second-order difference has obvious peak values at the edge junction, with a maximum value of 0.4. The normal vector calculation result clearly shows the planes, cylindrical surfaces, and transition surfaces of the part. The surface orientation parameter changes less than 3 degrees per pixel in the plane area and up to 30 degrees per pixel in the edge area.

[0074] The depth map quality is significantly improved after iterative optimization with depth smoothness constraint and normal consistency constraint. The depth noise standard deviation is reduced from 1.8 mm to 0.6 mm. The flatness of planar regions is improved by 70%, and the edge sharpness is improved by 50%. The noise in the normal map is reduced by 65%, and the edge transition is smoother and more coherent. The final depth map and normal map have a resolution of 1920x1080 pixels and a data size of 12 MB. Each pixel in the depth map stores a 16-bit depth value ranging from 0 to 65535, corresponding to an actual depth of 0 to 5 meters. Each pixel in the normal map stores a normal vector with three components, each with 8-bit precision, ranging from 0 to 255, corresponding to a normal vector component of -1 to 1.

[0075] The geometry of the target can be further analyzed, and the depth map and normal map can be used as surface property data for grasping the target. The size, position, and contour information of the target can be extracted from the depth map; planar, curved, and edge features can be identified from the normal map. The depth map and normal map are combined and analyzed to extract graspable points and grasping poses, providing a basis for subsequent grasping planning. In the case of processing metal parts, four planar regions suitable for grasping are successfully identified, with areas of 18, 22, 15, and 12 square centimeters, and normal consistency scores of 0.92, 0.89, 0.85, and 0.81, respectively. The largest and highest normal consistency region is selected as the preferred grasping region to ensure grasping stability.

[0076] In an alternative embodiment, the cross-modal coupling is constructed based on Tucker decomposition, and the transient response channel and the steady-state response channel are constructed based on the surface property data and the stability feature, comprising:

[0077] The surface property data is decoupled in the spatial dimension to obtain deformation components and force components, and the stability feature is decoupled in the time dimension to obtain transient response components and steady-state response components; the deformation components and the transient response components are aligned and combined in the spatial dimension and the time dimension, respectively, to obtain a transient feature tensor; the force components and the steady-state response components are aligned and combined in the spatial dimension and the time dimension, respectively, to obtain a steady-state feature tensor; the transient feature tensor and the steady-state feature tensor are subjected to Tucker decomposition to obtain a transient core tensor and a steady-state core tensor, respectively;

[0078] A coupling measure matrix is constructed based on the tensor outer product operation of the transient core tensor and the steady-state core tensor, and the product operation result of the transient feature tensor and the coupling measure matrix is taken as the transient response channel, and the product operation result of the steady-state feature tensor and the transpose of the coupling measure matrix is taken as the steady-state response channel.

[0079] As shown in Figure 2 the method comprises:

[0080] The principal component analysis technique is used to decouple the surface property data in the spatial dimension to obtain the deformation component and the force component. The surface property data composed of the depth map and the normal map is represented as a third-order tensor of 512x512x4, where 512x512 represents the spatial resolution and 4 represents the feature dimension (depth value and three-dimensional normal vector). The principal component analysis is applied to the feature dimension to extract two principal components, the first principal component corresponding to the deformation component, which describes the geometric shape change of the object surface, and the second principal component corresponding to the force component, which reflects the elastic properties of the surface under force. Both the deformation component and the force component are tensors of 512x512x2. For metal material objects, the deformation component accounts for about 85% of the total variance, and the force component accounts for about 15%; for rubber material objects, the deformation component accounts for about 60%, and the force component accounts for about 40%. The decoupled data is normalized, and the value range is mapped to the [0, 1] interval.

[0081] The frequency domain analysis method is used to decouple the stability features in the time dimension to obtain the transient response component and the steady-state response component. The Fourier transform is applied to the stability features extracted from the dynamic contact information, converting the stability feature tensor of 16x16x32 (16x16 represents the spatial resolution of the sensor array, and 32 represents the feature dimension) to the frequency domain. The frequency domain data is divided into two parts according to the frequency: the part above 10 Hz corresponds to the transient response component, which describes the rapid change signal in the initial stage of contact; the part below 10 Hz corresponds to the steady-state response component, which describes the slow change signal in the sustained contact state. The two components are converted back to the time domain by inverse Fourier transform, both being tensors of 16x16x16. For hard surface contact, the transient response component energy accounts for about 70%; for soft surface contact, the steady-state response component energy accounts for about 65%.

[0082] The alignment and combination of the deformation component and the transient response component in the spatial and time dimensions use the space-time mapping technique. The spatial alignment maps the visual spatial resolution of 512x512 to the tactile spatial resolution of 16x16, using the bicubic interpolation method for downsampling. The temporal alignment synchronizes the visual data (30 Hz) with the tactile transient data (1000 Hz) by linear interpolation to expand the time resolution of the visual data. The aligned deformation component and transient response component are combined by channel splicing to obtain a transient feature tensor of 16x16x18, where the 18 channels include 2 deformation component channels and 16 transient response component channels. In the combination process, weight coefficients are applied to the visual and tactile features, with initial values of 0.4 and 0.6, respectively, which are dynamically adjusted according to the signal-to-noise ratio.

[0083] The alignment and combination process of the force component and the steady-state response component is similar to the above, but the time synchronization window is larger. The spatial alignment also uses bicubic interpolation to downsample the force component to 16x16 resolution. The time alignment uses a sliding average window (window size 200ms) to smooth the steady-state response component, making its time characteristics match the force component. The aligned force component and steady-state response component are combined through channel splicing to obtain a 16x16x18 steady-state feature tensor, containing 2 force component channels and 16 steady-state response component channels. The weight coefficient during combination is dynamically adjusted according to the contact stability, and the tactile feature weight is increased to 0.7 in the stable contact state.

[0084] The transient feature tensor and the steady-state feature tensor are respectively decomposed by Tucker to obtain the transient core tensor and the steady-state core tensor using the high-order singular value decomposition algorithm. For the 16x16x18 transient feature tensor, the decomposition rank is selected as (8, 8, 6), that is, the dimension is reduced to 8, 8, and 6 respectively in three dimensions. The decomposition process is iterated 100 times, and the convergence threshold is set to 0.001, to obtain an 8x8x6 transient core tensor and three factor matrices (sizes are 16x8, 16x8, and 18x6 respectively). For the steady-state feature tensor, the decomposition rank is also (8, 8, 6), to obtain an 8x8x6 steady-state core tensor and the corresponding three factor matrices. The core tensor retains about 92% of the energy of the original tensor, with a compression ratio of 75%. For the metal object grasping scene, the first three singular values in the transient core tensor are 42.5, 28.3, and 15.9 respectively; the first three singular values in the steady-state core tensor are 36.2, 24.8, and 18.5 respectively.

[0085] The 8x8x6 transient core tensor and the 8x8x6 steady-state core tensor are reshaped into 384-dimensional vectors (8x8x6=384), the outer product of the two vectors is calculated to obtain a 384x384 coupling measure matrix. Each element in the matrix represents the association strength between the corresponding modal features. To enhance the cross-modal association, a sparse constraint is applied to the matrix, retaining the top 20% of the elements in terms of association strength, and setting the rest to zero. The matrix is normalized so that the sum of rows and columns is 1, ensuring the balance of information transmission between modes. For the metal object grasping task, the average strength of the transient-steady-state association region in the matrix is 0.72, higher than the average strength of the intra-modal association 0.55.

[0086] First, the 16x16x18 transient feature tensor is reshaped into a 384-dimensional vector (16x16x1.5=384, the original data is compressed by feature compression), and then multiplied by the 384x384 coupling measure matrix to obtain a 384-dimensional coupling response vector. The vector is reshaped back into a 16x16x1.5 tensor form as the output of the transient response channel. Sparse matrix multiplication algorithm is used in the operation process, and the computational complexity is reduced from O(n 2) to O(n log(n)). The transient response channel has high sensitivity to the rapidly changing signal in the initial stage of contact, and the response delay is less than 15 milliseconds.

[0087] The steady-state feature tensor of 16x16x18 is reshaped into a 384-dimensional vector, multiplied by the transpose of the 384x384 coupling metric matrix, and a 384-dimensional coupling response vector is obtained, which is reshaped into a 16x16x1.5 tensor form as the output of the steady-state response channel. The steady-state response channel has good stability to the slowly changing signal in the continuous contact state, can effectively filter high-frequency noise, and the signal-to-noise ratio is improved by about 6dB. By using the matrix transpose operation, the information flow between the transient and steady-state channels is bidirectional and symmetrically transmitted, enhancing the system robustness.

[0088] In an actual application case, a ceramic cup is handled for a grasping task. The surface property data obtained by the light field camera array shows that the cup surface is smooth, the depth value changes gently, and the normal vector changes significantly at the cup mouth and bottom edge. After decoupling of the surface property data, the deformation component highlights the geometric profile of the cup, with an energy proportion of 82%; the force component reflects the surface smoothness and material properties, with an energy proportion of 18%. The dynamic contact information obtained by the tactile sensing array when contacting the cup contains obvious high-frequency vibration signals (in the initial stage of contact) and stable pressure distribution (in the continuous grasping stage). After decoupling of the signals, the transient response component has an energy concentrated in the first 100 milliseconds, with a peak frequency of about 120 Hz; the steady-state response component tends to be stable after 200 milliseconds, with a main frequency component less than 5 Hz.

[0089] In the space-time alignment process, the 512x512 visual data is down-sampled to 16x16, retaining the detailed information around the key feature points. Time synchronization aligns the 30 Hz visual data with the 1000 Hz tactile data, with an interpolation error of less than 2 milliseconds. The combined transient feature tensor captures the correlation between the geometric features of the cup edge and the initial contact vibration, with a correlation coefficient of 0.78; the steady-state feature tensor captures the correlation between the surface smoothness and the continuous pressure distribution, with a correlation coefficient of 0.85.

[0090] In the Tucker decomposition process, an appropriate decomposition rank is selected to balance information retention and computational efficiency. For the transient feature tensor, the decomposition retains 94% of the energy of the original data; for the steady-state feature tensor, 91% of the energy is retained. The transient core tensor obtained by decomposition is 8x8x6, which is only 15% of the size of the original tensor; the steady-state core tensor is also 8x8x6, which is 15% of the size of the original tensor. The decomposition process takes about 45 milliseconds, meeting the real-time processing requirements.

[0091] In the process of coupling measurement matrix construction, a 384x384 matrix is calculated, in which the proportion of non-zero elements is about 20%. The elements with larger values (greater than 0.8) in the matrix are mainly distributed in the regions representing the association between the geometric characteristics of the cup edge and the tactile vibration signal, and the regions representing the association between the surface smoothness and the pressure distribution. After sparse and normalized processing, the data volume is reduced from about 590KB to 120KB, and the calculation efficiency is improved by about 4 times.

[0092] The calculation process of the transient response channel and the steady-state response channel both use sparse matrix multiplication algorithm, and the calculation time is less than 10 milliseconds. The transient response channel is sensitive to the signal response in the initial stage of contact, and can produce obvious output change within 15 milliseconds for the tiny sliding (displacement less than 0.2mm) detected on the edge of the cup. The steady-state response channel has good stability to the pressure distribution change in the continuous grabbing stage, and the output fluctuation is less than 3% under external vibration interference (acceleration less than 0.5g). Through the combination of the two channels, the dynamic changes and steady-state characteristics in the grabbing process can be fully captured.

[0093] The dual-channel processing architecture significantly improves the robustness and adaptability of the system. Compared with single modal processing, the transient-steady dual-channel architecture improves the success rate of grabbing by 18% and the stability of grabbing by 25%. Especially in the scene of changing light conditions (light intensity changes ± 50%) or diversified surface materials (from smooth ceramic to rough paper cup), the system performance decreases by less than 10%, while the performance of single modal method decreases by more than 30%. Through the cross-modal coupling mechanism constructed by Tucker decomposition, the efficient fusion of visual and tactile information is realized, which provides a reliable perception basis for subsequent grabbing control.

[0094] In an alternative embodiment, a mapping parameter between the dual-loop coupled output and the grabbing parameter is established, and an adaptive sliding time window is constructed based on the change rate of the grabbing parameter obtained by mapping. The time-varying characteristics of the grabbing parameter in each time window are calculated, including:

[0095] The difference value of the grabbing parameter at adjacent sampling time points is calculated, the change rate of the grabbing parameter is obtained by dividing the difference value by the sampling time interval, and an adaptive sliding time window is constructed based on the change rate. The length of the adaptive sliding time window is obtained by the negative exponential operation of the preset basic window length and the change rate.

[0096] The state transition matrix of the grasping parameter is established within the adaptive sliding time window, the state estimation gain of the grasping parameter is calculated based on the Kalman filter, and the state transition matrix is iteratively updated using the state estimation gain; the eigenvalue of the state transition matrix is calculated, the norm difference value of adjacent two iterations of the state transition matrix is counted, and the time-varying characteristics of the grasping parameter are obtained according to the distribution of the eigenvalue and the change trend of the norm difference value.

[0097] The double-ring coupling output contains the fusion characteristics of the transient response channel and the steady-state response channel, and is a vector with a dimension of 32x1. The grasping parameter contains the clamping force, the grasping position and the grasping posture, and is represented as a 6x1 vector, wherein the clamping force is a scalar, and the grasping position and the grasping posture are 3-dimensional vectors. The mapping relationship is established through a three-layer neural network, the network structure is 32-16-6, the activation function adopts the ReLU function, and the linear activation is used in the output layer. The network training adopts the stochastic gradient descent method, the learning rate is set to 0.01, the weight decay coefficient is 0.0005, and the momentum parameter is 0.9. The training sample contains 5000 groups of corresponding data of the double-ring coupling output and the grasping parameter, the cross-validation method is used in the training process, and the verification set proportion is 20%. After 500 iterations of training, the mean square error on the verification set is reduced to below 0.03, and the mapping parameters are stable.

[0098] The sampling frequency of the grasping parameter is 100 Hz, that is, the sampling time interval is 10 ms. The grasping parameter values of 10 consecutive samples are recorded to form a sliding buffer. For adjacent samples in the buffer, the parameter difference value is calculated. Taking the clamping force as an example, if the sampling values of two consecutive samples are 5.2 N and 5.5 N, the difference value is 0.3 N. Divide this difference value by the sampling time interval of 10 ms to obtain the change rate of 0.03 N / ms, that is, 30 N / s. Similarly, the change rate of the grasping position is in units of mm / s, and the change rate of the grasping posture is in units of ° / s. Considering the change rates of all parameters, the weighted average value of the overall change rate is taken as the weighted average value, and the weights are: clamping force 0.4, grasping position 0.4, and grasping posture 0.2.

[0099] The preset base window length is 500 milliseconds, which is suitable for a stable state with a change rate close to zero. The actual window length is obtained by the base window length and the change rate negative exponential operation. In the specific implementation, the exponential function is expressed as the result of the base window length multiplied by the negative change rate of e divided by the threshold coefficient. The threshold coefficient is set to 5, which has the same unit as the change rate. Taking the case of a change rate of 30 Newtons per second as an example, the window length is calculated as 500 milliseconds multiplied by -30 / 5 of e, which is about 500 milliseconds multiplied by 0.0025, and the result is 1.25 milliseconds. Considering the system processing capacity and stability requirements, the lower limit of the window length is set to 50 milliseconds, and the upper limit is set to 1000 milliseconds. Therefore, in the above example, the actual window length takes the lower limit value of 50 milliseconds. For a smaller change rate, such as 2 Newtons per second, the window length is 500 milliseconds multiplied by -2 / 5 of e, which is about 303 milliseconds.

[0100] The adaptive window length dynamically adjusts in response to parameter changes during the grabbing process. For a stable grabbing stage, the change rate is usually less than 0.5 Newtons per second, and the window length is close to 500 milliseconds; for a fast adjustment stage, such as when sliding is detected, the change rate can reach 50 Newtons per second, and the window length is reduced to 50 milliseconds, improving the system response speed. The window length changes use smooth transitions, and the change in adjacent window lengths does not exceed 30%, avoiding system instability caused by sudden changes.

[0101] A 6x6 matrix is used to represent the state transition relationship of the grabbing parameters, and the matrix elements represent the degree of mutual influence between parameters. The state vector includes the current grabbing parameter value and its first derivative, so the matrix dimension is 12x12. The initial state transition matrix uses a unit matrix with random perturbations, and the perturbation amplitude is 0.1. In each time window, N sample points are collected, and N is equal to the window length divided by the sampling interval. Taking a window length of 300 milliseconds as an example, the sampling interval is 10 milliseconds, and a total of 30 sample points are collected.

[0102] The least squares method is used to establish the state transition matrix. For two consecutive state vectors X(k) and X(k+1), the matrix A that satisfies X(k+1)=A×X(k) is calculated, where A is the state transition matrix. Considering the influence of measurement noise, all adjacent state pairs in the collection window are collected to construct an over-constrained equation system, and the optimal matrix A is solved by the least squares method. The singular value decomposition method is used in the solving process, and the computational complexity is O(n 3 ), where n is the dimension of the state vector 12. For 30 sample points in a 300 millisecond window, 29 pairs of state transition relationships are formed, and 348 equations (29x12) are constructed to solve the 12x12 matrix A.

[0103] The state space model is established, including state equation and observation equation. The state equation is composed of state transition matrix and process noise, and the observation equation is composed of observation matrix and observation noise. The process noise covariance matrix is initially set as a diagonal matrix with diagonal element value of 0.01, representing the uncertainty of state transition. The observation noise covariance matrix is also set as a diagonal matrix with diagonal element value of 0.05, representing the uncertainty of sensor measurement. The observation matrix is a unit matrix, representing that the state vector can be directly observed.

[0104] The state estimation gain calculation includes two steps of prediction and update. The prediction step uses the state transition matrix to predict the next state and its covariance; the update step corrects the prediction result according to the actual observation value. The state estimation gain K is a 12x12 matrix, and its element value represents the degree of trust in the observation value. The larger the gain value, the more trust in the observation value; the smaller the gain value, the more trust in the model prediction. The gain matrix K is calculated by a recursive equation, involving the state covariance matrix, the observation covariance matrix and the observation matrix. Typically, the diagonal element value of the gain matrix of the stable state is about 0.6-0.8, representing the balanced trust in the observation value and the model prediction.

[0105] The state transition matrix is iteratively updated by the state estimation gain. In each time window, the initial state transition matrix is used for prediction, the prediction error is calculated by the observation value, and the state transition matrix is corrected by the state estimation gain. The correction process is to add the product of the gain matrix and the prediction error to the original matrix. The iteration process is repeated several times until the matrix converges or reaches the maximum number of iterations. The maximum number of iterations is set to 10, and the convergence threshold is 0.001, that is, when the norm difference of the matrix between adjacent two iterations is less than the threshold, it is considered to be converged. For the stable grasping stage, usually 3-5 iterations can converge; for the fast changing stage, all 10 iterations are needed.

[0106] The 12x12 matrix has 12 eigenvalues, including real and complex numbers. Eigenvalue modulus greater than 1 indicates system divergence, modulus less than 1 indicates system convergence, and modulus equal to 1 indicates system critical stability. The distribution of eigenvalues is concerned, including the maximum eigenvalue modulus, the minimum eigenvalue modulus, the average eigenvalue modulus and the standard deviation of eigenvalue modulus. Eigenvalue decomposition uses QR algorithm with computational complexity of O(n 3 ) For stable grasping state, the eigenvalue modulus is usually distributed between 0.8-1.0; for unstable state, the maximum eigenvalue modulus is more than 1.2.

[0107] The Frobenius norm is used to calculate the norm difference of the state transition matrix. For a matrix A, its Frobenius norm is defined as the square root of the sum of the squares of all elements in the matrix. The norm difference is the absolute value of the difference between the norms of two consecutive iterations of the matrix. The norm difference reflects the magnitude of the change in the matrix, with a smaller difference indicating a more stable matrix. The norm differences of the last 10 windows are recorded to analyze the trend. The trend is determined by the slope of the linear regression, with a positive slope indicating increasing instability and a negative slope indicating stability. The slope threshold is set to ±0.005, and an absolute value less than the threshold is considered stable.

[0108] The time-varying characteristics of the grasping parameters are determined by the eigenvalue distribution and the norm difference trend. Four types of time-varying characteristics are defined: stable convergence, stable oscillation, unstable divergence, and unstable oscillation. Stable convergence is characterized by eigenvalue modulus less than 0.95 and a negative norm difference trend. Stable oscillation is characterized by eigenvalue modulus between 0.95 and 1.05 and a near-zero norm difference trend. Unstable divergence is characterized by a maximum eigenvalue modulus greater than 1.1 and a positive norm difference trend. Unstable oscillation is characterized by periodic eigenvalue modulus exceeding 1.0 and periodic norm difference. The control strategy is adjusted based on the identified time-varying characteristics, such as reducing the control gain in stable convergence or increasing the damping coefficient in unstable divergence.

[0109] In a practical application case, a glass cup is handled. In the initial grasping stage, the clamping force increases from 0 to 3 Newton, with a rate of about 20 Newton / s, and the adaptive window length is 80 milliseconds. The maximum eigenvalue modulus of the state transition matrix is 1.15, and the norm difference trend is positive, indicating unstable divergence. The control system increases the damping coefficient to reduce the clamping force increase rate. In the stable grasping stage, the clamping force fluctuates around 3 Newton with a rate less than 0.3 Newton / s, and the window length increases to 450 milliseconds. The eigenvalue modulus is distributed between 0.85 and 0.98, and the norm difference tends to zero, indicating stable convergence.

[0110] When an external disturbance is detected (such as a sudden 0.5 mm drop of the glass cup), the clamping force increases sharply to 4.5 Newton with a rate of 40 Newton / s, and the window length decreases to 50 milliseconds. The state transition matrix changes rapidly, the norm difference increases to 0.15, and a dominant eigenvalue with a modulus of 1.3 appears, indicating unstable divergence. The control system responds immediately by adjusting the grasping position to move up 0.6 mm while maintaining high clamping force, and the system stabilizes again after 800 milliseconds. The maximum eigenvalue modulus of the stable state transition matrix is 0.92, and the norm difference is less than 0.001, indicating stable convergence.

[0111] By analyzing the time-varying characteristics of the adaptive window, different stages and abnormal situations in the grabbing process can be accurately identified. Compared with the fixed window method, the abnormality detection sensitivity is improved by 35%, the response speed is improved by 50%, and the false negative rate is reduced to below 3%. Especially for the grabbing of smooth surface objects, the adaptive window can detect abnormalities at the initial stage of sliding (displacement less than 0.3 mm), preventing grabbing failure. Experiments show that this method has good adaptability in grabbing tasks of objects of different materials (metal, glass, plastic, cloth) and different shapes (regular, irregular), and the grabbing success rate is improved by 23% and the grabbing stability is improved by 31%.

[0112] In an optional implementation, the grasping parameters are optimized using the time-varying characteristics combined with Lyapunov stability criteria, and the grasping posture and clamping force of the robot are adjusted according to the optimized grasping parameters, while the influence of environmental disturbances and modeling errors is eliminated through a dynamic compensation mechanism, including:

[0113] The state transition eigenvalue and norm difference in the time-varying characteristics are combined to form a state vector, a positive definite matrix is constructed based on the state transition eigenvalue, the product of the state vector and the positive definite matrix is added to the integral term of the state vector and the time derivative is calculated, and the product of the time derivative and a preset convergence coefficient is less than zero as a stability constraint condition;

[0114] Under the premise of meeting the stability constraint condition, the weighted squared error between the grasping parameters and a preset expected parameter value is calculated, the gradient of the weighted squared error with respect to the grasping parameters is calculated, and the product of the gradient and a preset learning rate is subtracted from the grasping parameters to obtain optimized grasping parameters;

[0115] A basic control quantity is generated through the optimized grasping parameters, and the grasping posture and clamping force of the robot are adjusted using the basic control quantity;

[0116] The environmental disturbance is decomposed into multi-dimensional disturbance features through wavelet transform, and based on the multi-dimensional disturbance features, a disturbance compensation control quantity is obtained by using fuzzy rule reasoning, a modeling error signal is calculated, and an error compensation control quantity is generated by combining a robust adaptive control strategy; The disturbance compensation control quantity and the error compensation control quantity are superimposed on the basic control quantity.

[0117] The state transition eigenvalues in the time-varying characteristics contain 12 complex values, representing the dynamic characteristics of the grasping system. The module lengths of these eigenvalues are sorted, and the first 6 principal eigenvalues are taken as λ1 to λ6, with typical values ranging from 0.8 to 1.2. The norm difference reflects the stability of the system, and the norm difference of the last 5 time windows is usually taken as d1 to d5, with a value range of usually 0 to 0.2. The state vector is composed of these 11 parameters, with a dimension of 11x1. For a stable grasping state, the eigenvalue module length is usually less than 1, and the norm difference is close to 0; for an unstable state, at least one eigenvalue module length is greater than 1, and the norm difference is large and fluctuates obviously.

[0118] The positive definite matrix is constructed based on the state transition eigenvalues. First, the statistical characteristics of the eigenvalues are calculated, including the average module length μ, the maximum module length λ max , and the module length standard deviation σ. The positive definite matrix P adopts a diagonal matrix form, and the diagonal elements are determined by the statistical characteristics of the eigenvalues. Specifically, the first 6 diagonal elements of the matrix are p1 to p6, which are calculated by dividing 1 by the corresponding eigenvalue module length λ i , and then multiplied by the adjustment coefficient α. The initial value of the adjustment coefficient α is set to 1.5, which is dynamically adjusted according to the stability of the system, with a smaller value for a stable system and a larger value for an unstable system. The last 5 diagonal elements of the matrix are q1 to q5, which are set to the reciprocal of the norm difference multiplied by the adjustment coefficient β, with β initially set to 2.0. The physical meaning of the positive definite matrix is to normalize and assign weights to different state components, making the stability evaluation more accurate.

[0119] The state vector is multiplied by the positive definite matrix to obtain a weighted state vector, with a dimension of 11x1. The integral term of the state vector in the past T time is calculated, with the integral time T usually set to 20% of the duration of the grasping task, such as 1 second. The integral term reflects the cumulative effect of the system state, and for a stable system, the integral term tends to a stable value; for an unstable system, the integral term continuously increases. The weighted state vector and the integral term are added to obtain the Lyapunov function value V, and the larger the function value, the more unstable the system.

[0120] The time derivative of the Lyapunov function is calculated by the finite difference method. Record the function values V(t-2Δt), V(t-Δt) and V(t) at three consecutive time points, and use the central difference formula to calculate the derivative at t-Δt, with Δt being the system sampling period, usually 10 milliseconds. In order to reduce the influence of noise, the derivative value is smoothed by a moving average filter with a window size of 5 sampling points. The derivative value is multiplied by a preset convergence coefficient γ, with γ ranging from 0.1 to 1.0, which is adjusted according to task requirements. The stability constraint condition requires that this product be less than zero, i.e. the derivative is negative, indicating that the system energy function decreases with time and tends to be stable.

[0121] The grasping parameters include the gripping force F, the grasping position coordinates (x, y, z), and the grasping attitude angles (α, β, γ), totaling 7 parameters. Preset expected parameter values ​​are set according to the grasping task. For example, the expected gripping force F is determined based on the object's mass and coefficient of friction; for a 100-gram smooth plastic object, a typical value for F is 3 Newtons; the expected grasping position... ) represents the coordinates of the object's center of gravity, and the expected value of the grasping posture is ( This ensures that the gripping surface is parallel to the object's surface. The parameter weights are set according to their importance; typically, the gripping force weight wF = 0.4, the position weights wx = wy = wz = 0.15, and the attitude weights wα = wβ = wγ = 0.05.

[0122] The weighted squared error is calculated as the sum of the squares of the differences between each parameter and its expected value, multiplied by its corresponding weight. For a clamping force of 3.5 Newtons, an expected value of 3 Newtons, and a weight of 0.4, its contribution is 0.4 × (3.5 - 3). 2 =0.1; the total error value is obtained by adding the contribution values ​​of each parameter. The gradient of the error function with respect to the grasping parameters is calculated analytically. For the grasping force F, the gradient is 2×wF×( Other parameters are similar. The gradient vector has a dimension of 7×1, the same as the parameter vector dimension.

[0123] The optimization process employs gradient descent, subtracting the gradient vector multiplied by the learning rate η from the parameter vector in each iteration. The initial learning rate is set to 0.05 and dynamically adjusted based on stability constraints: increasing the learning rate accelerates convergence when constraint satisfaction is high, and decreasing it when constraints approach critical levels to ensure stability. The maximum number of iterations is set to 50, and the convergence threshold is 0.001, meaning convergence is considered achieved when the parameter change is less than the threshold for three consecutive iterations. The optimization algorithm also sets parameter boundary constraints, such as a clamping force range of 0 to 10 Newtons, with values ​​outside this range truncated to the boundary value.

[0124] The basic control variables are generated from the optimized gripping parameters. The gripping force control variable CF is directly equal to the optimized gripping force value F, in Newtons; the position control variables Cx, Cy, and Cz are calculated from the difference between the current position and the target position, in millimeters; the attitude control variables Cα, Cβ, and Cγ are calculated from the difference between the current attitude and the target attitude, in degrees. After gain adjustment and filtering smoothing, the basic control variables are sent to the robot controller to control the gripper to perform the gripping action. The gain coefficient is usually set to 1.2, and the smoothing filter uses the exponential moving average method with a smoothing coefficient of 0.8.

[0125] The disturbance compensation control quantity and the error compensation control quantity are superimposed on the base control quantity to obtain the final control quantity. Weighting coefficients are set during the superposition process; typically, the disturbance compensation weight is 0.6 and the error compensation weight is 0.4. The final control quantity is then limited to ensure it remains within the actuator's allowable range, such as limiting the clamping force control quantity to 0 to 10 Newtons, the position control quantity to ±20 mm, and the attitude control quantity to ±30 degrees. A control quantity change rate limit is also set to prevent sudden changes in control input from causing mechanical shock; typical limits are 2 Newtons / second for clamping force, 10 mm / second for position, and 15 degrees / second for attitude.

[0126] In a practical application case, a task of grasping a metal part was handled. The part weighed 250 grams, had a moderately smooth surface, and the initial clamping force was set to 4 Newtons. During the grasping process, a sudden environmental vibration disturbance occurred, with an amplitude of approximately 0.5g and a frequency of 12 Hz. After wavelet decomposition, the energy of the disturbance signal detected by the sensor was mainly concentrated in the 10-15 Hz frequency band, and the corresponding component value in the eigenvector increased to 0.78. The fuzzy system identified it as a moderate-intensity vibration disturbance and generated disturbance compensation control variables: clamping force compensation +1.2 Newtons, position Z compensation -0.8 mm, and other compensation variables were relatively small.

[0127] Meanwhile, the actual clamping force was found to be 0.6 Newtons lower than the model prediction, indicating a friction modeling error. Robust adaptive control generated error compensation control quantities: clamping force compensation +0.8 Newtons, while position and attitude compensations were close to zero. After the two compensation control quantities were superimposed, the final clamping force increased by 1.7 Newtons (0.6 × 1.2 + 0.4 × 0.8 + basic compensation), reaching 5.7 Newtons; the Z-direction position shifted down by 0.5 mm (0.6 × (-0.8) + basic compensation). The compensation action was completed within 150 milliseconds after the vibration began, effectively suppressing the disturbance and preventing the object from slipping.

[0128] After the vibration stopped, the stability was reassessed. The maximum modulus of the state transition eigenvalue decreased from 1.15 to 0.93, the norm difference decreased from 0.12 to 0.03, and the derivative of the Lyapunov function was negative, satisfying the stability constraints. The optimization algorithm calculated new target parameters: a clamping force of 5.2 Newtons, with fine-tuning of position and attitude. The control variables were gradually adjusted to the new target values, completing the transition within 200 milliseconds and achieving stable grasping. Throughout the process, the maximum object position offset did not exceed 1.2 mm, far less than the 5.6 mm offset without compensation, demonstrating the effectiveness of the dynamic compensation mechanism.

[0129] Multiple experiments show that this method performs well under various disturbance conditions. For periodic vibration disturbance (frequency 5-20 Hz, amplitude 0.2-0.8 g), the suppression rate reaches 85%; for impact disturbance (peak acceleration 2 g), the response time is less than 100 milliseconds, and the maximum displacement is controlled within 2 millimeters; for slowly changing environmental factors (such as a ±20% change in friction coefficient caused by temperature changes), it can automatically adapt and maintain stable grasping, with a clamping force error of ±5%. Compared with traditional PID control, this method improves the anti-disturbance ability by 46% and the energy efficiency by 23%, while significantly reducing the risk of object deformation caused by excessive clamping.

[0130] In an alternative embodiment, the environmental disturbance is decomposed into multi-dimensional disturbance features by wavelet transform, and based on the multi-dimensional disturbance features, a disturbance compensation control amount is obtained by fuzzy rule reasoning, a modeling error signal is calculated, and an error compensation control amount is generated by combining a robust adaptive control strategy, including:

[0131] The environmental disturbance is decomposed into multi-scale coefficients by wavelet transform, the energy distribution of the multi-scale coefficients in different frequency bands is calculated to obtain a multi-dimensional disturbance feature vector, the membership value of the multi-dimensional disturbance feature vector is calculated, the membership value is matched with a preset fuzzy rule set to obtain a rule activation strength, and the rule activation strength is weighted and averaged to obtain a disturbance compensation control amount;

[0132] The deviation of the state transition matrix from the actual system response is calculated to obtain a modeling error signal, the two-norm of the modeling error signal is calculated, and a projection boundary value is set according to the two-norm, the adaptive dynamic parameter of the modeling error signal is calculated, the adaptive dynamic parameter is projected into the projection boundary value and integrated to obtain an adaptive gain matrix, and the modeling error signal is multiplied by the adaptive gain matrix to obtain an error compensation control amount.

[0133] The environmental disturbance signal is collected by the force tactile sensor and proximity sensor at the end of the robot, with a sampling frequency of 1000 Hz and a signal dimension of 6, including three-dimensional force signals and three-dimensional torque signals. The raw signals collected are preprocessed, including median filtering to remove sudden noise and bandpass filtering (cutoff frequency 1-200 Hz) to eliminate DC bias and high-frequency interference. The preprocessed signals form data segments with a length of 512 points, with an overlap of 256 points to ensure the continuity of the disturbance analysis. For normal grasping state, the force signal fluctuation range is usually within ±0.2 Newton, and the torque signal fluctuation is within ±0.01 Newton-meter; while disturbance occurs, the signal fluctuation can reach ±2 Newton and ±0.1 Newton-meter.

[0134] The wavelet transform uses a discrete wavelet transform method to process the environmental disturbance signal. The Daubechies wavelet basis DB4 is selected because it has a good balance in capturing abrupt signals and smooth changes. The transform order is set to 5, and the corresponding frequency band is divided into: D1 (125-250 Hz), D2 (62.5-125 Hz), D3 (31.25-62.5 Hz), D4 (15.625-31.25 Hz), D5 (7.8125-15.625 Hz), and A5 (0-7.8125 Hz). Each frequency band corresponds to a specific type of disturbance: the D1 band usually contains sensor electronic noise, the D2 and D3 bands correspond to mechanical vibrations and impacts, the D4 and D5 bands reflect object sliding and operator intervention, and the A5 band contains slow environmental changes such as temperature drift.

[0135] The multi-scale coefficients are obtained through the wavelet transform, including the detail coefficients of each level (D1-D5) and the approximation coefficient (A5). The length of each level of detail coefficient is 256, 128, 64, 32, and 16, respectively, and the length of the approximation coefficient is 16. The energy features of each level of coefficient are calculated, including the absolute mean, root mean square, peak value, variance, and kurtosis. Taking a typical sliding disturbance as an example, the root mean square value of the D4 level coefficient is 0.45 Newton, which is significantly higher than the normal state of 0.08 Newton; the kurtosis of the D4 level coefficient is 6.2, indicating that the signal contains a sharp peak feature, corresponding to the mutation at the initial moment of sliding. The energy ratio of adjacent frequency bands is also calculated, such as the D4 / D3 ratio, which usually rises to more than 3 at the beginning of sliding.

[0136] The multi-dimensional disturbance feature vector is composed of the energy features of each frequency band, with a dimension of 40 (6 frequency bands x 5 energy features plus 10 frequency band ratios). To improve computational efficiency, principal component analysis is used to reduce the feature vector to 12 dimensions, retaining more than 95% of the original information. The reduced feature vector is more compact and facilitates subsequent fuzzy rule processing. The physical meaning of each dimension of the feature vector is clear, such as the first dimension mainly reflecting the low-frequency force change amplitude, which is related to the weight change of the object; the second dimension reflects the medium-frequency vibration intensity, which is related to environmental vibration; and the third dimension reflects the high-frequency transient characteristics, which is related to impact and sliding.

[0137] The membership value calculation of the disturbance feature vector adopts fuzzy set theory. Three fuzzy sets are defined for each dimension of the feature vector: low (L), medium (M) and high (H). The membership function adopts a combination of triangle and trapezoidal shapes to ensure the sum of membership values for any input value is 1. The center points and widths of the triangle functions are determined by analyzing historical data. For the first dimension feature (low frequency force variation amplitude), the center point of the low fuzzy set is 0.1, the center point of the medium fuzzy set is 0.5, and the center point of the high fuzzy set is 0.9, with a fuzzy range of ±0.3. There are 36 fuzzy sets in total (12 dimensions x 3 sets), so the dimension of the membership value vector is 36. For a typical vibration disturbance, the value of the second dimension of the feature vector is 0.7, and the corresponding membership values are: low 0.0, medium 0.6, and high 0.4.

[0138] The preset fuzzy rule set contains 25 typical rules, covering the processing strategies of common disturbance types. The rules adopt the IF-THEN format, such as "IF feature 1 is high AND feature 2 is medium AND feature 3 is low THEN clamping force compensation is positive large AND position Z compensation is negative small". The rule set is verified by both expert knowledge and experimental data to ensure the effectiveness and completeness of the rules. The output variables include 7 control quantities: clamping force compensation, position X / Y / Z compensation, and attitude roll / pitch / yaw compensation. Each output variable defines 5 fuzzy sets: negative large (NB), negative small (NS), zero (ZO), positive small (PS), and positive large (PB), corresponding to different degrees of compensation action.

[0139] The rule activation strength is calculated by matching the membership values with the rule conditions. For each rule, the feature dimensions involved in the rule conditions are extracted, and the corresponding membership values are obtained. The combination of multiple conditions adopts fuzzy AND operation, that is, the minimum membership value is taken as the activation strength of the rule. For example, for the rule "IF feature 1 is high (0.8) AND feature 2 is medium (0.6) AND feature 3 is low (0.9) THEN...", the activation strength is 0.6. The activation strengths of all rules form a 25-dimensional vector, reflecting the applicability of each rule.

[0140] The disturbance compensation control quantity is calculated by weighted averaging of the rule activation intensities. First, the output fuzzy set corresponding to each rule is determined; for example, "clamping force compensation is positive" corresponds to an output value of 1.0, and "position Z compensation is negative" corresponds to an output value of -0.3. The fuzzy output of each control quantity is calculated by weighted averaging of the rule activation intensities. If multiple rules point to different output values ​​of the same control quantity, the results are merged based on the activation intensities. Defuzzification uses the centroid method to convert the fuzzy outputs into precise numerical control quantities. The final disturbance compensation control quantity has a dimension of 7×1, corresponding to 7 control variables. For the vibration disturbance example above, the calculated clamping force compensation is +1.5 Newtons, the position Z compensation is -0.8 mm, and other control quantities are relatively small.

[0141] Calculating the modeling error signal requires comparing the system response predicted by the state transition matrix with the actual system response. The system response includes seven state variables: clamping force, position (X / Y / Z), and attitude (roll / pitch / yaw). For the current state vector X(k), the state transition matrix predicts the next state as A×X(k), while the actual measured next state is X(k+1). The difference between the two is the modeling error signal e(k+1), with a dimension of 7×1. To reduce the influence of random noise, a moving average is taken for the error signal over five consecutive frames. Under typical operating conditions, the modeling error of the clamping force is approximately ±0.3 Newtons, the position error is approximately ±0.5 millimeters, and the attitude error is approximately ±1 degree. These errors originate from factors such as variations in the coefficient of friction, sensor drift, and mechanical clearance.

[0142] The L2 norm of the modeling error signal is calculated as the square root of the sum of the squares of its components, reflecting the overall magnitude of the error. For the aforementioned error value, the calculated L2 norm is approximately 1.15. Projection boundary values ​​are set based on the L2 norm using a piecewise function: when the L2 norm is less than the threshold T1 (e.g., 0.5), the boundary value is set to a fixed small value b1 (e.g., 0.2); when the L2 norm is greater than the threshold T2 (e.g., 2.0), the boundary value is set to a fixed large value b2 (e.g., 1.0); when the L2 norm is between T1 and T2, the boundary value is calculated using linear interpolation. This adaptive boundary strategy can maintain control stability when the error is small and provide sufficient compensation capability when the error is large.

[0143] Adaptive dynamic parameters calculation is based on modeling error signal and system state. A sliding surface s is defined as the product of error vector and a preset gain matrix C, which is a diagonal matrix. The diagonal element values are set according to the importance of each state variable, for example, the corresponding element of clamping force is 0.8, the corresponding element of position is 0.5, and the corresponding element of attitude is 0.3. The physical meaning of the sliding surface is the weighted combination of errors, which is the target that the control system strives to approach to zero. The adaptive law uses the sliding surface and the error signal to calculate the dynamic parameter change rate, which is proportional to the product of the sliding surface value and the error vector, and the proportional coefficient γ is set to 0.4, which can be adjusted according to the response speed requirement of the system.

[0144] Adaptive dynamic parameter projection ensures that the parameter change is within a reasonable range. The calculated parameter change rate is compared with the projection boundary value. If it exceeds the boundary, it is truncated to the boundary value; if it is within the boundary, it remains unchanged. The projection operation prevents parameter divergence and improves the stability of the control system. The projected parameter change rate is obtained by integral operation to obtain the adaptive gain matrix. The integral uses the trapezoidal rule, and the integral step is equal to the sampling period of the control system (usually 10 milliseconds). The adaptive gain matrix is 7x7, corresponding to the mutual influence between the 7 state variables. The initial gain matrix is set to a diagonal matrix, and the diagonal element values are set according to experience, for example, the initial value of the corresponding element of clamping force is 0.5, which is continuously updated adaptively as the system runs.

[0145] The error compensation control quantity is calculated by multiplying the modeling error signal and the adaptive gain matrix. The calculation result is a 7x1 vector corresponding to the compensation quantity of the 7 control variables. To prevent overcompensation, set the saturation limit, for example, the clamping force compensation is limited to ±2 Newton, the position compensation is limited to ±2 mm, and the attitude compensation is limited to ±3 degrees. For the above error signal example, assuming that the main diagonal element value of the adaptive gain matrix is [0.6, 0.4, 0.4, 0.4, 0.3, 0.3, 0.3], the calculated error compensation control quantity is: clamping force compensation +0.18 Newton, position X / Y / Z compensation is-0.2 / -0.1 / +0.2 mm, and attitude compensation is about ±0.3 degrees.

[0146] In a third aspect, an electronic device is provided, comprising:

[0147] a processor;

[0148] a memory for storing processor-executable instructions;

[0149] wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0150] In a fourth aspect, the present application provides a computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method described above.

[0151] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for performing various aspects of the present application.

[0152] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A dual-loop coupled adaptive grasping control method integrating robot tactile and visual sensing, characterized in that, include: During the robot's movement, an array of light field cameras is used to acquire images from adjacent viewpoints. Based on these adjacent viewpoint images, the depth map and normal map of the target surface are calculated to obtain surface characteristic data. When the robot reaches the preset grasping position, the sensor array set on the grasping surface acquires the dynamic contact information of the grasping target through piezoelectric and piezoresistive dual-mode response, and extracts stability features from the dynamic contact information based on wavelet transform. Cross-modal coupling is constructed based on Tucker decomposition, and transient response channels and steady-state response channels are constructed based on the surface characteristic data and the stability features. By combining temporal consistency constraints, a recursive optimization algorithm is used to synchronize transient and steady-state response channels and adaptively fuse them based on an attention mechanism to achieve dual-loop coupling of tactile and visual senses. A mapping parameter is established between the dual-loop coupled output and the grasping parameters. An adaptive sliding time window is constructed based on the rate of change of the grasping parameters obtained from the mapping. The time-varying characteristics of the grasping parameters are calculated within each time window. The grasping parameters are optimized using the time-varying characteristics in combination with the Lyapunov stability criterion. The robot's grasping posture and clamping force are adjusted according to the optimized grasping parameters. At the same time, the influence of environmental disturbances and modeling errors is eliminated through a dynamic compensation mechanism to achieve adaptive grasping control.

2. The method according to claim 1, characterized in that, Images from adjacent viewpoints are acquired using a light field camera array. Based on these adjacent viewpoint images, a depth map and normal map of the target surface are calculated to obtain surface characteristic data, including: Extract epipolar pairs from adjacent viewpoint images acquired by the light field camera array, calculate the grayscale difference value of corresponding pixels on the epipolar pairs, select the disparity value corresponding to the smallest grayscale difference value, and obtain a disparity map. The surface depth value is calculated using the disparity map. Based on the surface depth value, the first-order difference and the second-order difference are calculated in the pixel coordinate system. At the same time, the local normal vector is calculated based on the depth difference between adjacent pixels. The direction projection of the local normal vector in the pixel coordinate system is used as the surface orientation parameter. The deviation between the first-order difference and the preset smoothness threshold is used as the depth smoothing constraint. The rate of change between the second-order difference and the surface orientation parameter is used as the normal consistency constraint. The surface depth value and the local normal vector are iteratively adjusted to minimize the deviation between the depth smoothing constraint and the normal consistency constraint, thus obtaining the depth map and the normal map. The depth map and the normal map are used as surface characteristic data of the grasping target.

3. The method according to claim 1, characterized in that, The construction of cross-modal coupling based on Tucker decomposition, and the construction of transient and steady-state response channels based on the surface property data and the stability features, include: The surface characteristic data is decoupled in the spatial dimension to obtain deformation and force components, and the stability features are decoupled in the temporal dimension to obtain transient and steady-state response components. The deformation and transient response components are aligned and combined in the spatial and temporal dimensions, respectively, to obtain a transient feature tensor. The force and steady-state response components are aligned and combined in the spatial and temporal dimensions, respectively, to obtain a steady-state feature tensor. The transient and steady-state feature tensors are decomposed using Tucker decomposition to obtain a transient core tensor and a steady-state core tensor, respectively. A coupling metric matrix is ​​constructed based on the tensor outer product operation of the transient core tensor and the steady-state core tensor. The product result of the transient feature tensor and the coupling metric matrix is ​​used as the transient response channel, and the product result of the steady-state feature tensor and the transpose of the coupling metric matrix is ​​used as the steady-state response channel.

4. The method according to claim 1, characterized in that, A mapping parameter is established between the dual-loop coupled output and the grasping parameters. An adaptive sliding time window is constructed based on the rate of change of the grasping parameters obtained from the mapping. The time-varying characteristics of the grasping parameters are calculated within each time window, including: The difference between the grasping parameters at adjacent sampling times is calculated, and the difference is divided by the sampling time interval to obtain the rate of change of the grasping parameters. An adaptive sliding time window is constructed based on the rate of change. The length of the adaptive sliding time window is obtained by calculating the negative exponent of the rate of change using a preset base window length. Within the adaptive sliding time window, a state transition matrix for the grasping parameters is established. The state estimation gain of the grasping parameters is calculated based on a Kalman filter. The state transition matrix is ​​iteratively updated using the state estimation gain. The eigenvalues ​​of the state transition matrix are calculated. The norm difference between two adjacent iterations of the state transition matrix is ​​statistically analyzed. The time-varying characteristics of the grasping parameters are obtained based on the distribution of the eigenvalues ​​and the changing trend of the norm difference.

5. The method according to claim 1, characterized in that, The grasping parameters are optimized using the time-varying characteristics combined with the Lyapunov stability criterion. The robot's grasping posture and clamping force are adjusted based on the optimized grasping parameters. Simultaneously, a dynamic compensation mechanism is used to eliminate the effects of environmental disturbances and modeling errors, including: The state vector is formed by combining the state transition eigenvalues ​​and norm differences in the time-varying characteristics. A positive definite matrix is ​​constructed based on the state transition eigenvalues. The state vector is multiplied by the positive definite matrix, and then added to the integral term of the state vector to calculate the time derivative. The product of the time derivative and the preset convergence coefficient is less than zero as a stability constraint condition. Under the premise of satisfying the stability constraint, calculate the weighted squared error between the grasping parameters and the preset expected parameter value, calculate the gradient of the weighted squared error with respect to the grasping parameters, and subtract the product of the gradient and the preset learning rate from the grasping parameters to obtain the optimized grasping parameters. The optimized grasping parameters are used to generate basic control quantities, which are then used to adjust the robot's grasping posture and clamping force. The environmental disturbance is decomposed into multidimensional disturbance features through wavelet transform. Based on the multidimensional disturbance features, the disturbance compensation control quantity is obtained by fuzzy rule reasoning. The modeling error signal is calculated and combined with a robust adaptive control strategy to generate the error compensation control quantity. The disturbance compensation control quantity and the error compensation control quantity are superimposed on the basic control quantity.

6. The method according to claim 5, characterized in that, Environmental disturbances are decomposed using wavelet transform to obtain multidimensional disturbance features. Based on these features, fuzzy rule inference is used to obtain the disturbance compensation control quantity. The modeling error signal is calculated, and the error compensation control quantity is generated by combining a robust adaptive control strategy, including: The environmental disturbance is decomposed into multi-scale coefficients through wavelet transform, and the energy distribution of the multi-scale coefficients in different frequency bands is calculated to obtain a multi-dimensional disturbance feature vector. The membership value of the multi-dimensional disturbance feature vector is calculated, and the membership value is matched with a preset fuzzy rule set to obtain the rule activation intensity. The rule activation intensity is weighted and averaged to obtain the disturbance compensation control amount. The deviation between the state transition matrix and the actual system response is calculated to obtain the modeling error signal. The L2 norm of the modeling error signal is calculated and the projection boundary value is set according to the L2 norm. The adaptive dynamic parameters of the modeling error signal are calculated. The adaptive dynamic parameters are projected into the projection boundary value and integrated to obtain the adaptive gain matrix. The modeling error signal is multiplied by the adaptive gain matrix to obtain the error compensation control quantity.

7. A dual-loop coupled adaptive grasping control system integrating robot tactile and visual sensing, used to implement the method as described in any one of claims 1-6, characterized in that, include: The first unit is used to acquire images from adjacent viewpoints using a light field camera array during robot movement, and to calculate the depth map and normal map of the target surface based on the adjacent viewpoint images to obtain surface characteristic data. The second unit is used to obtain dynamic contact information of the grasping target through piezoelectric and piezoresistive dual-mode response of the sensor array set on the grasping surface when the robot reaches the preset grasping position, and extract stability features from the dynamic contact information based on wavelet transform. The third unit is used to construct cross-modal coupling based on Tucker decomposition, and to construct transient response channels and steady-state response channels based on the surface characteristic data and the stability features. By combining temporal consistency constraints, a recursive optimization algorithm is used to synchronize transient and steady-state response channels and adaptively fuse them based on an attention mechanism to achieve dual-loop coupling of tactile and visual senses. The fourth unit is used to establish the mapping parameters between the dual-loop coupled output and the grasping parameters. Based on the rate of change of the grasping parameters obtained by the mapping, an adaptive sliding time window is constructed. Within each time window, the time-varying characteristics of the grasping parameters are calculated. The grasping parameters are optimized using the time-varying characteristics in combination with the Lyapunov stability criterion. The robot's grasping posture and clamping force are adjusted according to the optimized grasping parameters. At the same time, the influence of environmental disturbances and modeling errors is eliminated through a dynamic compensation mechanism to achieve adaptive grasping control.

8. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images