A method and system for online prediction of gripping stability of a flexible robotic arm

Abstract

This invention relates to the field of intelligent gripping control technology, specifically to an online prediction method and system for the gripping stability of a flexible robotic arm. The method involves: first, collecting gripping operation signals through a sensor network in a simulation experimental platform to construct a dataset labeled with the probability of gripping instability; then, using dynamic time warping incorporating physical constraints to align signals of unequal length; subsequently, constructing an online model that includes adaptive extraction and fusion of multimodal features, enhancement of temporal features, dynamic detection of keyframes, and prediction of stability probability; and using mean squared error loss and mini-batch gradient descent to complete model training and optimization; finally, deploying the model to the robotic arm system to achieve real-time data processing and online output of gripping instability probability. This invention can effectively adapt to biochemical vessel gripping scenarios, improve the alignment accuracy of temporal signals and the reliability of risk prediction, and provide real-time assurance for the operational stability of flexible robotic arms.

Description

Technical Field

[0001] This invention relates to the field of intelligent gripping control technology, and in particular to an online prediction method and system for the gripping stability of a flexible robotic arm. Background Technology

[0002] As life sciences, drug development, and other fields move towards high throughput and automation, the automation and intelligentization of biochemistry laboratories have become a clear trend. In typical automated experimental processes, robots need to accurately and reliably manipulate various glassware and experimental consumables, such as test tubes, beakers, and pipette tips. The core foundation of these operations is stable and safe grasping and holding. However, unlike the regular, rigid parts in industrial environments, laboratory glassware is characterized by smooth surfaces (prone to slipping), diverse shapes, and fragile materials (glass). Furthermore, the operating environment often contains interference factors such as liquids and vibrations, placing extremely high demands on the gripping stability of robotic arms. Traditional rigid grippers, lacking compliance and precise force sensing capabilities, are prone to grasping failures or damage to glassware. Therefore, flexible robotic arms equipped with flexible tactile sensor arrays and six-dimensional force sensors, due to their ability to adapt to object shapes and sense contact force information, have become an ideal choice for solving this problem. Despite hardware advancements, enabling robotic arms to "sense" in real time whether the object being gripped is about to slip, just like a human, and make adjustments in advance remains a huge challenge. This is essentially an online prediction and instability warning problem of gripping states.

[0003] Existing technologies objectively suffer from the following shortcomings: Conventional multimodal signal time alignment typically employs simple linear interpolation or conventional dynamic time warping, focusing only on signal waveform similarity and failing to incorporate the physical coupling relationship between sensors, resulting in a mismatch in the alignment results in a physical sense; Conventional multimodal feature fusion often uses early splicing or simple weighted averaging, failing to effectively model the spatial multi-scale characteristics of tactile signals and the nonlinear dynamics of force signals, and also ignoring the differences in the contribution of different modes to the prediction task at different times; Conventional time series processing methods typically process the entire sequence, resulting in low computational efficiency and treating all time steps equally, making them susceptible to interference from a large amount of stationary or noisy data, and making it difficult to focus on and enhance the key moment information representing state transitions; Conventional clamping stability judgments often rely on force / torque thresholding methods based on fixed thresholds or simple classification models, with outputs mostly being discrete "stable / unstable" labels, unable to provide continuous risk probability assessment, and lacking sufficient sensitivity and predictive timeliness for critical instability states.

[0004] Therefore, this invention proposes an online prediction method and system for the gripping stability of a flexible robotic arm to solve the above problems. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes an online prediction method and system for the gripping stability of flexible robotic arms. This invention can effectively adapt to biochemical vessel gripping scenarios, improve the alignment accuracy of timing signals and the reliability of risk prediction, and provide real-time assurance for the operational stability of flexible robotic arms.

[0006] On the one hand, the technical solution of this invention to solve the technical problem is an online prediction method for the gripping stability of a flexible robotic arm, which includes the following steps:

[0007] S1. In the robot experimental platform under simulated environment, a flexible manipulator completes various biochemical vessel clamping tasks. By installing a sensor network at the end of the flexible manipulator to collect operation signal data, the data collected under a complete clamping operation constitutes an original sample. The operation of the original sample is evaluated by the human and the clamping instability probability is labeled. The labeled original samples constitute a dataset.

[0008] S2. For signal data with inconsistent lengths in the original sample, dynamic time warping incorporating physical constraints is adopted to construct a weighted cost function containing signal difference terms and physical constraint terms. The optimal alignment path is solved through dynamic programming to generate the aligned unified time series signal.

[0009] S3. Construct an online prediction model for clamping stability. The model extracts and fuses a temporal fusion feature sequence containing discriminative information from the aligned unified temporal signal through a multimodal feature adaptive extraction and fusion module. Then, it dynamically filters and enhances key frame features through a temporal feature enhancement and key frame dynamic detection module to generate an enhanced key frame feature sequence. Finally, it predicts the clamping instability probability value for each clamping operation task through a clamping stability probability prediction module.

[0010] S4. Calculate the loss function of the model using mean squared error loss. Divide the dataset in S1 into training set, validation set and test set. Train the model using the training set and validation set. Iteratively optimize the model using mini-batch gradient descent algorithm to minimize the loss function and obtain the trained model. Then evaluate the model using the validation set.

[0011] S5. Deploy the trained model into the flexible robotic arm system. After the flexible robotic arm system collects data, the model deployed in the system processes the data and outputs the clamping instability probability value for the corresponding clamping operation task.

[0012] S1 is as follows:

[0013] In the robotic experimental platform that simulates the operating environment of a real biochemical laboratory, a flexible robotic arm completes various biochemical vessel clamping tasks. The clamping tasks include grasping, holding, translating and slightly shaking various laboratory vessels with different forces, angles and speeds, and introducing unstable or critical unstable scenarios such as object sliding, tilting, rotating and about to fall off.

[0014] The sensor network includes tactile sensing arrays and six-dimensional force / torque sensors;

[0015] The tactile sensor array outputs a tactile pressure distribution time-series signal at a fixed frequency. The data format is a three-dimensional array, represented as [time step, tactile array height, tactile array width].

[0016] The six-dimensional force / torque sensor synchronously acquires data at the same or different frequencies to form a force / torque time sequence signal. The data format is a two-dimensional array, represented as [time step, 6].

[0017] Manually label the original samples with the probability of clamping instability based on a comprehensive analysis of the process video recording of each clamping operation, sensor data curves, and the final clamping result. The label value ranges from [0,1], where 0 represents complete stability and 1 represents complete instability. The larger the value between 0 and 1, the higher the risk of instability.

[0018] Each sample in the dataset includes a tactile pressure distribution time series signal, a force / torque time series signal, and a clamping instability probability label.

[0019] S2 is as follows:

[0020] The original samples include original tactile pressure distribution time-series signals and original force / torque time-series signals with inconsistent signal lengths. By finding a mapping function, tactile pressure distribution time-series signals and force / torque time-series signals with uniform lengths are obtained.

[0021] An alignment cost function, weighted by signal difference terms and physical constraint terms, is used to incorporate the physical consistency between sensor signals into time warping by measuring the degree of matching between the time steps of the tactile pressure distribution time series signal and the time steps of the force / torque time series signal.

[0022] The dynamic time warping algorithm is used to find a path that minimizes the cumulative alignment cost over the lengths of the original tactile pressure distribution time-series signal and the original force / torque time-series signal. This path satisfies the boundary conditions, monotonicity and continuity constraints. The goal of minimizing the cumulative cost is to optimize through dynamic programming to obtain the optimal alignment path.

[0023] Based on the optimal alignment path, a mapping relationship is established between the aligned unified time axis and the original signal time step to obtain the aligned unified timing signal, including the aligned tactile pressure distribution timing signal and the aligned force / torque timing signal.

[0024] S3 is as follows:

[0025] S3.1 The multimodal feature adaptive extraction and fusion module adopts an adaptive feature extraction and fusion network composed of multi-scale spatial convolution, nonlinear transformation and cross attention mechanism to automatically learn and fuse temporal fusion feature sequence containing discriminative information from the aligned unified temporal signal;

[0026] S3.2 The temporal feature enhancement and key frame dynamic detection module adopts an adaptive key frame detection and enhancement module. It dynamically filters key time steps by evaluating the feature changes of each time step, and uses a local context attention mechanism to enhance key frame features, generate an enhanced key frame feature sequence, and then sorts it to generate a compressed enhanced key frame feature sequence.

[0027] S3.3 The clamping stability probability prediction module mines global temporal patterns and evolution rules based on compressed enhanced keyframe feature sequences, and obtains a clamping instability probability prediction value for the instability risk of a complete clamping operation task through mapping.

[0028] S3.1 is as follows:

[0029] The aligned unified timing signal includes the aligned tactile pressure distribution timing signal and the aligned force / torque timing signal;

[0030] S3.1.1 For the spatial characteristics of the aligned tactile pressure distribution time-series signal, a parallel multi-scale two-dimensional convolutional kernel is used to simultaneously extract local and global spatial patterns under different receptive fields, generating tactile feature maps obtained by applying convolutional kernels of different scales at each time step. The tactile feature maps of each scale are stitched together in the channel dimension, and the spatial dimension of each tactile feature map is compressed by global average pooling to obtain the tactile spatial features of each aligned time step.

[0031] S3.1.2 For the aligned force / torque time series signal, a nonlinear mapping is performed through a fully connected layer to project it onto a high-dimensional feature space that matches the tactile spatial features, thereby obtaining the high-order nonlinear dynamic features of the force / torque time series signal at each aligned time step;

[0032] S3.1.3 Define learnable query vectors for tactile spatial features and high-order nonlinear dynamic features respectively, and use the query vectors to perform dot product operations with the feature vectors of the corresponding modes respectively, and then calculate the unnormalized importance score of each mode at each time step, where the mode represents two signals;

[0033] The Softmax function is used to normalize the unnormalized importance scores of the two modalities calculated at the same time step, and then the two scores are transformed into probability distributions to obtain normalized attention weights.

[0034] The tactile spatial features and higher-order nonlinear dynamic features are weighted according to the normalized attention weights, and the weighted results are summed to obtain the fusion feature vector of the current time step. Then, the fusion feature vectors of all time steps are arranged in chronological order to form a complete temporal fusion feature sequence.

[0035] S3.2 is as follows:

[0036] S3.2.1 By calculating the difference between the fused feature vectors of adjacent time steps in the temporal fusion feature sequence and combining the learnable parameters for nonlinear mapping, the importance of each time step to the clamping state prediction task is evaluated, and the importance score of each time step is obtained.

[0037] S3.2.2 A dynamic keyframe number determination mechanism combining the base ratio and overall importance level is adopted. A corresponding number of time steps are selected as keyframes based on their importance scores, from highest to lowest. The total number of keyframes is calculated based on the base ratio and the overall importance score of the sequence. Based on the calculated total number of keyframes, the time steps with the highest importance scores are selected from all time steps. Each time step constitutes a set of keyframe indices;

[0038] S3.2.3. An attention mechanism is used to aggregate features within a certain window before and after each keyframe. Specifically, for each selected keyframe index, a local context window is defined with it as the center. By calculating the attention weight, the features of all time steps within the window are aggregated in a weighted manner to generate an enhanced keyframe feature sequence. Then, the enhanced feature vectors of all keyframes are arranged in their original time order to form a compressed enhanced keyframe feature sequence.

[0039] S3.3 is as follows:

[0040] S3.3.1. Perform deep temporal modeling on the compressed enhanced keyframe feature sequence. Use a temporal encoder based on a self-attention mechanism to process the sequence, aggregate the information of all keyframes in the sequence, and output a global feature vector that includes the temporal context of the entire clamping process.

[0041] S3.3.2 Next, the global feature vector is input into a predictor consisting of fully connected layers. The predictor compresses and maps the high-dimensional global time-series features step by step through nonlinear transformation, and finally outputs a single scalar value. Then, the Sigmoid activation function is applied to the two tables to constrain their value range to the interval between 0 and 1, so as to obtain the predicted value of the clamping instability probability.

[0042] S4 is as follows:

[0043] Training employs a mini-batch gradient descent algorithm. In each iteration, samples are randomly sampled from the training set. Through forward propagation, temporal signal alignment preprocessing, multimodal feature extraction and fusion, keyframe enhancement, and instability probability prediction are completed. The mean squared error is used as the loss function, and the model parameters are updated through backpropagation and the optimizer to minimize the error. Training is terminated early based on the validation set error. If there is no performance improvement after multiple consecutive rounds, the training stops and the optimal model is saved. At the same time, a maximum number of training rounds is set as a backup constraint, and finally, the well-trained model is obtained.

[0044] S5 is detailed below:

[0045] The trained model is deployed on a flexible robotic arm system in a biochemistry laboratory for real-time online prediction of gripping stability. During gripping, the flexible robotic arm system collects tactile array pressure image streams and six-dimensional force / torque signals in real time. After adaptive time alignment preprocessing consistent with the training, the data is input into the trained model. The model then undergoes multimodal feature extraction and fusion, temporal feature enhancement, and keyframe dynamic detection. Finally, the prediction module outputs the gripping instability probability and feeds it back to the robot control system or monitoring interface in real time.

[0046] On the other hand, the present invention also provides an online prediction system for the gripping stability of a flexible robotic arm, including a module for executing processing instructions for each step in an online prediction method for the gripping stability of a flexible robotic arm, as follows:

[0047] Data acquisition and annotation module: Used in a simulated environment robot experimental platform to collect biochemical vessel clamping operation signal data through a flexible manipulator end sensor network, form raw samples, and complete manual annotation and dataset construction;

[0048] The timing signal alignment processing module is used to align raw signal data of inconsistent lengths by employing a dynamic time warping method that incorporates physical constraints, constructing a weighted cost function and achieving optimal alignment of timing signals through dynamic programming, thereby generating a unified timing signal.

[0049] The clamping stability online prediction model module includes a multimodal feature adaptive extraction and fusion module, a temporal feature enhancement and keyframe dynamic detection module, and a clamping stability probability prediction module, which are used to complete feature extraction, keyframe screening and enhancement, and instability probability prediction from a unified temporal signal.

[0050] Model training and optimization module: Based on the mean squared error loss function, the dataset is divided into training, validation and test sets, and the mini-batch gradient descent algorithm is used to complete model training, iterative optimization and performance evaluation.

[0051] Online deployment prediction module: used to deploy the trained model to the flexible robotic arm system, process the collected data in real time and output the instability probability value corresponding to the gripping operation online.

[0052] The effects described in the invention are merely those of the embodiments, and not all the effects of the invention. The above technical solutions have the following advantages or beneficial effects:

[0053] This invention discloses an online prediction method and system for the gripping stability of a flexible robotic arm. It employs a dynamic time warping method incorporating physical constraints, embedding the physical law that "the total pressure on the contact surface should be related to the magnitude of the resultant force" as a soft constraint into the multimodal signal alignment process. This overcomes the shortcomings of traditional linear interpolation, which only considers time mapping and ignores physical coupling relationships. A multimodal feature adaptive fusion module based on cross-attention is used to dynamically calculate the attention weight of each modality at each time step through a learnable query vector, achieving a dynamic trade-off and fusion of tactile spatial features and high-order nonlinear features of the force signal. An adaptive keyframe detection and enhancement mechanism is employed, using a learnable network to evaluate the significance of feature changes to dynamically select key time steps, and combining local contextual attention for feature enhancement, effectively compressing sequence redundancy and highlighting key information for instability prediction. An end-to-end prediction framework from the original multimodal signal to the probability of continuous gripping instability is adopted, integrating physical guided alignment, adaptive feature extraction, dynamic key information condensation, and temporal context modeling, achieving a delicate, real-time, and more interpretable evaluation of gripping stability. Attached Figure Description

[0054] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0055] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0056] Figure 2 This is the operation flowchart for step S1.

[0057] Figure 3The following are example data diagrams illustrating the temporal signal sequence of tactile pressure distribution in a flexible robotic arm. Figure (a) shows the data at the initial contact moment, Figure (b) shows the data during the pressure build-up period, Figure (c) shows the data during the stable gripping period, Figure (d) shows the data during the slight slippage period, Figure (e) shows the data during the adjustment and recovery period, and Figure (f) shows the data at the end moment.

[0058] Figure 4 The following are example data graphs illustrating the six-dimensional force / torque time-series signal. Figures (a), (b), and (c) show the data graphs of the three-dimensional force components changing with time on the x, y, and z axes, respectively. Figures (d), (e), and (f) show the data graphs of the three-dimensional torque components changing with time on the x, y, and z axes, respectively.

[0059] Figure 5 This diagram illustrates the accuracy and timeliness of the clamping instability probability of the method of the present invention. Detailed Implementation

[0060] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0061] Example 1

[0062] like Figure 1 As shown, an online prediction method for the gripping stability of a flexible robotic arm includes the following steps:

[0063] S1. In the robot experimental platform under simulated environment, a flexible manipulator completes various biochemical vessel clamping tasks. By installing a sensor network at the end of the flexible manipulator to collect operation signal data, the data collected under a complete clamping operation constitutes an original sample. The operation of the original sample is evaluated by the human and the clamping instability probability is labeled. The labeled original samples constitute a dataset.

[0064] S2. For signal data with inconsistent lengths in the original sample, dynamic time warping incorporating physical constraints is adopted to construct a weighted cost function containing signal difference terms and physical constraint terms. The optimal alignment path is solved through dynamic programming to generate the aligned unified time series signal.

[0065] S3. Construct an online prediction model for clamping stability. The model extracts and fuses a temporal fusion feature sequence containing discriminative information from the aligned unified temporal signal through a multimodal feature adaptive extraction and fusion module. Then, it dynamically filters and enhances key frame features through a temporal feature enhancement and key frame dynamic detection module to generate an enhanced key frame feature sequence. Finally, it predicts the clamping instability probability value for each clamping operation task through a clamping stability probability prediction module.

[0066] S4. Calculate the loss function of the model using mean squared error loss. Divide the dataset in S1 into training set, validation set and test set. Train the model using the training set and validation set. Iteratively optimize the model using mini-batch gradient descent algorithm to minimize the loss function and obtain the trained model. Then evaluate the model using the validation set.

[0067] S5. Deploy the trained model into the flexible robotic arm system. After the flexible robotic arm system collects data, the model deployed in the system processes the data and outputs the clamping instability probability value for the corresponding clamping operation task.

[0068] In specific implementation methods, such as Figure 2 As shown, S1 is as follows:

[0069] S1.1 Collect and construct a high-quality model training dataset, specifically in a robotic experimental platform that simulates the operating environment of a real biochemical laboratory. The end of the flexible manipulator is equipped with a flexible tactile sensor array to capture in real time the spatial distribution of pressure and its dynamic deformation process on the contact surface with the object being held (such as a glass rod, test tube, beaker, etc.). At the same time, a six-dimensional force / torque sensor is integrated into the wrist of the manipulator or the base of the gripper to synchronously measure the three-dimensional force and three-dimensional torque transmitted to the manipulator body during the gripping process.

[0070] During data acquisition, the flexible robotic arm is controlled to perform a variety of gripping tasks, including grasping, holding, translating and slightly shaking various laboratory instruments with different forces, angles and speeds. Various typical instability or critical instability scenarios are deliberately introduced, including objects sliding, tilting, rotating and even about to fall off.

[0071] The tactile sensor array outputs a tactile pressure distribution sequence at a fixed frequency, forming a tactile pressure distribution time-series signal. Its data format is a three-dimensional array, with dimensions represented as [time step, tactile array height, tactile array width]. This signal completely records the dynamic evolution of the contact surface pressure distribution over time during the entire clamping process; for example... Figure 3 The image shows example data illustrating the time-series signal sequence of tactile pressure distribution in a flexible robotic arm, selecting the pressure distribution at six key time points during the gripping process. Figure 3 Each subplot shows the spatial distribution of pressure collected by an 8×8 flexible tactile sensor array at a specific moment. The horizontal axis represents the width direction of the sensor array (0-7 sensing units), the vertical axis represents the height direction of the sensor array (0-7 sensing units), and the color intensity represents the normalized pressure intensity (0 for no pressure, 1 for maximum pressure).

[0072] The six-dimensional force / torque sensor synchronously acquires data at the same or different frequencies, forming a force / torque time sequence signal. The data format is a two-dimensional array with dimensions [time step, 6]. This sequence records, in chronological order, the forces acting on the sensor during the same clamping process. Force on the axis and around Continuous change of torque on the shaft; such as Figure 4 The image shows example data illustrating a six-dimensional force / torque time series signal. Figure 4 The graph contains six sub-graphs corresponding to the changes in three-dimensional force (Fx, Fy, Fz) and three-dimensional torque (Mx, My, Mz) components over time. The horizontal axis of each sub-graph represents the time step, and the vertical axis represents the magnitude of each component (force is measured in Newtons, and torque is measured in Newton-meters). These curves record the continuous changes in force and torque transmitted to the robotic wrist or gripper base during the gripping process. The force components reflect the magnitude and direction of the force exerted by the gripped object on the robotic hand, while the torque components reflect the torque state.

[0073] The raw data collected from each complete clamping experiment constitutes a raw sample, which includes a tactile pressure distribution time-series signal and a force / torque time-series signal.

[0074] S1.2 For each collected sample, data annotation is required to provide a target for model learning. The annotation work is based on a comprehensive analysis of the clamping process video, sensor data curves, and the final clamping result. The label is a scalar value that specifically represents the "clamping instability probability" of that clamping process, and its value ranges from 0 to 1.

[0075] The specific labeling categories and methods are as follows: If the clamping is stable and reliable from beginning to end, with no slippage or unexpected changes in position, it is labeled as 0, indicating complete stability; if the object completely falls off or the container is damaged during clamping, it is labeled as 1, indicating complete instability; for a large number of critical situations in the intermediate state, such as observable slight slippage, slight tilting, or significantly insufficient grip but not yet falling off, a continuous probability value between 0 and 1 is manually assessed based on the severity and risk of instability, such as 0.3, 0.7, etc., with higher values ​​indicating a greater risk or degree of instability. Finally, the format of each training sample is standardized as: a tactile pressure distribution time-series signal, a force / torque time-series signal, and a scalar label representing the probability of clamping instability.

[0076] By collecting and labeling a large amount of data covering various clamping objects, operations, and stable / unstable states, a dataset for model training, validation, and testing is constructed.

[0077] In a specific implementation, S2 is as follows:

[0078] Because the tactile pressure timing signal and the six-dimensional force / torque timing signal of the flexible manipulator are multimodal, high-dimensional, and time-asynchronous, directly fusing them due to differences in physical sampling rates and signal transmission delays of different sensors can lead to a mismatch in the time dimension. Conventional time alignment techniques use linear interpolation methods, which only consider a simple mapping of the time axis and ignore the dynamic coupling relationship between tactile pressure and force / torque signals determined by the physical contact mechanism. Therefore, they cannot accurately capture the fine temporal patterns of the interaction forces during the gripping process. Therefore, this invention employs a dynamic time warping method combined with physical constraints to achieve adaptive high-precision time synchronization between the tactile signal and the force / torque signal. The specific steps are as follows:

[0079] S2.1 Signal Characteristic Analysis and Problem Definition

[0080] The tactile pressure distribution time-series signal is a three-dimensional time-series array, while the force / torque time-series signal is a two-dimensional time-series array. The time lengths of the two are usually different, so it is necessary to find an optimal time mapping relationship to map the original signals onto a unified time axis.

[0081] Specifically, the original tactile pressure distribution time-series signal is defined as follows: The original force / torque timing signal is By finding the mapping function, a uniform length of [missing information] is obtained. Alignment of tactile pressure distribution timing signal and alignment force / torque timing signals ;

[0082] Among them, the original tactile signal It is a three-dimensional array with dimensions of ; This represents the total number of time steps of the original tactile signal; This indicates the number of units in the height direction of the tactile sensor array; This indicates the number of units in the width direction of the tactile sensor array; The original six-dimensional force / torque signal is represented by a two-dimensional array with dimensions of . ; This represents the total number of time steps of the original force / torque signal; Indicates the time step index; This represents the length of the aligned timing signal, and its value is equal to the length of the optimal alignment path.

[0083] S2.2 Calculate the adaptive alignment cost function with introduced physical constraints.

[0084] An alignment cost function, weighted by signal difference terms and physical constraint terms, is used to incorporate the physical consistency between sensor signals into time warping. This is expressed as a measure of the matching degree between the time steps of the tactile pressure distribution time series signal and the force / torque time series signal:

[0085] ,

[0086] In the formula, Indicates the first The time step of the tactile signal and the first The alignment cost function value between time steps of force / torque signals is a scalar. The smaller the value, the more matched the signals are in waveform and physical meaning at these two moments. The time step index representing the original tactile signal is an integer, with a value range of 1. , used to refer to a specific sampling moment in the original tactile signal; The time step index represents the original force / torque signal; it is an integer with a value range of 1. , used to refer to a specific sampling moment in the original force / torque signal; Represents primitive tactile signals In the The pressure distribution matrix at each time step has dimensions of . This characterizes the spatial pressure distribution collected by the tactile sensor array at that moment; Represents the original force / torque signal In the A six-dimensional vector at each time step, the first three components representing the time steps. The force on the axis, the last three components represent the force around the axis. Torque on the shaft; Indicates the tactile signal at the first The local reference pressure distribution matrix at each time step is calculated using a sliding window average, and the calculation method is expressed as follows: It is used to smooth transient noise and provide a stable local reference signal for alignment; The force / torque signal is represented in the first... The local reference vector at each time step is calculated by averaging using a sliding window, and the calculation method is expressed as follows: ; Represents the original force / torque signal In the A six-dimensional vector at each time step; This represents the length of the sliding reference window, which is the number of time steps used to calculate the local reference signal. It is a preset hyperparameter, with an example value of 5. Indicates difference from Time step index; This represents the Frobenius norm, used to measure the overall difference between two pressure distribution matrices; This represents the L2 norm, used to measure the Euclidean distance between two force / moment vectors. This represents the weighting coefficient of the force / torque signal alignment cost term. It is a positive scalar used to adjust the relative importance of the force signal difference term in the overall alignment cost. An example value is 0.5. This represents the weighting coefficient of the physical constraint term. It is a positive scalar used to adjust the strength of the physical consistency constraint. An example value is 2.0. The physical constraint term function is designed based on the physical law that "the total pressure at the contact surface should be related to the magnitude of the resultant force," and its calculation method is expressed as follows: ; Indicates the extraction matrix located at the th line, number Column elements; The L1 norm is used to calculate the sum of the absolute values ​​of the components of the force / torque vector, and approximates the overall magnitude of the resultant torque. The constraints are designed to ensure that the time steps of the matching are consistent in terms of physical magnitude.

[0087] It should be noted that, The term is the signal difference term for tactile signals, the second term. The term represents the signal difference of the force / torque signal, the third term. The term refers to physical constraint terms. The cost function of a conventional dynamic time warping algorithm typically only calculates the Euclidean distance (or a similar distance metric) between corresponding points in two time series, for example... In this invention, the single signal distance metric is expanded into a composite cost function consisting of a weighted sum of signal difference terms (each measuring the smoothness of its own signal relative to a local reference signal) and a physical constraint term. By introducing the physical constraint term, the tactile signal is not required to be numerically directly similar to the force signal, but rather to be similar at the alignment time point. The physical quantities reflected by both (total contact pressure and resultant force amplitude) are consistent in magnitude, enabling the embedding of domain knowledge as a soft constraint into the alignment process. This makes the alignment result not only smooth in time but also reasonable in physical sense, thus enabling a more accurate capture of the dynamic coupling relationship determined by the physical contact mechanism. Item Calculation The sum of pressure sensed by all units in the tactile sensor array at any given moment is an approximate measure of the total pressure on the contact surface. Item Calculation The sum of the absolute values ​​of the components of the six-dimensional force / torque vector at time step 1, using the L1 norm, can serve as a simple and effective approximation of the overall force / torque magnitude. During alignment, the algorithm tends to select the component that minimizes this difference. Pairing forces the aligned moments to match the physical quantities of "total pressure" and "total force amplitude".

[0088] S2.3 Solving the optimal alignment path based on dynamic time warping

[0089] A dynamic time warping algorithm is employed to find a path that minimizes the cumulative alignment cost across the lengths of the original tactile signal and the original force / torque signal. This path satisfies boundary conditions, monotonicity, and continuity constraints. The objective of minimizing the cumulative cost is... The optimal alignment path is obtained by solving this optimization problem using dynamic programming. ;

[0090] in, This represents finding the minimum sum of cumulative costs for all possible alignment paths. An alignment path is represented by a series of matching pairs. The sequence constitutes a correspondence between the time steps of the original tactile signal and the time steps of the force signal; The optimal alignment path is the one obtained through dynamic programming, which is also the path that minimizes the cumulative alignment cost.

[0091] It should be noted that the boundary conditions (path starting point is...) The destination is Monotonicity and The non-decreasing nature of the path and the continuity (the path can only move to adjacent points at a time) are the core constraints defined by the dynamic time warping algorithm itself to achieve reasonable time curvature. In the dynamic programming solution process, the state transition equation guarantees that any legal path must satisfy these constraints.

[0092] S2.4 Generate aligned unified timing signals

[0093] According to the optimal alignment path Establish the mapping relationship between the aligned unified time axis and the original signal time steps to obtain ,and, , ;

[0094] in, Indicates the optimal alignment path Determined, from the alignment time step The mapping function to the time step of the original tactile signal; Indicates based on the mapping function From alignment time step The indexed original tactile signal pressure distribution matrix; Indicates the optimal alignment path Determined, from the alignment time step The mapping function to the time step of the original force / torque signal; Indicates based on the mapping function From alignment time step The indexed original force / torque signal six-dimensional vector; Indicates a unified time step after alignment. The tactile signal at the location is a pressure distribution matrix obtained after time alignment, with dimensions of [missing information]. ; Indicates a unified time step after alignment. The force / torque signal at that location is a six-dimensional vector obtained after time alignment.

[0095] In a specific implementation, S3 is as follows:

[0096] S3.1 Multimodal Feature Adaptive Extraction and Fusion Module

[0097] Because tactile signals contain complex spatial distribution patterns, while force signals have a lower dimension but contain nonlinear dynamic information, and these two modalities are heterogeneous, conventional techniques manually extract spatial and statistical features separately and then simply concatenate them. This fails to effectively model the multi-scale characteristics of tactile space and the high-order nonlinearity of force signals, and ignores the dynamic interaction between the two modal features, resulting in insufficient discriminative power of the fused features. This invention employs an adaptive feature extraction and fusion network composed of multi-scale spatial convolution, nonlinear transformation, and cross-attention mechanisms. It automatically learns and fuses these features from the aligned bimodal signals to obtain a unified temporal feature representation rich in discriminative information. The specific steps are as follows:

[0098] S3.1.1 Spatial Multi-Scale Feature Extraction of Aligned Tactile Pressure Distribution Temporal Signal

[0099] To address the spatial characteristics of the aligned temporal signal of tactile pressure distribution, a parallel multi-scale two-dimensional convolutional kernel is employed to simultaneously extract local and global spatial patterns under different receptive fields, avoiding information loss from single-scale convolution. This is represented as follows:

[0100]

[0101] In the formula, Indicates the time step of uniform alignment Application No. The tactile feature map obtained after convolutional kernels of various scales represents the tactile feature map at time step. Tactile pressure distribution image in the first The local / global spatial patterns extracted at a specific spatial scale contain spatial correlation information within a specific receptive field captured by the convolutional kernel at that scale; This represents the LeakyReLU activation function, which is the activation function of a linear rectifier unit with leakage, used to introduce nonlinear transformation capability; Indicates the first The two-dimensional convolution kernel parameters are at a scale of , with a size of . ,different The values ​​correspond to receptive fields at different spatial scales and are trainable parameters; Indicates the first The height of a two-dimensional convolutional kernel at each scale; Indicates the first The width of a two-dimensional convolutional kernel at each scale; Indices representing different kernel scales in parallel multi-scale convolution. ; Indicates the first The bias parameters corresponding to each scale convolution operation are trainable parameters; Represents a two-dimensional convolution operation; This represents the total number of scales for parallel convolutions. It is a preset hyperparameter, with an example value of 3.

[0102] Then The feature maps at each scale are concatenated along the channel dimension, and the spatial dimension of each feature map is compressed using global average pooling to obtain the tactile spatial feature vector for each aligned time step, represented as:

[0103]

[0104] In the formula, Indicates the first The tactile spatial feature vectors are aligned at each time step, with dimensions of . It preserves the essence of multi-scale spatial information and represents tactile signals at different time steps. Global spatial features; This represents the total dimension of the tactile feature vector, and ; Indicates the time step of uniform alignment Application No. The tactile feature map obtained after convolutional kernels of various scales; This represents the number of feature map channels output by each scale of the convolution, and the number of feature map channels output by each scale of the convolution is set to the same value. This indicates a splicing operation along the channel dimension; This indicates a global average pooling operation.

[0105] S3.1.2 Extraction of high-order nonlinear features from aligned force / torque time-series signals

[0106] For aligned force / torque time-series signals with low dimensionality but important information, a fully connected layer is used for nonlinear mapping to project them into a high-dimensional feature space that matches tactile features, in order to mine their potential high-order dynamic patterns, represented as:

[0107]

[0108] In the formula, Indicates the time step of uniform alignment The extracted force / torque feature vector has a dimension of By using nonlinear transformation, low-dimensional force signals are mapped to a high-dimensional feature space, characterizing the force / torque signal at time steps. High-order nonlinear dynamic characteristics; This represents the weight matrix of the fully connected layer, with dimension 1. Used to map 6-dimensional input to The feature space is a set of trainable parameters; This represents the bias vector of the fully connected layer, with dimension . , are trainable parameters; The dimension representing the force / torque eigenvector needs to be set to ensure consistent dimensionality during fusion. .

[0109] S3.1.3 Dynamic Feature Fusion Based on Cross-Attention

[0110] To adaptively balance the relative importance of tactile features and force / torque features for gripping state prediction at each time step, a cross-attention mechanism is introduced to dynamically calculate fusion weights and generate a unified temporal fusion feature sequence. The specific steps are as follows:

[0111] (1) Calculate the unnormalized importance score

[0112] By defining learnable query vectors for tactile features and force / torque features respectively, and performing dot product operations between these query vectors and the feature vectors of the corresponding modalities, the unnormalized importance score of each modality at each time step is calculated, reflecting the importance of each feature vector to the current fusion task, as expressed as:

[0113]

[0114]

[0115] In the formula, Indicates the first Tactile spatial feature vectors of each aligned time step The unnormalized importance score is a scalar; the larger the value, the more important the tactile features at the corresponding time moment are for predicting the gripping state. The query vector representing tactile features, with dimension . The parameter used to calculate the importance score of the modality feature is a trainable parameter. express transpose; Indicates the time step of uniform alignment Force / torque feature vector extracted at the point The unnormalized importance score is a scalar; the larger the value, the more important the force / torque characteristics at the corresponding time moment are for predicting the clamping state. A query vector representing force / torque features, with dimensions of The parameter used to calculate the importance score of the modality feature is a trainable parameter. express The transpose of .

[0116] (2) Calculate the normalized attention weights

[0117] The Softmax function is used to normalize the tactile feature importance scores and force / torque feature importance scores calculated at the same time step. These two scores are then transformed into a probability distribution to obtain the dynamic attention weights, expressed as follows:

[0118]

[0119]

[0120] In the formula, Indicates the first Tactile spatial feature vectors of each aligned time step The normalized attention weights are scalars between 0 and 1; Indicates the time step of uniform alignment Force / torque feature vector extracted at the point The normalized attention weights are scalars between 0 and 1, and satisfy the following at the same time step: ; This represents the natural exponential function.

[0121] (3) Generate temporal fusion feature sequences

[0122] The tactile feature vector and the force / torque feature vector are weighted according to the normalized attention weights, and the weighted results are summed to obtain the fused feature vector at the current time step. Then, the fused feature vectors of all time steps are arranged in chronological order to form a complete temporal fused feature sequence, represented as:

[0123]

[0124]

[0125] In the formula, Represents a temporal fusion feature sequence with dimension . It is a unified temporal feature that is dynamically fused from tactile and force signals throughout the entire clamping process; Indicates the first The fused feature vector at each time step has a dimension of . Adaptively fuses bimodal information to represent the time step The most important multimodal feature combination for predicting clamping states; The dimension representing the fused feature; since the fused dimensions are consistent, then... .

[0126] S3.2, Temporal Feature Enhancement and Keyframe Dynamic Detection Module

[0127] Because temporal fusion feature sequences contain a large number of redundant time steps with stable states and noise interference, conventional techniques use recurrent neural networks to process the entire sequence. This is not only computationally inefficient but also fails to distinguish between critical and redundant information, making the model susceptible to interference from a large amount of secondary information, thus weakening its ability to capture critical states of clamping instability. This invention employs an adaptive keyframe detection and enhancement module. By evaluating the significance of feature changes at each time step, it dynamically selects key time steps and utilizes a local contextual attention mechanism to enhance the keyframe feature representation, thereby compressing the sequence length and highlighting important information. The specific steps are as follows:

[0128] S3.2.1 Calculate the time series characteristic changes and importance scores

[0129] By calculating the difference between the fused feature vectors of adjacent time steps and combining them with learnable parameters for nonlinear mapping, the importance of each time step to the gripping state prediction task is evaluated. Time steps with drastic feature changes usually correspond to critical moments when the gripping state transitions, as shown below:

[0130]

[0131] In the formula, Indicates the first The importance score of each time step is a scalar, and the larger the value, the more important the time step is for characterizing the key state changes in the clamping process. This represents the weight vector used to calculate importance scores, with dimension 1. , are trainable parameters; This represents the bias scalar used to calculate the importance score, and is a trainable parameter; This represents the Sigmoid activation function, used to map input values ​​to... interval; Indicates the first The change in the fused feature vector at each time step, with dimension [missing information]. It is used to quantify the magnitude of change of features at the current time relative to the previous time, i.e. ; Indicates the first The fused feature vector at each time step, for the start time of the sequence ,definition That is, the zero vector.

[0132] S3.2.2 Dynamically Determine and Select Keyframes

[0133] To avoid the limitations of fixed thresholds or fixed number selections, a dynamic keyframe number determination mechanism combining the base ratio and overall importance level is adopted. A corresponding number of time steps are selected as keyframes based on the importance score from high to low. The total number of keyframes is calculated based on the base ratio and the overall importance score of the sequence, expressed as:

[0134]

[0135] In the formula, The total number of keyframes is determined through a dynamic mechanism, specifically from the temporal fusion feature sequence. The number of important time steps selected from the data; This represents the base scaling hyperparameter, used to ensure a minimum number of keyframe samples. Example values ​​are shown below. ; This represents the score-weighted coefficient hyperparameter, used to dynamically adjust the number of additional keyframes based on the overall importance level of the sequence. Example values ​​are shown below. ; This represents the round-up operator; This represents the floor operation.

[0136] Then, based on the calculated total number of keyframes, the top keyframes with the highest importance scores are selected from all time steps. Each time step constitutes a set of keyframe indices. It contains the time step indices of all selected keyframes:

[0137] in, This indicates the top [number] after sorting by importance score from highest to lowest. A sequence of keyframe indexes, namely .

[0138] S3.2.3 Keyframe Feature Enhancement Based on Local Contextual Attention

[0139] To enrich the feature representation of keyframes and incorporate their local temporal context information, an attention mechanism is used to aggregate features within a certain window before and after each keyframe to generate an enhanced keyframe feature sequence.

[0140] Specifically, for each selected keyframe index A local context window is defined centered on this window. By calculating attention weights, the features of all time steps within the window are weighted and aggregated to enhance the original features of the keyframe, as shown below:

[0141]

[0142] In the formula, Indicates the first The feature vectors enhanced for each keyframe have a dimension of [dimensionality]. Characterized at key time points The keyframe and its neighboring moments together reflect the clamping state characteristics. Through the attention mechanism, it emphasizes contextual information similar to the current key state and weakens irrelevant interference, thus providing a more stable and comprehensive description of the clamping moment represented by the keyframe. This represents the half-width of the context window, i.e., the maximum number of time steps to look forward or backward. The total window length is... Example of a value: ; Indicates in keyframe The offset at is The attention weights corresponding to the context time steps are scalars that measure the impact on the enhancement keyframes. When the feature is defined, the first one in its context window The weights are calculated based on the similarity between keyframe features and context features. Higher similarity results in higher weights, allowing the enhanced features to adaptively focus on the local information most relevant to the keyframe, achieving selective context fusion. The calculation method is expressed as follows: And satisfy ; Represents relative to the center keyframe The offset is an integer, and its value ranges from 1 to 2. ; Indicates the first The fused feature vectors at each time step represent the position of the feature vector relative to the center keyframe within the context window. Offset is The fused feature vector at the location; Indicates the first The fused feature vectors at each time step represent the position of the feature vector relative to the center keyframe within the context window. Offset is The fused feature vector at the location; Indicates difference from Relative to the center keyframe The offset.

[0143] It should be noted that, for boundary cases, when Out of sequence range When, define That is, using zero vectors for padding.

[0144] Then, the feature vectors of all keyframe enhancements are arranged in their original temporal order to form a compressed enhanced keyframe feature sequence. , dimension It is a sequence composed of the enhanced feature vectors of all keyframes arranged in their original temporal order, with a sequence length of... Much smaller than the length of the aligned uniform timing signal However, the feature information density and discriminative power of each time step are significantly improved, representing the condensed temporal features of all key state points and their local dynamic evolution throughout the entire clamping process.

[0145] S3.3 Clamping Stability Probability Prediction Module

[0146] After processing by the temporal feature enhancement and keyframe dynamic detection modules, an enhanced keyframe feature sequence is obtained, which can condense and characterize the key dynamics of the clamping process. The task of the clamping stability probability prediction module is to further mine the global temporal patterns and evolution laws contained in this sequence, and finally map them into a scalar probability value representing the instability risk of the entire clamping task.

[0147] This module first performs deep temporal modeling on the enhanced keyframe feature sequence. Since the enhanced keyframe feature sequence has removed a large number of redundant time steps, but still maintains a strict temporal order, a temporal encoder based on a self-attention mechanism is used to process the sequence. This encoder can adaptively capture the long-range dependencies between any two keyframe features in the sequence, understand the complete dynamic evolution process of the gripping state from the beginning, stable gripping to possible instability, and aggregate the information of all keyframes to output a global feature vector that can comprehensively summarize the temporal context of the entire gripping process.

[0148] Then, the global feature vector is fed into a predictor consisting of fully connected layers. The predictor uses nonlinear transformations to progressively compress and map the high-dimensional global temporal features, ultimately outputting a single scalar value. After this scalar value is output, a sigmoid activation function is applied to constrain its value range to between 0 and 1. The final output value after this constraint is defined as the predicted "clamping instability probability" for this clamping process, directly corresponding to the label in the training data. The closer the value is to 1, the higher the probability that the model predicts instability in the current clamping process (such as object slippage or detachment); the closer the value is to 0, the more stable and reliable the predicted clamping state. Based on this, the model completes the end-to-end mapping from the original multimodal sensing signals to the final clamping stability probability.

[0149] It should be noted that self-attention-based temporal encoders are a mature existing technology. Their core idea originates from the Transformer model architecture proposed by Google in 2017. In one implementation, the Transformer's Encoder applies a "self-attention mechanism" to encode sequential data, specifically the input enhanced keyframe feature sequence. The model calculates a set of "queries," "keys," and "values" for each feature vector. By calculating the similarity (attention score) between the "query" and all "keys," it determines which parts (which "values") of the sequence should be extracted and how much information should be extracted when summarizing information. In this implementation, the standard temporal encoder (Transformer's Encoder) includes input embedding and position encoding, multi-head self-attention layers, feedforward neural network layers, residual connections, and layer normalization, which are stacked in sequence. After several layers (e.g., 2 or 4 layers) of the above structure, each layer allows the information in the feature sequence to be more fully interacted and refined. The encoder finally outputs an "encoded keyframe feature sequence" that is identical to the input sequence. The features at each time step contain the contextual information of the entire sequence. Then, pooling operations (e.g., taking the features of the first time step of the sequence, or averaging the features of all time steps) are used to aggregate this sequence into a single, fixed-length "global feature vector," which is then fed into the predictor to generate the final probability value.

[0150] In a specific implementation, S4 is as follows:

[0151] The completed online prediction model for clamping stability needs to be trained to learn all its internal trainable parameters. The training process uses the dataset collected and labeled in step S1.

[0152] First, the dataset is divided into non-overlapping training, validation, and test sets. The training set is used to directly optimize model parameters, the validation set is used to monitor model performance and tune hyperparameters during training, and the test set is used to finally evaluate the model's generalization ability. During training, a mini-batch gradient descent algorithm is used for iterative optimization.

[0153] In each iteration, a batch of samples is randomly sampled from the training set. Each sample contains the original tactile signal before alignment, the force / torque signal, and the corresponding ground truth label of the clamping instability probability. The forward propagation process of the model strictly follows the steps described in S2 and S3: first, the original signal of each sample in the batch is preprocessed by adaptive temporal alignment; then, it is passed sequentially through the multimodal feature adaptive extraction and fusion module, the temporal feature enhancement and keyframe dynamic detection module, and finally, the clamping stability probability prediction module outputs the predicted probability value of the sample.

[0154] The mean squared error loss function is chosen to calculate the average squared error between the predicted probability values ​​and the true probability labels for all samples in the batch. Then, the gradient of the loss function with respect to all model parameters is calculated using the backpropagation algorithm, and an optimizer (such as Adam) is used to update the parameters based on the gradient to minimize the loss function.

[0155] The training process will continue with multiple iterations. The criterion for stopping iterations is mainly based on performance monitoring on the validation set: when the model's prediction error (loss) on the validation set no longer decreases within several consecutive training epochs (i.e., a preset patience value), or even begins to increase, it is considered that the model performance has reached saturation and overfitting may have occurred. At this point, training is stopped early, and the model parameters corresponding to the best performance on the validation set are saved as the final trained model. In addition, a maximum number of training epochs can be set as a backup stopping condition to prevent infinite training loops.

[0156] In a specific implementation, S5 is as follows:

[0157] Once the online prediction model for gripping stability has been trained, it can be deployed in a flexible robotic arm system in a real biochemistry laboratory to achieve real-time, online prediction of gripping stability.

[0158] During the online prediction phase, the system operates continuously in a pipeline manner of "data acquisition-processing-prediction," with the specific process as follows:

[0159] During the gripping task performed by the flexible robotic arm (such as grasping a test tube), the system synchronously acquires pressure distribution image streams from the flexible tactile array and reading streams from the six-dimensional force / torque sensor in real time, forming a real-time sensing signal segment of the current gripping process. First, preprocessing and time alignment are performed on this real-time signal segment. The same adaptive time alignment operation as in the training phase is applied to generate a time-synchronized tactile signal sequence and force / torque signal sequence F. Then, this aligned sequence is input into a pre-trained prediction model. The model's internal multimodal feature adaptive extraction and fusion module automatically extracts spatial and higher-order force features and dynamically fuses them. The temporal feature enhancement and keyframe dynamic detection module compresses and enhances the fused temporal features, focusing on critical state moments. Finally, the gripping stability probability prediction module outputs a current "gripping instability probability" prediction value based on the enhanced keyframe sequence. This prediction value is fed back to the robot control system or monitoring interface in real time.

[0160] Operators or upper-level control algorithms can make decisions based on this probability value: when the predicted probability value is below a certain safety threshold (e.g., 0.3), the gripping is considered stable, and subsequent operations (such as pipetting or stirring) can continue; when the probability value exceeds a warning threshold (e.g., 0.4), it indicates a high risk of instability, and the system can immediately trigger an alarm or automatically adjust the gripping force and attitude to re-stabilize the grip, thereby achieving online monitoring and instability warning of the gripping state, greatly improving the safety and reliability of automated experimental operations. The entire online prediction process is efficient and automated, meeting the real-time requirements of laboratory robots.

[0161] Example 2

[0162] A flexible robotic arm gripping stability online prediction system includes modules for executing processing instructions for each step of a flexible robotic arm gripping stability online prediction method, as follows:

[0163] Data acquisition and annotation module: Used in a simulated environment robot experimental platform to collect biochemical vessel clamping operation signal data through a flexible manipulator end sensor network, form raw samples, and complete manual annotation and dataset construction;

[0164] The timing signal alignment processing module is used to align raw signal data of inconsistent lengths by employing a dynamic time warping method that incorporates physical constraints, constructing a weighted cost function and achieving optimal alignment of timing signals through dynamic programming, thereby generating a unified timing signal.

[0165] The clamping stability online prediction model module includes a multimodal feature adaptive extraction and fusion module, a temporal feature enhancement and keyframe dynamic detection module, and a clamping stability probability prediction module, which are used to complete feature extraction, keyframe screening and enhancement, and instability probability prediction from a unified temporal signal.

[0166] Model training and optimization module: Based on the mean squared error loss function, the dataset is divided into training, validation and test sets, and the mini-batch gradient descent algorithm is used to complete model training, iterative optimization and performance evaluation.

[0167] Online deployment prediction module: used to deploy the trained model to the flexible robotic arm system, process the collected data in real time and output the instability probability value corresponding to the gripping operation online.

[0168] Example 3

[0169] To verify the accuracy and timeliness of the method of the present invention in predicting the instability probability in real time during actual clamping, such as... Figure 5 As shown, an experiment was conducted to simulate a complete clamping operation, lasting 10 seconds, with the horizontal axis in seconds. Three typical instability events were designed during the clamping process: slight slippage around the 2nd-3rd second, significant slippage around the 5th second, and a near-detachment critical state around the 8th second. The experiment compared the performance of the conventional fixed-threshold keyframe detection method with the adaptive keyframe detection and enhancement method of this invention in real-time prediction of clamping instability probability. Conventional methods use fixed time intervals or a fixed number of keyframes for feature extraction, while the method of this invention dynamically selects key time steps by evaluating the significance of feature changes at each time step and utilizes a local contextual attention mechanism to enhance the keyframe feature representation. Figure 5 The vertical axis represents the probability of clamping instability; a higher value indicates greater instability. The actual curve is compared with the predicted curves of the two methods, and the vertical axis region is divided into three parts: 0 to 0.3 (green background, stable region), 0.3 to 0.7 (orange background, warning region), and 0.7 to 1.0 (red background, danger region). Experimental results show that the predicted curve (blue solid line) of the method of this invention is closest to the actual instability probability curve (black solid line), accurately and timely capturing the occurrence of three instability events, and the trend of the predicted value is highly consistent with the actual curve. The predicted curve of the conventional method (red dashed line) exhibits significant lag, with a large response delay to instability events, and large fluctuations in the predicted value in the stable and warning regions, showing a high level of noise. The adaptive keyframe detection and enhancement module of the method of this invention can effectively compress redundant time steps, highlight important information, and accurately capture the key states of clamping instability, thus providing more timely and accurate real-time prediction capabilities.

[0170] Although the specific embodiments of the invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the invention. Based on the technical solutions of the invention, various modifications or variations that can be made by those skilled in the art without creative effort are still within the scope of protection of the invention.

Claims

1. An online prediction method for the gripping stability of a flexible robotic arm, characterized in that, Includes the following steps: S1. In the robot experimental platform under simulated environment, a flexible manipulator completes various biochemical vessel clamping tasks. By installing a sensor network at the end of the flexible manipulator to collect operation signal data, the data collected under a complete clamping operation constitutes an original sample. The operation of the original sample is evaluated by the human and the clamping instability probability is labeled. The labeled original samples constitute a dataset. S2. For signal data with inconsistent lengths in the original sample, dynamic time warping incorporating physical constraints is adopted to construct a weighted cost function containing signal difference terms and physical constraint terms. The optimal alignment path is solved through dynamic programming to generate the aligned unified time series signal. S3. Construct an online prediction model for clamping stability. The model extracts and fuses a temporal fusion feature sequence containing discriminative information from the aligned unified temporal signal through a multimodal feature adaptive extraction and fusion module. Then, it dynamically filters and enhances key frame features through a temporal feature enhancement and key frame dynamic detection module to generate an enhanced key frame feature sequence. Finally, it predicts the clamping instability probability value for each clamping operation task through a clamping stability probability prediction module. S4. Calculate the loss function of the model using mean squared error loss. Divide the dataset in S1 into training set, validation set and test set. Train the model using the training set and validation set. Iteratively optimize the model using mini-batch gradient descent algorithm to minimize the loss function and obtain the trained model. Then evaluate the model using the validation set. S5. Deploy the trained model into the flexible robotic arm system. After the flexible robotic arm system collects data, the model deployed in the system processes the data and outputs the clamping instability probability value for the corresponding clamping operation task.

2. The online prediction method for gripping stability of a flexible robotic arm according to claim 1, characterized in that, S1 is as follows: In the robotic experimental platform that simulates the operating environment of a real biochemical laboratory, a flexible robotic arm completes various biochemical vessel clamping tasks. The clamping tasks include grasping, holding, translating and slightly shaking various laboratory vessels with different forces, angles and speeds, and introducing unstable or critical unstable scenarios such as object sliding, tilting, rotating and about to fall off. The sensor network includes tactile sensing arrays and six-dimensional force / torque sensors; The tactile sensor array outputs a tactile pressure distribution time-series signal at a fixed frequency. The data format is a three-dimensional array, represented as [time step, tactile array height, tactile array width]. The six-dimensional force / torque sensor synchronously acquires data at the same or different frequencies to form a force / torque time sequence signal. The data format is a two-dimensional array, represented as [time step, 6]. Manually label the original samples with the probability of clamping instability based on a comprehensive analysis of the process video recording of each clamping operation, sensor data curves, and the final clamping result. The label value ranges from [0,1], where 0 represents complete stability and 1 represents complete instability. The larger the value between 0 and 1, the higher the risk of instability. Each sample in the dataset includes a tactile pressure distribution time series signal, a force / torque time series signal, and a clamping instability probability label.

3. The online prediction method for gripping stability of a flexible robotic arm according to claim 1, characterized in that, S2 is as follows: The original samples include original tactile pressure distribution time-series signals and original force / torque time-series signals with inconsistent signal lengths. By finding a mapping function, tactile pressure distribution time-series signals and force / torque time-series signals with uniform lengths are obtained. An alignment cost function, weighted by signal difference terms and physical constraint terms, is used to incorporate the physical consistency between sensor signals into time warping by measuring the degree of matching between the time steps of the tactile pressure distribution time series signal and the time steps of the force / torque time series signal. The dynamic time warping algorithm is used to find a path that minimizes the cumulative alignment cost over the lengths of the original tactile pressure distribution time-series signal and the original force / torque time-series signal. This path satisfies the boundary conditions, monotonicity and continuity constraints. The goal of minimizing the cumulative cost is to optimize through dynamic programming to obtain the optimal alignment path. Based on the optimal alignment path, a mapping relationship is established between the aligned unified time axis and the original signal time step to obtain the aligned unified timing signal, including the aligned tactile pressure distribution timing signal and the aligned force / torque timing signal.

4. The online prediction method for gripping stability of a flexible robotic arm according to claim 1, characterized in that, S3 Specifically as follows: S3.1 The multimodal feature adaptive extraction and fusion module adopts an adaptive feature extraction and fusion network composed of multi-scale spatial convolution, nonlinear transformation and cross attention mechanism to automatically learn and fuse temporal fusion feature sequence containing discriminative information from the aligned unified temporal signal; S3.2 The temporal feature enhancement and key frame dynamic detection module adopts an adaptive key frame detection and enhancement module. It dynamically filters key time steps by evaluating the feature changes of each time step, and uses a local context attention mechanism to enhance key frame features, generate an enhanced key frame feature sequence, and then sorts it to generate a compressed enhanced key frame feature sequence. S3.3 The clamping stability probability prediction module mines global temporal patterns and evolution rules based on compressed enhanced keyframe feature sequences, and obtains a clamping instability probability prediction value for the instability risk of a complete clamping operation task through mapping.

5. The online prediction method for gripping stability of a flexible robotic arm according to claim 4, characterized in that, S3.1 is as follows: The aligned unified timing signal includes the aligned tactile pressure distribution timing signal and the aligned force / torque timing signal; S3.1.1 For the spatial characteristics of the aligned tactile pressure distribution time-series signal, a parallel multi-scale two-dimensional convolutional kernel is used to simultaneously extract local and global spatial patterns under different receptive fields, generating tactile feature maps obtained by applying convolutional kernels of different scales at each time step. The tactile feature maps of each scale are stitched together in the channel dimension, and the spatial dimension of each tactile feature map is compressed by global average pooling to obtain the tactile spatial features of each aligned time step. S3.1.2 For the aligned force / torque time series signal, a nonlinear mapping is performed through a fully connected layer to project it onto a high-dimensional feature space that matches the tactile spatial features, thereby obtaining the high-order nonlinear dynamic features of the force / torque time series signal at each aligned time step; S3.1.3 Define learnable query vectors for tactile spatial features and high-order nonlinear dynamic features respectively, and use the query vectors to perform dot product operations with the feature vectors of the corresponding modes respectively, and then calculate the unnormalized importance score of each mode at each time step, where the mode represents two signals; The Softmax function is used to normalize the unnormalized importance scores of the two modalities calculated at the same time step, and then the two scores are transformed into probability distributions to obtain normalized attention weights. The tactile spatial features and higher-order nonlinear dynamic features are weighted according to the normalized attention weights, and the weighted results are summed to obtain the fusion feature vector of the current time step. Then, the fusion feature vectors of all time steps are arranged in chronological order to form a complete temporal fusion feature sequence.

6. The online prediction method for gripping stability of a flexible robotic arm according to claim 4, characterized in that, S3.2 is as follows: S3.2.1 By calculating the difference between the fused feature vectors of adjacent time steps in the temporal fusion feature sequence and combining the learnable parameters for nonlinear mapping, the importance of each time step to the clamping state prediction task is evaluated, and the importance score of each time step is obtained. S3.2.2 A dynamic keyframe number determination mechanism combining the base ratio and overall importance level is adopted. A corresponding number of time steps are selected as keyframes based on their importance scores, from highest to lowest. The total number of keyframes is calculated based on the base ratio and the overall importance score of the sequence. Based on the calculated total number of keyframes, the time steps with the highest importance scores are selected from all time steps. Each time step constitutes a set of keyframe indices; S3.2.

3. An attention mechanism is used to aggregate features within a certain window before and after each keyframe. Specifically, for each selected keyframe index, a local context window is defined with it as the center. By calculating the attention weight, the features of all time steps within the window are aggregated in a weighted manner to generate an enhanced keyframe feature sequence. Then, the enhanced feature vectors of all keyframes are arranged in their original time order to form a compressed enhanced keyframe feature sequence.

7. The online prediction method for gripping stability of a flexible robotic arm according to claim 4, characterized in that, S3.3 is as follows: S3.3.

1. Perform deep temporal modeling on the compressed enhanced keyframe feature sequence. Use a temporal encoder based on a self-attention mechanism to process the sequence, aggregate the information of all keyframes in the sequence, and output a global feature vector that includes the temporal context of the entire clamping process. S3.3.2 Next, the global feature vector is input into a predictor consisting of fully connected layers. The predictor compresses and maps the high-dimensional global time-series features step by step through nonlinear transformation, and finally outputs a single scalar value. Then, the Sigmoid activation function is applied to the two tables to constrain their value range to the interval between 0 and 1, so as to obtain the predicted value of the clamping instability probability.

8. The online prediction method for gripping stability of a flexible robotic arm according to claim 1, characterized in that, S4 is as follows: Training employs a mini-batch gradient descent algorithm. In each iteration, samples are randomly sampled from the training set. Through forward propagation, temporal signal alignment preprocessing, multimodal feature extraction and fusion, keyframe enhancement, and instability probability prediction are completed. The mean squared error is used as the loss function, and the model parameters are updated through backpropagation and the optimizer to minimize the error. Training is terminated early based on the validation set error. If there is no performance improvement after multiple consecutive rounds, the training stops and the optimal model is saved. At the same time, a maximum number of training rounds is set as a backup constraint, and finally, the well-trained model is obtained.

9. The online prediction method for gripping stability of a flexible robotic arm according to claim 1, characterized in that, S5 is detailed below: The trained model is deployed on a flexible robotic arm system in a biochemistry laboratory for real-time online prediction of gripping stability. During gripping, the flexible robotic arm system collects tactile array pressure image streams and six-dimensional force / torque signals in real time. After adaptive time alignment preprocessing consistent with the training, the data is input into the trained model. The model then undergoes multimodal feature extraction and fusion, temporal feature enhancement, and keyframe dynamic detection. Finally, the prediction module outputs the gripping instability probability and feeds it back to the robot control system or monitoring interface in real time.

10. An online prediction system for the gripping stability of a flexible robotic arm, used to execute the online prediction method for the gripping stability of a flexible robotic arm as described in any one of claims 1 to 9, characterized in that, Includes the following modules: Data acquisition and annotation module: Used in a simulated environment robot experimental platform to collect biochemical vessel clamping operation signal data through a flexible manipulator end sensor network, form raw samples, and complete manual annotation and dataset construction; The timing signal alignment processing module is used to align raw signal data of inconsistent lengths by employing a dynamic time warping method that incorporates physical constraints, constructing a weighted cost function and achieving optimal alignment of timing signals through dynamic programming, thereby generating a unified timing signal. The clamping stability online prediction model module includes a multimodal feature adaptive extraction and fusion module, a temporal feature enhancement and keyframe dynamic detection module, and a clamping stability probability prediction module, which are used to complete feature extraction, keyframe screening and enhancement, and instability probability prediction from a unified temporal signal. Model training and optimization module: Based on the mean squared error loss function, the dataset is divided into training, validation and test sets, and the mini-batch gradient descent algorithm is used to complete model training, iterative optimization and performance evaluation. Online deployment prediction module: used to deploy the trained model to the flexible robotic arm system, process the collected data in real time and output the instability probability value corresponding to the gripping operation online.

Images