Industrial robot adaptive control method and system based on multi-modal sensor fusion

Abstract

The invention relates to the technical field of robot control, and discloses an industrial robot adaptive control method and system based on multi-modal sensor fusion, and the method comprises the steps: collecting multi-modal original data, and carrying out the time-space alignment; capturing space-time semantic association of visual textures, tactile pressure distribution and force sense fluctuation in the multi-modal data through a multi-head attention mechanism guided by a physical model, and performing space-time registration; a CNN-LSTM hybrid model is adopted to extract visual texture features and time sequence tactile features in the physical information enhanced multi-modal feature matrix; and carrying out dynamic weight distribution on the fusion feature vectors with physical consistency by utilizing a weight distribution model driven by element reinforcement learning to generate dynamic weighted fusion features. According to the method, the spatial positioning precision of the industrial robot in a precise assembly scene is greatly improved, the contact force control stability is greatly improved, and the control robustness in a complex operation scene is remarkably enhanced.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to an adaptive control method and system for industrial robots based on multimodal sensor fusion. Background Technology

[0002] In the current field of industrial robot control, multimodal sensor fusion technology has become a key research direction for improving the robot's environmental perception and adaptive capabilities. Traditional industrial robots mostly rely on single-modal sensors for basic operations, but with the increasing demands of complex work scenarios, the limitations of single-modal perception are becoming increasingly apparent. In recent years, multimodal fusion technology has made significant progress in improving the spatial positioning accuracy and contact force control stability of robots by integrating data from heterogeneous sensors such as vision, touch, and force sensors. For example, data-level fusion achieves preliminary information integration by directly stitching together multi-source data, while feature-level fusion utilizes models such as CNN and LSTM to extract cross-modal shared features, forming a more robust perceptual representation. These methods have already been commercially applied in standardized work scenarios, such as vision-force collaborative control in automotive parts assembly lines.

[0003] However, existing technologies have the following drawbacks: First, the spatiotemporal semantic association is weak. Most methods only achieve simple data or feature splicing without deeply exploring the deep semantic associations of visual texture, tactile pressure distribution, and force fluctuations in the spatiotemporal dimension, making it difficult for fused features to reflect real physical constraints. Second, the weight allocation mechanism is rigid. Existing systems mostly use static weights or weight parameters trained offline, which cannot dynamically adjust the weights of each modality according to real-time environmental parameters, making them prone to control deviations in scenarios with sudden environmental changes. Third, the adaptive capability is isolated. Existing methods are mainly optimized for specific tasks of a single robot and lack an effective mechanism for knowledge sharing and co-evolution among multiple robots, resulting in the need for a lot of repeated debugging when facing new tasks, and failing to achieve the accumulation and rapid deployment of collective intelligence.

[0004] No effective solutions have yet been proposed to address the problems in the relevant technologies. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an adaptive control method and system for industrial robots based on multimodal sensor fusion. It has the advantages of deep semantic fusion guided by physical models, rapid adaptation driven by meta-reinforcement learning, and cross-platform collaborative knowledge transfer. This addresses the core problems of existing multimodal fusion technologies, such as weak physical correlation of features and poor adaptability to environmental changes, which make it difficult to meet the needs of flexible manufacturing.

[0006] (II) Technical Solution To achieve the advantages of deep semantic fusion guided by physical models, rapid adaptation driven by meta-reinforcement learning, and cross-platform collaborative knowledge transfer, the specific technical solution adopted in this invention is as follows: Firstly, an adaptive control method for industrial robots based on multimodal sensor fusion is provided, including: S1. Collect multimodal raw data and perform spatiotemporal alignment to obtain the spatiotemporally aligned multimodal data basis; S2. Based on a multimodal data base, a multi-head attention mechanism guided by a physical model is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution and force fluctuation in multimodal data, and spatiotemporal registration is performed to obtain a multimodal feature matrix with enhanced physical information. S3. A CNN-LSTM hybrid model is used to extract visual texture features and temporal tactile features from the multimodal feature matrix with enhanced physical information. Feature-level fusion is performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency. S4. Obtain real-time environmental parameters and use a meta-reinforcement learning-driven weight allocation model to dynamically allocate weights to the fusion feature vectors with physical consistency, generating dynamically weighted fusion features. S5. A dual-mode control strategy switching mechanism is constructed based on fuzzy PID and model predictive control. The dual-mode control strategy switching mechanism generates mode switching signals and control calculation requirements based on dynamic weighted fusion features. S6. Based on the dynamic weighted fusion characteristics and control computation requirements, select the optimal control strategy for the task through a pre-built control algorithm library and generate the corresponding control instructions. S7. Use control commands to drive the robot to perform tasks, synchronously collect control process data of the tasks, and encapsulate it into a shared standardized experience data package to realize knowledge transfer across robot platforms.

[0007] Furthermore, based on a multimodal data base, a multi-head attention mechanism guided by a physical model is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution, and force wave in the multimodal data, and spatiotemporal registration is performed to obtain a multimodal feature matrix enhanced with physical information. This includes: S201, preprocessing the visual data in the multimodal data base to extract visual texture feature maps; S202, preprocessing the tactile data in the multimodal data base to extract tactile pressure distribution feature maps; S203, preprocessing the force data in the multimodal data base to extract force wave feature sequences; and S204, generating spatial physics based on the robot kinematics model and prior physical knowledge of the task. S205. Associating the bias matrix with the temporal physical constraint sequence; combining the visual texture feature map, tactile pressure distribution feature map, and force wave feature sequence with the spatial physical association bias matrix and the temporal physical constraint sequence, inputting them into a multi-head attention mechanism model guided by the physical model, calculating the spatiotemporal attention weights between different modalities, capturing the semantic association between visual texture, tactile pressure distribution, and force wave, and obtaining the global attention weight matrix; S206. Based on the global attention weight matrix, performing feature fusion and spatiotemporal registration on the visual texture feature map, tactile pressure distribution feature map, and force wave feature sequence, adjusting the alignment of features in the temporal and spatial dimensions, and generating a multimodal feature matrix with enhanced physical information.

[0008] Furthermore, based on the robot's kinematic model and prior physical knowledge of the task, a spatial physical correlation bias matrix and a temporal physical constraint sequence are generated, including: S2041, based on the robot's forward kinematics model and the calibration parameters of the vision camera, calculating the expected contact area between the robot's end effector and the task in the visual data, and generating a spatial physical correlation bias matrix to enhance the attention weight of visual-tactile features within the expected contact area; S2042, based on the mechanical model of the task and Coulomb's law of friction, deriving the expected fluctuation frequency and amplitude range of the force signal, and generating a temporal physical constraint sequence, which is used to constrain the distribution of temporal attention weights to conform to physical laws.

[0009] Furthermore, a CNN-LSTM hybrid model is used to extract visual texture features and temporal tactile features from the physically enhanced multimodal feature matrix. Feature-level fusion is performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency, including: S301, using a convolutional neural network model to process the visual part features in the physically enhanced multimodal feature matrix and extracting visual texture features; S302, using a long short-term memory model to process the tactile part features in the physically enhanced multimodal feature matrix and extracting temporal tactile features; S303, performing feature-level fusion of visual texture features and temporal tactile features through a cross-modal attention mechanism, calculating the attention weights between different modalities, and generating a fused feature vector with physical consistency; S304, calculating the consistency score between the fused feature vector and the rules in the pre-built physical constraint library. If the score is lower than a preset threshold, the spatial physical association bias matrix and the temporal physical constraint sequence are recalculated until the consistency score is greater than the preset threshold.

[0010] Furthermore, real-time environmental parameters are acquired, and a meta-reinforcement learning-driven weight allocation model is used to dynamically allocate weights to the physically consistent fusion feature vectors to generate dynamically weighted fusion features. This process includes: S401, acquiring real-time environmental parameters and extracting real-time environmental feature vectors; S402, constructing a meta-policy network based on the meta-reinforcement learning framework, obtaining initialization parameters adapted to new tasks through learning in a multi-task training set; the initialization parameters of the meta-policy network can be loaded from a collaborative knowledge base; S403, constructing a weight allocation model, which takes the real-time environmental feature vectors and physically consistent fusion feature vectors as input states, and uses the meta-policy network for parameter initialization and fine-tuning, generating real-time weight coefficients corresponding to each modality data through the mapping relationship between the state space and the action space; S404, calculating the basic weights based on the confidence scores of each modality data in the physically consistent fusion feature vectors, and performing weighted summation in combination with the real-time weight coefficients to generate dynamically weighted fusion features. The meta-policy network, constructed based on a meta-reinforcement learning framework, acquires initialization parameters adapted to new tasks through learning in a multi-task training set, including: S4021, constructing a meta-training task set containing various industrial robot tasks, each task corresponding to different environmental conditions, workpiece objects, and control objectives; S4022, during the meta-training phase of the meta-policy network, optimizing the network parameters using a model-independent meta-learning algorithm, and obtaining a new weight allocation strategy through gradient updates when the network faces a new task from the task set; S4023, during the deployment phase, when a new task is discovered, fine-tuning the network parameters based on online-collected interactive data to achieve adaptive adjustment of the weight allocation strategy.

[0011] Furthermore, a dual-modal control strategy switching mechanism is constructed based on fuzzy PID and model predictive control. This mechanism generates mode switching signals and control computation requirements based on dynamically weighted fusion features, including: S501, performing multi-dimensional analysis of the dynamically weighted fusion features to extract task type identifiers, environmental sensitivity indicators, and modal confidence weights, and generating mode switching signals through pattern recognition; S502, when the mode switching signal triggers the fuzzy PID control mode, dynamically adjusting the PID parameter library based on the real-time weight coefficients in the dynamically weighted fusion features to generate control computation requirements; S503, when the mode switching signal triggers the model predictive control mode, constructing a state-space model based on the dynamically weighted fusion features. This state-space model integrates visual spatial constraints, tactile contact constraints, and force-mechanical constraints derived from the multimodal feature matrix enhanced by physical information, and generates an optimized control sequence through a quadratic programming solver.

[0012] Furthermore, based on the dynamic weighted fusion characteristics and control computation requirements, the optimal control strategy for the task is selected through a pre-built control algorithm library, and corresponding control instructions are generated, including: S601, based on the dynamic weighted fusion characteristics and control computation requirements, the fuzzy PID algorithm or MPC algorithm in the pre-built control algorithm library is matched with the pattern identifier to generate an initial control strategy framework; S602, the dynamic weighted fusion characteristics are analyzed in multiple dimensions to extract the spatial constraint parameters of visual texture features, the temporal fluctuation coefficient of tactile pressure distribution, and the mechanical stability index of force fluctuation, and these parameters are aligned with the constraints in the control computation requirements to form an optimized input vector; S603, based on the initial control strategy framework and the optimized input vector, control instructions are generated through a fuzzy inference engine or a quadratic programming solver.

[0013] Furthermore, by using control commands to drive the robot to execute tasks, control process data of the tasks is collected synchronously and encapsulated into a shared standardized experience data package to achieve knowledge transfer across robot platforms. This includes: S701, after the task is completed, recording and encapsulating the key data of the current task to form a standardized experience data package. The key data includes: task description information, fine-tuned meta-policy network parameter increments, successful attention pattern characteristics, control parameter combinations, and final control performance indicators; S702, uploading the standardized experience data package to the cloud collaborative knowledge base; S703, the cloud collaborative knowledge base clusters, denoises, and extracts knowledge from experience data packages from multiple industrial robot terminals to form an optimized knowledge set with different task labels; S704, when any industrial robot terminal faces a new task, providing its meta-policy network with optimized knowledge from the cloud collaborative knowledge base that has been recommended through similarity matching as prior initialization values ​​for network parameters to accelerate the adaptation process.

[0014] Secondly, an adaptive control system for industrial robots based on multimodal sensor fusion is also provided. This system includes: a data acquisition module for acquiring raw multimodal data and performing spatiotemporal alignment to obtain a spatiotemporally aligned multimodal data base; a feature extraction module for capturing the spatiotemporal semantic relationships of visual texture, tactile pressure distribution, and force fluctuations in the multimodal data based on the multimodal data base using a multi-head attention mechanism guided by a physical model, and performing spatiotemporal registration to obtain a physically enhanced multimodal feature matrix; a feature fusion module for extracting visual texture features and temporal tactile features from the physically enhanced multimodal feature matrix using a CNN-LSTM hybrid model, performing feature-level fusion through a cross-modal attention mechanism to generate a physically consistent fused feature vector; and a weighted allocation module. The system is used to acquire real-time environmental parameters and dynamically assign weights to physically consistent fusion feature vectors using a meta-reinforcement learning-driven weight allocation model to generate dynamically weighted fusion features. A mode switching module is used to construct a dual-modal control strategy switching mechanism based on fuzzy PID and model predictive control. This mechanism generates mode switching signals and control calculation requirements based on the dynamically weighted fusion features. An instruction generation module selects the optimal control strategy for the task from a pre-built control algorithm library based on the dynamically weighted fusion features and control calculation requirements, and generates corresponding control instructions. A transfer learning module uses control instructions to drive the robot to execute tasks, synchronously collects control process data for the tasks, and encapsulates it into a shared, standardized experience data package to achieve knowledge transfer across robot platforms.

[0015] (III) Beneficial Effects Compared with existing technologies, this invention provides an adaptive control method and system for industrial robots based on multimodal sensor fusion, which has the following advantages: (1) This invention uses a multi-head attention mechanism guided by a physical model to deeply mine the semantic associations of visual texture, tactile pressure distribution, and force fluctuations in the spatiotemporal dimension, and performs spatiotemporal registration operations, effectively solving the technical problem of "simple feature splicing and ignoring physical constraints" in existing multimodal fusion technologies. This mechanism captures the spatial matching degree of the assembly contact surface through a spatial attention matrix and analyzes the temporal characteristics of force fluctuations through a temporal attention matrix, so that the fused features can truly reflect the physical scene constraints. Compared with traditional methods, this invention greatly improves the spatial positioning accuracy of industrial robots in precision assembly scenarios, greatly enhances the stability of contact force control, and significantly improves the control robustness in complex operation scenarios.

[0016] (2) The present invention constructs a dynamic weight allocation model based on real-time environmental parameters, realizes real-time adaptive adjustment of the weights of each modality, and completely solves the defects of traditional systems such as "static weights and easy loss of control due to sudden environmental changes"; the model associates environmental feature vectors and fused feature vectors through state-action value functions, and can adjust the weight coefficients of visual / tactile / force modes in real time under scenarios such as sudden temperature changes and vibration interference, so that the dynamic weighted fused features always match the current environmental requirements and reduce the control deviation rate.

[0017] (3) This invention constructs a collaborative knowledge transfer and evolutionary learning mechanism across robot platforms, effectively solving the technical problem of isolated adaptive capabilities and the inability to achieve group knowledge sharing. By encapsulating the successful control experience of a single robot into a standardized experience data package and uploading it to the cloud collaborative knowledge base for refinement and sharing, any robot can obtain prior knowledge through similarity matching when facing a new task, and quickly initialize its control strategy, thereby realizing the leap from "single-machine intelligence" to "swarm intelligence". This significantly reduces the debugging cost and deployment time of robot groups when facing multi-variety, small-batch production tasks, and promotes the continuous collaborative evolution of group control performance. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an adaptive control method for industrial robots based on multimodal sensor fusion according to an embodiment of the present invention; Figure 2 This is a block diagram illustrating the principle of an adaptive control system for industrial robots based on multimodal sensor fusion, according to an embodiment of the present invention.

[0020] In the picture: 1. Data acquisition module; 2. Feature extraction module; 3. Feature fusion module; 4. Weighted allocation module; 5. Mode switching module; 6. Instruction generation module; 7. Transfer learning module. Detailed Implementation

[0021] According to an embodiment of the present invention, an adaptive control method for industrial robots based on multimodal sensor fusion is provided. The invention will now be further described in conjunction with the accompanying drawings and specific embodiments, as follows: Figure 1 As shown, the method includes: S1. Collect multimodal raw data and perform spatiotemporal alignment to obtain the spatiotemporally aligned multimodal data basis.

[0022] Specifically, the multimodal raw data includes assembly scene image data collected by the vision sensor, contact surface pressure distribution data collected by the tactile sensor, and assembly force fluctuation data collected by the force sensor.

[0023] The spatiotemporal alignment process first triggers synchronous acquisition by each sensor via a hardware synchronization signal, ensuring strict consistency of data in the temporal dimension. Secondly, it employs an existing image registration algorithm based on feature point matching, combined with spatial coordinate mapping of tactile pressure distribution, to achieve precise spatial registration of visual-tactile data. Finally, it compensates for time deviations caused by differences in sampling frequencies among different sensors through temporal interpolation of the force signal. The generated spatiotemporal aligned data substrate contains a three-dimensional vector group with a unified timestamp, where each vector contains visual pixel coordinates, tactile pressure values, and force fluctuation values ​​at the corresponding location.

[0024] This invention collects raw visual, tactile, and force sensor data from industrial work scenarios using multimodal sensors. After spatiotemporal alignment processing, a multidimensional data base with a unified time reference is formed. A multi-head attention mechanism is used to deeply analyze the semantic relationships between the data in each modality across the spatiotemporal dimensions. A spatial attention matrix captures the geometric matching features of assembly contact surfaces, such as the alignment of part edges. Simultaneously, a temporal attention matrix is ​​used to analyze the temporal fluctuation characteristics of force signals, such as the periodic changes in contact force. A dynamic weight allocation model based on real-time environmental parameters automatically adjusts the weight coefficients of the visual / tactile / force sensory modalities according to the characteristics of the work scenario, generating dynamically weighted fusion features that reflect physical constraints.

[0025] After the feature input is used to switch between dual-modal control strategies, the task type and environmental sensitivity are analyzed by the pattern recognition module. The system then intelligently selects either fuzzy PID control mode or model predictive control mode. Finally, the control algorithm library generates precise control commands that are adapted to the current operation requirements, enabling high-precision adaptive control of industrial robots in complex environments.

[0026] S2. Based on a multimodal data base, a multi-head attention mechanism guided by a physical model is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution and force fluctuation in multimodal data, and spatiotemporal registration is performed to obtain a multimodal feature matrix with enhanced physical information.

[0027] In the description of this invention, based on a multimodal data basis, a multi-head attention mechanism guided by a physical model is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution, and force fluctuations in multimodal data, and spatiotemporal registration is performed to obtain a multimodal feature matrix enhanced with physical information, including: S201. Preprocess the visual data in the multimodal data base and extract the visual texture feature map.

[0028] Specifically, a texture feature extraction method based on Gabor filters is adopted. Multi-scale, multi-directional Gabor kernel functions are used to convolve the assembly scene image to generate a texture feature map containing orientation, scale, and phase information. This feature map can characterize the microscopic geometric structure of the part surface, such as surface roughness and texture orientation.

[0029] S202. Preprocess the tactile data in the multimodal data base and extract the tactile pressure distribution feature map.

[0030] Specifically, to extract tactile pressure distribution features, the raw pressure data collected by the tactile sensor is normalized to eliminate numerical bias caused by differences in sensor range. A bilinear interpolation algorithm is used to map discrete pressure sampling points onto a continuous two-dimensional spatial grid, generating a pressure distribution heatmap with resolution matching the visual feature map. Finally, an edge detection operator is used to extract the pressure gradient features of the contact surface, forming a tactile pressure distribution feature map containing information on pressure amplitude and spatial distribution. This feature map can reflect the actual contact state of the assembly contact surface, such as contact area and pressure concentration areas.

[0031] S203. Preprocess the force perception data in the multimodal data base and extract the force perception fluctuation feature sequence.

[0032] Specifically, the method for extracting the force wave characteristic sequence is as follows: The raw force signal collected by the force sensor is filtered and denoised to eliminate high-frequency noise interference; the time-domain force signal is converted into a frequency-domain energy distribution map using short-time Fourier transform to analyze the frequency components of the force wave; the amplitude, mean, and standard deviation of the force wave within the window are calculated using a sliding window statistical method to form a time-series characteristic sequence reflecting the dynamic characteristics of force; finally, the characteristic sequence is normalized to ensure the consistency of characteristic dimensions under different operating scenarios. This characteristic sequence can characterize the mechanical interaction state during the assembly process.

[0033] S204. Based on the robot's kinematics model and prior physical knowledge of the task, generate a spatial physical correlation bias matrix and a temporal physical constraint sequence.

[0034] In the description of this invention, the generation of a spatial physical correlation bias matrix and a temporal physical constraint sequence based on the robot's kinematic model and prior physical knowledge of the task includes: S2041. Based on the robot's forward kinematics model and the calibration parameters of the vision camera, calculate the expected contact area between the robot's end effector and the task in the visual data, and generate a spatial physical association bias matrix to enhance the attention weight of visual-tactile features within the expected contact area.

[0035] Specifically, the generation process of the spatial physical correlation bias matrix is ​​based on the deep fusion of robot kinematics and visual perception. The forward kinematics model of the robot is a mathematical model of the robotic arm established through the Denavit-Hartenberg (DH) parameter method. This model includes four core parameters: link length, link torsion angle, link offset, and joint variables. By substituting the real-time acquired joint angles into the DH parameter table, the pose transformation matrix of the end effector relative to the base coordinate system can be calculated step by step.

[0036] The calibration parameters of a vision camera include the intrinsic parameter matrix (focal length, principal point coordinates) obtained by Zhang Zhengyou calibration method and the extrinsic parameter matrix obtained by hand-eye calibration, which is the transformation relationship between the camera and the base coordinate system.

[0037] The expected contact area is represented by Boolean operations on the geometric model of the robot's end effector and the 3D model of the target object: the pose of the end effector in the base coordinate system is calculated using forward kinematics, then transformed to the camera coordinate system using a hand-eye calibration matrix, and finally the 3D geometric model is projected onto the 2D image plane using a perspective projection model. Specifically, a ray casting algorithm is used to detect the minimum distance field between the end effector model and the target model. When the distance is less than a set threshold, it is identified as a potential contact area. A conservative contact area estimate, considering positioning errors, is then generated through morphological dilation.

[0038] Finally, in the image coordinate system, pixels within the expected contact area are assigned a higher positive bias value, while non-contact areas are assigned a zero bias value. The images are then smoothed using Gaussian filtering to form a two-dimensional bias matrix with the same size as the image, namely the spatial physical correlation bias matrix.

[0039] S2042. Based on the mechanical model of the task and Coulomb's law of friction, the expected fluctuation frequency and amplitude range of the force sensory signal are derived, and a temporal physical constraint sequence is generated. This sequence is used to constrain the distribution of temporal attention weights so that they conform to physical laws.

[0040] Specifically, the mechanical model of the task is modeled using a spring-damped-mass system, where the stiffness coefficient is obtained through material mechanical property testing, and the damping coefficient is calibrated through vibration attenuation experiments. The static and dynamic friction coefficients in Coulomb's law of friction are determined through inclined plane sliding experiments.

[0041] The expected fluctuation frequency of the force-feed signal is obtained by solving the system's characteristic equations: first, the state-space equations of the contact system are established, the transfer function is obtained through Laplace transform, and then the imaginary part of the characteristic roots is solved to obtain the system's natural frequency. The amplitude range is determined by solving the system's steady-state response under a unit step input, while considering the limiting cycle oscillations caused by friction. Specifically, when generating the constraint sequence, a short-time Fourier transform is used to perform time-frequency analysis on the historical force-feed signal, extracting the main frequency components and their harmonics. Combined with the natural frequencies predicted by the mechanical model, a weight distribution in the form of a bandpass filter is generated. For the transient impact phase, a Gaussian function is used to simulate the decay process of the attention weights; for the steady-state contact phase, a comb filter is used to enhance the harmonic components that match the model frequency.

[0042] The method for generating a time-physical constraint sequence is as follows: The expected contact time point is determined, and a time window is set centered on this time point based on the calculated natural frequency. Within the window, the distribution of bias weights is set according to the law of damped vibration, with the highest weight set at the instant of contact, decaying exponentially. Simultaneously, based on the expected force signal fluctuation frequency, corresponding periodic modulation is superimposed on the weight distribution to enable the attention mechanism to better capture the periodic characteristics of the force signal. Finally, a one-dimensional bias sequence with the same length as the force signal sequence is generated.

[0043] S205. Combine the visual texture feature map, tactile pressure distribution feature map, and force wave feature sequence with the spatial physical association bias matrix and the temporal physical constraint sequence, input them into the multi-head attention mechanism model guided by the physical model, calculate the spatiotemporal attention weights between different modalities, capture the semantic association between visual texture, tactile pressure distribution, and force wave, and obtain the global attention weight matrix.

[0044] Specifically, the multi-head attention mechanism module guided by the physical model performs feature embedding transformation on the visual texture feature map, tactile pressure distribution feature map and force wave feature sequence through self-attention calculation, and introduces the spatial physical correlation bias matrix and temporal physical constraint sequence generated based on robot kinematics and contact model as prior knowledge guidance.

[0045] The visual feature map is unfolded into a two-dimensional vector sequence, and the tactile heatmap is transformed into a spatial pressure vector set. The force feature sequence maintains its temporal vector form, and a query vector Q, a key vector K, and a value vector V are generated through linear transformation. To address the interaction relationships between different modalities, a physical bias term is incorporated into the attention score calculation when calculating cross-modal attention weights. Taking visual-tactile attention as an example, the calculation process is as follows: visual query vector Q... v With tactile key vector K t The dot product result is superimposed with the space physical correlation bias matrix. The corresponding element values, after Softmax normalization, yield a visual-tactile attention weight matrix with enhanced physical information. The element values ​​of this matrix simultaneously reflect the data-driven association strength and the expected attention level under physical constraints. Similarly, a temporal physical constraint sequence is introduced when calculating visual-force attention. .

[0046] By weighted summation and aggregation of feature vectors from various modalities, a fused feature vector containing multimodal semantic associations and conforming to physical laws is generated. Furthermore, a multi-head parallel structure is employed, independently calculating attention weights through eight different sets of linear transformation parameters. Each attention head can focus on different physical constraint dimensions, such as spatial contact geometry and temporal mechanical properties. The outputs of each head are concatenated and linearly projected to generate a global attention weight matrix. The row vectors of this matrix correspond to the spatiotemporal positions in the original data, and the column vectors represent the contribution of different modal features. The magnitude of this matrix reflects both the spatiotemporal coupling relationship between visual texture edge alignment, tactile pressure gradient change rate, and force wave periodicity, and the guiding role of the physical model in the attention distribution.

[0047] In the description of this invention, the expression for calculating the spatiotemporal attention weights between different modalities is as follows: ; ; ; ; in, Represents the spatial attention matrix; Represents the visual modality query matrix; , , Represents the learnable projection matrix; Represents the tactile modality key matrix; Indicates the spatial dimension scaling factor; Represents the temporal attention matrix; Represents the key matrix of force perception modes; Indicates the scaling factor for the time dimension; Indicates the first The final spatiotemporal attention weight matrix for each attention head; Indicates the dynamic gating coefficient; Represents the sigmoid function; Represents the learnable weight matrix; Indicates the learnable bias term; This represents the spatial physical correlation bias matrix generated based on the robot's kinematics and contact model; This represents a time-physical constraint bias sequence generated based on the physical laws of force-sensory waves. , All of these represent learnable scalar coefficients.

[0048] S206. Based on the global attention weight matrix, feature fusion and spatiotemporal registration are performed on the visual texture feature map, tactile pressure distribution feature map and force wave feature sequence, and the alignment of features in time and space dimensions is adjusted to generate a multimodal feature matrix with enhanced physical information.

[0049] Specifically, the global attention weight matrix dynamically fuses visual, tactile, and force features through a weighted aggregation mechanism, where the embedded physical prior knowledge ensures that the fusion process conforms to robot kinematics and dynamics constraints.

[0050] For each pixel position in the visual texture feature map, based on the modal contribution of the corresponding row vector in the global attention weight matrix (which has been corrected by physical bias), the pressure values ​​at the same position in the tactile pressure distribution feature map and the synchronous temporal values ​​of the force wave feature sequence are weighted and summed to generate a fusion feature value with enhanced physical information.

[0051] For example, when edge alignment is required on the assembly contact surface, spatial attention weights guided by the physical model more accurately strengthen the association between the visual texture edge region and the tactile pressure gradient region. This allows the fused features to highlight the contact state at that location not only based on data association but also on the contact probability predicted by the kinematic model. When analyzing the contact force fluctuation cycle, the temporal attention weights introduced with mechanical priors more effectively adjust the temporal alignment accuracy between the force feature sequence and the visual / tactile features, ensuring that the multimodal data maintains strict synchronization in the time dimension and conforms to the dynamic laws. Through pixel-by-pixel weighted fusion and coordinate mapping, a three-dimensional feature matrix containing a unified spatiotemporal reference and deeply integrated with physical prior information is finally generated. Each matrix element is a weighted combination of visual coordinates, tactile pressure values, and force fluctuation values ​​guided by physical laws. This matrix not only retains the physical constraint information of the original data, such as assembly gaps and contact force stability, but also achieves explicit expression of cross-modal semantic association through the attention mechanism and enhances the physical rationality and reliability of the feature representation through physical model guidance.

[0052] S3. A CNN-LSTM hybrid model is used to extract visual texture features and temporal tactile features from the multimodal feature matrix with enhanced physical information. Feature-level fusion is performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency.

[0053] In the description of this invention, a CNN-LSTM hybrid model is used to extract visual texture features and temporal tactile features from a multimodal feature matrix enhanced with physical information. Feature-level fusion is performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency, including: S301. A convolutional neural network model is used to process the visual features in the multimodal feature matrix of physical information enhancement and extract visual texture features.

[0054] Specifically, the convolutional neural network model selected in this invention adopts a residual structure, which includes multiple convolutional layers, batch normalization layers, and ReLU activation function layers.

[0055] This convolutional neural network (CNN) performs initial feature extraction on the visual feature map using an initial convolutional layer with 7x7 kernels, generating a 64-channel feature map. It then proceeds to three residual blocks, each consisting of three 3x3 concatenated convolutional layers. The first convolutional layer expands the number of channels to 128, while the latter two maintain the same number of channels. The residual connections are adjusted in dimension using 1x1 convolutions and then added to the main path output, effectively mitigating the gradient vanishing problem in deep networks. Finally, a global average pooling layer compresses the feature map into a 512-dimensional visual feature vector. This network structure, through local receptive fields and weight sharing mechanisms, can automatically learn hierarchical representations of geometric features such as part edges and surface textures in assembly scenarios. It is particularly adept at capturing assembly alignment features requiring micron-level precision, such as chip pin spacing and gear tooth profile errors.

[0056] S302. Use a long short-term memory model to process the tactile features in the multimodal feature matrix of physical information enhancement and extract temporal tactile features.

[0057] Specifically, the Long Short-Term Memory (LSTM) model processes tactile sequence data through a gating mechanism, effectively capturing the temporal evolution of pressure distribution during assembly. The LSM model's structure comprises a triple control unit: an input gate, a forget gate, and an output gate. The input gate uses a sigmoid function to determine the retention ratio of pressure increment information at the current moment; the forget gate dynamically clears redundant pressure data based on historical states; and the output gate controls the transmission strength of the temporal feature vector. The model employs a two-layer LSTM architecture, with each layer containing 128 hidden units. The first layer receives a normalized tactile pressure heatmap sequence (128×128 resolution) and stores the migration trajectory of the pressure center on the contact surface through memory units. The second layer performs temporal abstraction on the output of the first layer, generating a 256-dimensional temporal feature vector containing the pressure fluctuation period and the rate of change of contact area.

[0058] Specifically, a peephole connection is introduced at each time step, allowing the current gating unit to access the memory unit state from the previous moment, enhancing the response capability to sudden pressure shocks. The model optimizes the cross-entropy loss function using a backpropagation algorithm. The training data includes 2000 sets of pressure time-series samples from typical assembly scenarios. The final tactile feature extractor improves the accuracy of pressure change pattern recognition on the test set, and is particularly suitable for continuous monitoring of micro-Newton level contact force changes in precision assembly.

[0059] S303. Visual texture features and temporal tactile features are fused at the feature level through a cross-modal attention mechanism, and attention weights between different modalities are calculated to generate a fused feature vector with physical consistency.

[0060] Specifically, the cross-modal attention mechanism achieves complementary fusion of multimodal information by dynamically calculating the correlation strength between visual and tactile features.

[0061] For visual feature vector V v and tactile feature vector V t Perform a linear transformation to generate the corresponding query vector Q. v / Q t Key vector K v / K t Sum vector V v' / V t' The visual feature dimension is 512 dimensions, and the tactile feature dimension is 256 dimensions. Next, the bidirectional attention weights are calculated, with the visual → tactile attention weights calculated via Q. t With K v The dot product operation is used to obtain the tactile → visual attention weights via Q. v With K t The transpose of the matrix is ​​multiplied to obtain the attention distribution matrix after Softmax normalization. The element values ​​reflect the degree of attention of tactile pressure temporal changes to visual texture features. Finally, a weighted summation strategy is used to generate a fusion feature vector. For each element of the visual feature vector, the tactile feature value is aggregated according to the tactile → visual attention weight, and vice versa, finally obtaining a cross-modal fusion feature containing 768 dimensions.

[0062] The cross-modal attention mechanism introduces a dynamic gating unit, generating an adjustment coefficient between 0 and 1 using a sigmoid function. When visual occlusion occurs in the assembly scene, the contribution of tactile features is automatically enhanced; when noise occurs in the tactile sensor, the stability of visual features is strengthened, ensuring the robustness of the fused features under complex conditions. The expression for calculating the attention weights between different modalities is as follows: ; , ; in, Represents the fused feature vector; Indicates the visual → tactile attention weight; Represents a visual modality value vector; This indicates the weight of tactile → visual attention. Represents the tactile modality value vector; Represents the visual modality key vector; Represents the transpose of the haptic modal key vector; This represents the cross-modal scaling factor.

[0063] S304. Calculate the consistency score between the fused feature vector and the rules in the pre-built physical constraint library. If the score is lower than the preset threshold, recalculate the spatial physical correlation bias matrix and the temporal physical constraint sequence until the consistency score is greater than the preset threshold.

[0064] Specifically, the physical constraint verification module includes a predefined physical rule base and a consistency scoring mechanism.

[0065] The physical constraint library contains rules such as kinematic constraints, dynamic constraints, and geometric constraints, including end effector range limits, maximum contact force thresholds, and minimum safe distances.

[0066] The consistency score is calculated as follows: relevant physical parameters are parsed from the fused feature vector and compared one by one with the rules in the constraint library to calculate the proportion that meets the constraints. Specifically, a fuzzy logic system is used for scoring, transforming the constraints into membership functions, and a weighted average is used to calculate the overall consistency score. When the score is lower than a preset threshold, a feedback mechanism is triggered to recalculate the physical bias parameters, including adjusting the expected contact area and the feature extraction parameters of the force sensor signal, until the output features meet the physical constraints.

[0067] S4. Obtain real-time environmental parameters and use a meta-reinforcement learning-driven weight allocation model to dynamically assign weights to the fusion feature vectors with physical consistency, generating dynamically weighted fusion features.

[0068] In the description of this invention, obtaining real-time environmental parameters and using a meta-reinforcement learning-driven weight allocation model to dynamically assign weights to fusion feature vectors with physical consistency to generate dynamically weighted fusion features includes: S401. Collect real-time environmental parameters and extract real-time environmental feature vectors.

[0069] Specifically, real-time environmental parameters are acquired through a multi-type sensor array, including temperature sensors, humidity sensors, vibration accelerometers, and light intensity meters.

[0070] Sensor data is converted by a 16-bit ADC and then synchronized with the sampling time using a time synchronization module to eliminate time deviations from multi-source data. The feature extraction process employs a hierarchical processing strategy: the bottom layer uses a sliding window statistical method to calculate the time-domain mean, variance, and frequency-domain energy spectral density of environmental parameters; the middle layer uses wavelet transform to decompose vibration signals and extract the resonance peak frequency and damping ratio within the 0-500Hz frequency band; the top layer combines assembly task types to construct environment-task related features, such as multiplying temperature fluctuation amplitude by the material's thermal expansion coefficient to generate environmental interference terms affecting assembly gaps. Finally, a 32-dimensional real-time environmental feature vector is generated through feature concatenation: the first 8 dimensions are temperature-related features, the middle 12 dimensions are vibration-related features, and the last 12 dimensions are humidity-light composite features. This vector is then dynamically weighted after normalization.

[0071] S402. Construct a meta-policy network based on a meta-reinforcement learning framework, and obtain initialization parameters adapted to new tasks by learning from a multi-task training set. The initialization parameters of the meta-policy network can be loaded from a collaborative knowledge base.

[0072] In the description of this invention, a meta-policy network is constructed based on a meta-reinforcement learning framework, and initialization parameters adapted to new tasks are obtained by learning from a multi-task training set, including: S4021. Construct a meta-training task set containing various industrial robot operation tasks, with each task corresponding to different environmental conditions, workpiece objects, and control objectives.

[0073] Specifically, the construction of the meta-training task set adopts a systematic task engineering approach.

[0074] Industrial robot tasks are scientifically categorized according to international standards, covering major industrial scenarios such as assembly, welding, and painting. Environmental conditions are constructed considering multi-dimensional parameters such as temperature, humidity, and vibration spectrum, and representative environmental combinations are generated through experimental design methods. A parametric design system is employed to fully describe key features such as geometric dimensions, material properties, and surface characteristics.

[0075] The control objectives are set based on key performance indicators of the task, including constraints such as positioning accuracy, force control accuracy, and operation time. The final meta-training task set contains a large number of independent task instances, each equipped with a complete simulation environment configuration and performance evaluation criteria.

[0076] S4022. In the meta-training phase of the meta-policy network, the parameters of the meta-policy network are optimized through a model-independent meta-learning algorithm. When the meta-policy network faces a new task from the task set, it obtains a new weight allocation strategy through gradient update.

[0077] Specifically, the implementation of the model-independent meta-learning algorithm adopts a hierarchical optimization strategy. The meta-policy network adopts a deep residual network architecture, which includes an input layer, multiple residual blocks, and an output layer.

[0078] The meta-training process is clearly divided into two stages: inner loop adaptation and outer loop optimization. The inner loop stage performs rapid parameter adjustments for each specific task, while the outer loop stage optimizes the network's initial parameters based on the validation results of multiple tasks. The training process employs a comprehensive loss function to balance multiple objectives such as policy optimization, value estimation, and exploration incentives. The entire training cycle includes a large number of iterations to ensure that the network achieves good generalization ability.

[0079] S4023. During the deployment phase, when a new task is discovered, the meta-policy network fine-tunes its parameters based on online-collected interactive data to achieve adaptive adjustment of the weight allocation strategy.

[0080] Specifically, the identification of new tasks is achieved through real-time feature comparison, and an adaptive mechanism is triggered when a feature pattern significantly different from that of historical tasks is detected. The fine-tuning process employs an incremental learning strategy, gradually adjusting network parameters while ensuring stability.

[0081] When the system identifies a new task through real-time data stream monitoring, if the difference between its feature pattern and the features of historical tasks in the knowledge base exceeds a set threshold, the online fine-tuning program is immediately started. First, a temporary data buffer is established, and interactive data during the execution of the current task is continuously collected, including environmental sensor readings, multimodal fusion feature vectors, control commands, and corresponding execution effect evaluation indicators.

[0082] The fine-tuning process employs an incremental gradient descent algorithm, using the network parameters obtained from meta-training as the initial point and aiming to minimize the control loss function for the current task. Specifically, the system performs a parameter update every time a certain amount of interaction data is accumulated, such as 50-100 complete data records over time steps: calculating the error gradient between the weight allocation scheme output by the policy network under the current parameters and the actual control effect, and adjusting the network weight parameters in the opposite direction of the gradient with a small step size. Simultaneously, an elastic weight consolidation mechanism is introduced, which imposes constraints on important parameters by calculating the parameter importance matrix, preventing excessive forgetting of the learned task strategy when adapting to new tasks. Adaptive adjustment is reflected in the real-time optimization of the dynamic weight allocation strategy. As the fine-tuning process progresses, the meta-policy network gradually learns the optimal allocation rules for the weights of each modality in the new task environment: when visual data has high reliability in recognizing new types of workpieces, the network automatically increases the weight coefficient of the visual modality; when the force signal exhibits abnormal fluctuations in an unknown contact dynamics scenario, the system correspondingly reduces the weight influence of the force mode. This adjustment process is achieved through changes in the internal parameters of the policy network, ensuring that the weight allocation strategy adapts to the characteristics of the current task in real time.

[0083] The fine-tuning effect is verified in real time through online performance monitoring: when the task completion indicators reach the preset standards within multiple consecutive time windows, such as the positioning error being stable within the allowable range and the force control accuracy meeting the requirements, the system automatically determines that the fine-tuning is complete, locks the current network parameters as the optimization strategy for this task type, and stores the adjusted parameters as new knowledge samples in the collaborative knowledge base, completing the closed loop from detection to adaptation.

[0084] S403. Construct a weight allocation model. The weight allocation model takes the real-time environment feature vector and the fused feature vector with physical consistency as input states, and uses the meta-policy network for parameter initialization and fine-tuning. Through the mapping relationship between the state space and the action space, it generates the real-time weight coefficients corresponding to each modality data.

[0085] Specifically, the expression for the weight allocation model is: , ; , ; in, Represents the dynamic weight coefficient vector; State-action value function; Represents a state-space vector; Represents the action space vector; Represents the set of parameters for a neural network; Represents the environmental feature vector; Represents the fused feature vector; Represents the symbol for the real number field; Characteristic dimensions representing environmental parameters; The feature dimension representing the fused features; This represents the incremental adjustment value for the visual modality weights; This represents the incremental adjustment value for the tactile modality weights; This represents the incremental adjustment value for the force mode weights; Indicates the activation function; , Represents the weight matrix of a neural network; , This represents the bias term of the neural network.

[0086] S404. Calculate the basic weights based on the confidence scores of each modality in the fusion feature vector with physical consistency, and perform weighted summation by combining the real-time weight coefficients to generate dynamic weighted fusion features.

[0087] Specifically, the expression for calculating the basic weights is: ; in, Represents the basic weight vector; Represents the confidence vector for each modality; Representing modal data Reliability metrics; Represents visual texture features; Indicates the characteristics of tactile pressure distribution; Indicates the characteristics of force perception fluctuation; This represents the element-wise multiplication operation; Indicates a spatial / temporal attention contribution ratio adjuster; Represents the spatial dimension of the attention matrix; This represents the attention matrix in the time dimension.

[0088] S5. A dual-mode control strategy switching mechanism is constructed based on fuzzy PID and model predictive control. The dual-mode control strategy switching mechanism generates mode switching signals and control calculation requirements based on dynamic weighted fusion features.

[0089] In the description of this invention, a dual-modal control strategy switching mechanism is constructed based on fuzzy PID and model predictive control. The dual-modal control strategy switching mechanism generates mode switching signals based on dynamically weighted fusion features, and the control calculation requirements include: S501. Perform multi-dimensional analysis on the dynamic weighted fusion features, extract task type identifiers, environmental sensitivity indicators and modal confidence weights, and generate mode switching signals through pattern recognition.

[0090] Specifically, the multi-dimensional analysis process employs a hierarchical feature extraction strategy. First, Principal Component Analysis (PCA) reduces the 1024-dimensional fused features to 128 dimensions, retaining 95% of the variance information. Second, Support Vector Machine (SVM) is used to classify task types, distinguishing typical assembly tasks such as precision insertion, thread tightening, and curved surface fitting based on the coupling relationship between visual texture features and tactile pressure distribution. The environmental sensitivity index is generated by calculating the gradient sensitivity of the feature vector to environmental parameters such as temperature and vibration; when the gradient change rate exceeds a threshold, an environmental interference warning is triggered. The modal confidence weights directly inherit the normalization results from step S404. The pattern recognition module uses a dynamic Bayesian network, inputting the above three types of features into a pre-trained decision tree model. This model is trained using historical assembly data and contains a decision path library of 2000 samples. When a task type switch or excessive environmental sensitivity is detected, a high-level mode switching signal is generated; otherwise, the current control mode is maintained.

[0091] S502. When the mode switching signal triggers the fuzzy PID control mode, the PID parameter library is dynamically adjusted based on the real-time weight coefficients in the dynamic weighted fusion feature to generate control calculation requirements.

[0092] Specifically, the fuzzy PID control mode adopts a parameter adaptive adjustment strategy, matching the optimal control parameters from a preset PID parameter library based on the real-time weight coefficients in the dynamic weighted fusion features.

[0093] First, a PID parameter library containing 50 sets of typical operating condition parameters is constructed. Each set of parameters corresponds to a different visual-tactile-force weight distribution range. For example, when the tactile weight accounts for more than 60%, the integral-separated PID parameter with strong anti-interference ability is preferentially selected. Second, dynamic parameter mapping is achieved through a fuzzy inference system. This system takes real-time weight coefficients as input, generates fuzzy membership degrees for each weight range through a membership function, and then generates parameter adjustment amounts through Mamdani inference rules. Finally, the precise PID parameter correction values ​​are obtained through defuzzification, and then superimposed with the basic parameters to generate the final control parameters. In particular, when a certain modal weight undergoes a sudden change, the system automatically triggers a parameter protection mechanism, limiting the parameter adjustment range to no more than 30% of the basic value to prevent control instability.

[0094] S503. When the mode switching signal triggers the model predictive control mode, a state space model is constructed based on the dynamic weighted fusion features. The state space model integrates visual space constraints, tactile contact constraints, and force-mechanical constraints derived from the multimodal feature matrix enhanced by physical information. An optimized control sequence is generated through a quadratic programming solver.

[0095] Specifically, the expression for the state-space model is: ; in, express The state variable vector at any given time; , and Represents the parameters of the state-space model; express The control input vector at each time step; Represents the feature fusion matrix; express Dynamic weighted fusion characteristics at any given time; express The observation output vector at time step; and Represents constraint parameters used to encode visual spatial constraints, tactile contact constraints, and force-mechanical constraints.

[0096] Model predictive control requires the construction of a state-space model with multimodal constraints, and the solution of the optimal control sequence is achieved through quadratic programming. This includes: the state variable vector is formed by concatenating dynamically weighted fusion features with historical control inputs. The dynamically weighted fusion features are averaged using a sliding window to eliminate instantaneous noise, and the historical control inputs are filtered using a first-order inertial filter to suppress input abrupt changes. Secondly, state-space parameters A, B, and C are extracted from historical assembly process data using a subspace identification method. A regularization term (λ=0.01) is introduced during identification to prevent overfitting, ensuring that the model's average prediction error within the 10-step prediction time domain is less than 3%. Finally, constraint parameters are implemented through a hierarchical coding mechanism: visual space constraints use the existing convex hull algorithm to generate the feasible region, limiting the three-dimensional position deviation of the assembly target point to within ±0.1 mm; tactile contact constraints describe the coupling relationship between pressure distribution and contact force using a friction cone model, activating the constraint when the standard deviation of the tactile pressure field exceeds a threshold; and force-sensory constraints extract the frequency domain features of the contact force through Fourier transform, limiting the energy of high-frequency vibration components (>50Hz) to within 15% of the total energy. The quadratic programming solver uses the effective set method to iteratively solve the problem within each control cycle. The objective function is the weighted sum of the rate of change of the control variable and the state tracking error. At the same time, slack variables are introduced to improve the robustness of the solution.

[0097] S6. Based on the dynamic weighted fusion characteristics and control calculation requirements, select the optimal control strategy for the task through a pre-built control algorithm library and generate the corresponding control instructions.

[0098] In the description of this invention, based on the dynamic weighted fusion characteristics and control computation requirements, the optimal control strategy for the task is selected through a pre-built control algorithm library, and corresponding control instructions are generated, including: S601. Based on dynamic weighted fusion characteristics and control calculation requirements, an initial control strategy framework is generated by matching the fuzzy PID algorithm or MPC algorithm in the pre-built control algorithm library through pattern identifier matching.

[0099] Specifically, the control algorithm library is built based on historical assembly data and simulation experiments, including a fuzzy PID module, an MPC module, and a hybrid mode of the two. The mode identifier is generated through the task type identifier and environmental sensitivity index in the dynamically weighted fusion features. For example, when a "curved surface fitting" task is detected and environmental vibration exceeds 2g, the mode identifier is automatically matched to "high-precision MPC mode". The matching process uses the nearest neighbor algorithm to calculate the Euclidean distance between the current feature vector and the pre-stored mode vectors in the algorithm library, selecting the module with the smallest distance as the initial strategy framework. If the modal confidence weight in the dynamically weighted fusion features shows that tactile feedback accounts for more than 55%, the tactile enhancement sub-mode in the fuzzy PID module is activated first; if the visual-tactile joint confidence is below the threshold, the system is forced to switch to the MPC module to ensure robustness.

[0100] Specifically, for multimodal conflict scenarios, the system generates mixed-mode instructions through a fuzzy logic synthesizer, while simultaneously leveraging the fast response characteristics of fuzzy PID and the global optimization capabilities of MPC. The mixing ratio is dynamically determined by a spatial / temporal attention contribution ratio adjuster. After the initial strategy framework is generated, the system further verifies the constraints in the control calculation requirements. If tactile contact constraints or force-based mechanical constraints are found to be activated, a constraint reinforcement layer is automatically added to the MPC module, and the physical feasibility of the control instructions is ensured by adjusting the weight coefficients of the quadratic programming solver.

[0101] S602. Perform multi-dimensional analysis on the dynamic weighted fusion features, extract the spatial constraint parameters of visual texture features, the temporal fluctuation coefficient of tactile pressure distribution, and the mechanical stability index of force fluctuation, and align the features with the constraints in the control calculation requirements to form an optimized input vector.

[0102] Specifically, the multi-dimensional analysis process is as follows: First, the spatial constraint parameters of visual texture features are extracted using a Gabor filter bank, which contains 24 channels across 8 directions and 3 scales. This filter bank quantifies the texture directionality and periodicity of the assembled surface, generating a spatial constraint parameter matrix (6×6 dimensions). Each element represents the texture direction consistency index of a local region (range 0-1). Second, the temporal fluctuation coefficient of tactile pressure distribution is decomposed using wavelet transform. The Daubechies4 wavelet basis is used to decompose the tactile pressure temporal signal into 5 layers. The energy proportion of high-frequency detail coefficients (D3-D5 layers) is extracted as a fluctuation index. When the energy proportion exceeds 15%, it is determined to be an abnormal fluctuation in the tactile signal. The mechanical stability index of force fluctuation is calculated using the Lyapunov exponent. A phase space reconstruction is constructed on the force temporal data, and the maximum Lyapunov exponent is calculated. When the exponent value is less than -0.05, the mechanical system is considered stable. In the feature alignment stage, a constraint projection transformation is used to map the three types of features to the constraint space of the control computation requirements: the visual space constraint parameters are aligned with the feasible region boundary in the MPC module through affine transformation to ensure that the position deviation of the assembly target point is still within the convex hull feasible region after mapping; the tactile temporal fluctuation coefficient is converted into the constraint activation threshold of the friction cone model through exponential mapping, and the tactile contact constraint is triggered when the mapping value exceeds 0.8; the force stability index is converted into frequency domain constraint weights through the existing logistic regression model, and the energy limit weight of the high-frequency vibration component is dynamically adjusted from 0.15 to 0.08 when the Lyapunov exponential mapping value is lower than the threshold. The final optimized input vector has an 18-dimensional dimension, including 6-dimensional visual constraint parameters, 6-dimensional tactile fluctuation coefficients, and 6-dimensional force stability indexes. Each component is normalized to the [0, 1] interval, and feature cross-validation is used to ensure strict matching with the constraints in the control computation requirements. Specifically, when a modal feature is missing data due to sensor failure, the system automatically activates the feature completion mechanism: visual constraint parameters are replaced by the spatial gradient information of the tactile pressure field (calculated by the Laplacian operator of the tactile heatmap), tactile fluctuation coefficients are replaced by the peak value of the autocorrelation function of the force time series data, and the force stability index inherits the mean value in the historical assembly data. The completed feature vector must pass the consistency check before it can participate in the construction of the optimized input vector.

[0103] S603 generates control commands based on the initial control strategy framework and optimized input vector through a fuzzy inference engine or a quadratic programming solver.

[0104] Specifically, the control command generation process employs different computational paths depending on the type of the initial control strategy framework (fuzzy PID module or MPC module). When the initial strategy framework is a fuzzy PID module, the fuzzy inference engine uses the tactile fluctuation coefficient and force stability index in the optimization input vector as input. It generates a fuzzy set through predefined membership functions (trapezoidal membership function for tactile fluctuation coefficient, Gaussian membership function for force stability index), and then performs inference based on the expert rule base, outputting the adjustment amount (ΔK) of the PID parameters. p ΔK i ΔK d The adjustment amount, after being defuzzified, is superimposed on the basic PID parameters to generate the final control command. If the initial strategy framework is an MPC module, the quadratic programming solver uses the visual constraint parameters, tactile fluctuation coefficient, and force stability index in the optimization input vector as constraints. Combined with the state-space model (the state variables include the current position, velocity, and dynamically weighted fusion features, and the control input is the control quantity sequence for the next 5 steps), it constructs an optimization problem with the objective of minimizing the tracking error (weight 0.7) and the rate of change of the control quantity (weight 0.3). It iteratively solves the problem in each control cycle using the effective set method to generate the optimal control sequence for the next 5 steps, and takes the first step as the current control command. Specifically, when there are constraint activations in the optimized input vector, the system automatically adjusts the weights of the objective function of the quadratic programming to ensure that the control commands meet the physical constraints. For multimodal conflict scenarios, the system generates compromise commands through a hybrid strategy coordination module. This module dynamically allocates the weights of fuzzy PID and MPC according to the spatial / temporal attention contribution ratio adjuster (when λ=0.6 by default, fuzzy PID accounts for 60% for fast response and MPC accounts for 40% for global optimization). The final generated control commands must pass a safety verification before they can be output to the actuator.

[0105] It should be noted that the control algorithm library in step S601 includes a basic control module layer, a strategy selection layer, and a parameter configuration layer. The control module layer is connected to the strategy selection layer, and the strategy selection layer is connected to the parameter configuration layer. The basic control module layer includes a fuzzy PID module, an MPC module, and a fail-safe module. The fuzzy PID module contains a pre-stored PID parameter pool and a fuzzy rule base. The PID parameter pool dynamically adjusts the baseline parameters through the real-time weight coefficients in step S4. The fuzzy rule base associates the visual / tactile / force modal weight increments based on the mode switching signal. Regarding the control algorithm library, a further explanation is needed regarding the fault-tolerant module in the basic control module layer. This module is primarily used to automatically take over control and execute preset safety operations, such as emergency shutdown and fault isolation, when the system detects a serious anomaly or malfunction, to ensure the safety of the industrial robot and its surrounding environment. This module monitors the robot's operating status, sensor data, and the execution of control commands in real time. Once a situation exceeds a safety threshold, it immediately triggers the safety mechanism.

[0106] The strategy selection layer includes a pattern matching engine and a switching logic controller. The pattern matching engine matches fuzzy PID or MPC modules based on task type identifiers, while the switching logic controller achieves smooth switching by predicting time-domain parameters and switching delay counters to prevent control jitter. It is also worth elaborating on the pattern matching engine in the strategy selection layer. This engine not only performs initial matching based on task type identifiers but also incorporates multi-dimensional information such as environmental sensitivity indicators, the robot's current state, and historical task execution data to achieve more accurate and intelligent pattern selection. For example, when faced with complex and ever-changing assembly tasks, the pattern matching engine can comprehensively consider various factors and automatically select the most suitable control mode for the current task, thereby improving assembly efficiency and accuracy.

[0107] The switching logic controller is responsible for achieving a smooth, disturbance-free switch between the fuzzy PID module and the MPC module. It assesses the robot's operational trend over a future period by predicting time-domain parameters and uses a switching delay counter to avoid control jitter caused by frequent switching. When a switch in control mode is detected, the switching logic controller gradually adjusts the output weights of the two modules to achieve a smooth transition of control.

[0108] The parameter configuration layer includes a dynamic parameter interface and an adaptive regulator. The dynamic parameter interface uses the ROS2 standard interface to dynamically inject real-time environmental feature vectors and weight coefficients. The adaptive regulator optimizes PID parameters or MPC constraints based on control performance indicators using a gradient descent algorithm.

[0109] The parameter configuration layer, through the ROS2 standard interface, can efficiently receive real-time environmental feature vectors from various sensors. These vectors contain rich information about the robot's current working environment, such as temperature, humidity, and light intensity. Simultaneously, the dynamic parameter interface is responsible for dynamically injecting the weighting coefficients calculated by the system into the control system. These weighting coefficients reflect the degree of influence of different environmental characteristics on the control system, thereby ensuring that the control system can make corresponding adjustments according to real-time environmental changes.

[0110] Adaptive regulators, based on control performance indicators such as tracking error, rate of change of control input, and system stability, automatically adjust PID parameters or MPC constraints using optimization methods such as gradient descent algorithms. They can continuously optimize the performance of the control algorithm according to actual operating conditions, improving the adaptive capability and control accuracy of industrial robots. For example, after prolonged operation, if the system's tracking error is found to gradually increase, the adaptive regulator will automatically adjust the PID parameters to reduce the error and restore system stability.

[0111] S7. Use control commands to drive the robot to perform tasks, synchronously collect control process data of the tasks, and encapsulate it into a shared standardized experience data package to realize knowledge transfer across robot platforms.

[0112] In the description of this invention, the robot is driven to perform tasks using control commands, and the control process data of the tasks is collected synchronously and encapsulated into a shared standardized experience data package to achieve knowledge transfer across robot platforms, including: S701. After the task is completed, record and encapsulate the key data of the current task to form a standardized experience data package. The key data includes: task description information, incremental parameters of the fine-tuned meta-policy network, successful attention pattern characteristics, control parameter combinations, and final control performance indicators.

[0113] Specifically, the standardized experience data package encapsulation process employs a structured data organization scheme. Task description information includes metadata such as the workpiece's 3D point cloud feature vector, target accuracy requirements, and environmental parameter ranges, which are serialized and encoded using Protocol Buffer format. Meta-policy network parameter increments are recorded as differential tensors of the weight matrix, retaining important parameter changes through a hierarchical pruning algorithm, achieving a compression rate of 80%. Successful attention pattern features are extracted from the global attention weight matrix, including statistical features such as spatial attention peak distribution and temporal attention active intervals. Control parameter combinations are recorded as fuzzy PID coefficient vectors and MPC constraint sets, with final control performance indicators including trajectory tracking error, force control overshoot, and task completion time—multi-dimensional evaluation data. All data items are timestamped and then packaged into a unified structured experience data package using Apache Avro format.

[0114] S702. Upload the standardized experience data package to the cloud-based collaborative knowledge base.

[0115] Specifically, the data upload mechanism employs a breakpoint resumption and incremental synchronization strategy. Terminal devices establish a secure connection with the cloud service via HTTPS, and experience data packets are first encrypted with AES-256 before being transmitted in segments. The transport layer uses the QUIC protocol to ensure the stability of high-latency networks, and each data packet includes an SHA-256 checksum. The cloud receiving end has a time-series database that automatically adds time indexes, device fingerprints, and version tags to incoming data packets, storing them in a specific namespace of the HDFS distributed file system. Simultaneously, corresponding metadata indexes are generated and stored in the Elasticsearch search engine.

[0116] The S703 cloud-based collaborative knowledge base clusters, denoises, and extracts knowledge from experience data packages from multiple industrial robot terminals, forming an optimized knowledge set with different task labels.

[0117] Specifically, a density-based clustering algorithm is used to group task feature vectors, automatically identify abnormal data points and remove noise; through knowledge distillation technology, the control strategy parameters of similar tasks are weighted and fused to extract common knowledge models; and a knowledge graph is constructed to establish the relationship between task features, control strategies and performance indicators, and finally generate an optimized knowledge set with multi-dimensional labels, which is stored as a node relationship network in the graph database Neo4j.

[0118] S704. When any industrial robot terminal faces a new task, it provides its meta-policy network with optimized knowledge from the cloud-based collaborative knowledge base, recommended through similarity matching, as prior initialization values ​​for network parameters to accelerate the adaptive process.

[0119] Specifically, the feature vector of the new task is encoded into a semantic embedding by Sentence-BERT, and then the Top-K similar cases are retrieved from the knowledge base using the approximate nearest neighbor algorithm (Annoy).

[0120] The matching process incorporates multi-objective optimization, balancing task feature similarity, historical success rate, and resource consumption metrics. The recommended knowledge package, after digital signature verification, is loaded into the meta-policy network using network parameter grafting technology: the base layer parameters use universal initialization values ​​from the knowledge base, while the attention layer parameters are adaptively adjusted based on the current task characteristics. A safety sandbox is set up during the loading process, and a gradual variable update strategy is used to prevent sudden performance changes, ensuring a stable transition of the system during knowledge transfer.

[0121] According to another embodiment of the present invention, such as Figure 2 As shown, an adaptive control system for an industrial robot based on multimodal sensor fusion is also provided. This system includes: Data acquisition module 1 is used to acquire multimodal raw data and perform spatiotemporal alignment to obtain a spatiotemporally aligned multimodal data basis.

[0122] Feature extraction module 2 is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution and force fluctuation in multimodal data based on a multimodal data basis and through a multi-head attention mechanism guided by a physical model, and to perform spatiotemporal registration to obtain a multimodal feature matrix with enhanced physical information.

[0123] Feature fusion module 3 is used to extract visual texture features and temporal tactile features from the multimodal feature matrix with physical information enhancement using a CNN-LSTM hybrid model, and perform feature-level fusion through a cross-modal attention mechanism to generate a fused feature vector with physical consistency.

[0124] The weighted allocation module 4 is used to obtain real-time environmental parameters and use a meta-reinforcement learning-driven weight allocation model to dynamically allocate weights to the fusion feature vectors with physical consistency, thereby generating dynamically weighted fusion features.

[0125] The mode switching module 5 is used to construct a dual-mode control strategy switching mechanism based on fuzzy PID and model predictive control. The dual-mode control strategy switching mechanism generates mode switching signals and control calculation requirements based on dynamic weighted fusion features.

[0126] The instruction generation module 6 is used to select the optimal control strategy for the job task based on the dynamic weighted fusion characteristics and control calculation requirements through a pre-built control algorithm library, and generate the corresponding control instructions.

[0127] The transfer learning module 7 is used to drive the robot to perform tasks using control commands, synchronously collect control process data of the tasks, and encapsulate it into a shared standardized experience data package to realize knowledge transfer across robot platforms.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive control method and system for industrial robots based on multimodal sensor fusion, characterized in that, include: S1. Collect multimodal raw data and perform spatiotemporal alignment to obtain the spatiotemporally aligned multimodal data basis; S2. Based on a multimodal data base, a multi-head attention mechanism guided by a physical model is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution and force fluctuation in multimodal data, and spatiotemporal registration is performed to obtain a multimodal feature matrix with enhanced physical information. S3. A CNN-LSTM hybrid model is used to extract visual texture features and temporal tactile features from the multimodal feature matrix with enhanced physical information. Feature-level fusion is performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency. S4. Obtain real-time environmental parameters and use a meta-reinforcement learning-driven weight allocation model to dynamically allocate weights to the fusion feature vectors with physical consistency, generating dynamically weighted fusion features. S5. A dual-mode control strategy switching mechanism is constructed based on fuzzy PID and model predictive control. The dual-mode control strategy switching mechanism generates mode switching signals and control calculation requirements based on the dynamic weighted fusion features. S6. Based on the dynamic weighted fusion characteristics and control computation requirements, select the optimal control strategy for the task through a pre-built control algorithm library and generate the corresponding control instructions. S7. Use control commands to drive the robot to perform tasks, synchronously collect control process data of the tasks, and encapsulate it into a shared standardized experience data package to realize knowledge transfer across robot platforms.

2. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The method, based on a multimodal data basis, uses a multi-head attention mechanism guided by a physical model to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution, and force fluctuations in multimodal data, and performs spatiotemporal registration to obtain a physically enhanced multimodal feature matrix, including: S201. Preprocess the visual data in the multimodal data basis and extract visual texture feature maps; S202. Preprocess the tactile data in the multimodal data base and extract the tactile pressure distribution feature map; S203. Preprocess the force perception data in the multimodal data base and extract the force perception fluctuation feature sequence; S204. Based on the robot's kinematics model and prior physical knowledge of the task, generate a spatial physical correlation bias matrix and a temporal physical constraint sequence. S205. The visual texture feature map, tactile pressure distribution feature map, and force wave feature sequence are combined with the spatial physical association bias matrix and temporal physical constraint sequence, and input into the multi-head attention mechanism model guided by the physical model to calculate the spatiotemporal attention weights between different modalities, capture the semantic association between visual texture, tactile pressure distribution and force wave, and obtain the global attention weight matrix. S206. Based on the global attention weight matrix, feature fusion and spatiotemporal registration are performed on the visual texture feature map, tactile pressure distribution feature map and force wave feature sequence, and the alignment of features in time and space dimensions is adjusted to generate a multimodal feature matrix with enhanced physical information.

3. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 2, characterized in that, The generation of the spatial physical correlation bias matrix and temporal physical constraint sequence based on the robot kinematics model and prior physical knowledge of the task includes: S2041. Based on the robot's forward kinematics model and the calibration parameters of the vision camera, calculate the expected contact area between the robot's end effector and the task in the visual data, and generate a spatial physical correlation bias matrix. S2042. Based on the mechanical model of the task and Coulomb's friction law, derive the expected fluctuation frequency and amplitude range of the force sensor signal, and generate a time-physical constraint sequence.

4. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 2, characterized in that, The expression for calculating the spatiotemporal attention weights between different modalities is as follows: ； ； ； ； in, Represents the spatial attention matrix; Represents the visual modality query matrix; , , Represents the learnable projection matrix; Represents the tactile modality key matrix; Indicates the spatial dimension scaling factor; Represents the temporal attention matrix; Represents the key matrix of force perception modes; Indicates the scaling factor for the time dimension; Indicates the first The final spatiotemporal attention weight matrix for each attention head; Indicates the dynamic gating coefficient; Represents the sigmoid function; Represents the learnable weight matrix; Indicates the learnable bias term; This represents the spatial physical correlation bias matrix generated based on the robot's kinematics and contact model; This represents a time-physical constraint bias sequence generated based on the physical laws of force-sensory waves. , All of these represent learnable scalar coefficients.

5. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The method employs a CNN-LSTM hybrid model to extract visual texture features and temporal tactile features from a multimodal feature matrix enhanced with physical information. Feature-level fusion is then performed through a cross-modal attention mechanism to generate a fused feature vector with physical consistency, including: S301. A convolutional neural network model is used to process the visual features in the multimodal feature matrix of the physical information enhancement, and visual texture features are extracted. S302. Use a long short-term memory model to process the tactile features in the multimodal feature matrix of physical information enhancement and extract temporal tactile features; S303. The visual texture features and temporal tactile features are fused at the feature level through a cross-modal attention mechanism, the attention weights between different modalities are calculated, and a fused feature vector with physical consistency is generated. S304. Calculate the consistency score between the fused feature vector and the rules in the pre-built physical constraint library. If the score is lower than the preset threshold, recalculate the spatial physical correlation bias matrix and the temporal physical constraint sequence until the consistency score is greater than the preset threshold.

6. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The process of acquiring real-time environmental parameters and dynamically assigning weights to physically consistent fusion feature vectors using a meta-reinforcement learning-driven weight allocation model to generate dynamically weighted fusion features includes: S401. Collect real-time environmental parameters and extract real-time environmental feature vectors; S402. Construct a meta-policy network based on a meta-reinforcement learning framework, and obtain initialization parameters adapted to new tasks by learning from a multi-task training set; the initialization parameters of the meta-policy network can be loaded from a collaborative knowledge base. S403. Construct a weight allocation model. The weight allocation model takes the real-time environment feature vector and the fused feature vector with physical consistency as input states, and uses the meta-policy network for parameter initialization and fine-tuning. Through the mapping relationship between the state space and the action space, it generates real-time weight coefficients corresponding to each modality data. S404. Calculate the basic weights based on the confidence scores of each modality in the fusion feature vector with physical consistency, and perform weighted summation in combination with the real-time weight coefficients to generate dynamic weighted fusion features; The meta-policy network constructed based on the meta-reinforcement learning framework, which obtains initialization parameters adapted to new tasks through learning on a multi-task training set, includes: S4021. Construct a meta-training task set containing various industrial robot operation tasks, with each task corresponding to different environmental conditions, workpiece objects, and control objectives. S4022. In the meta-training phase of the meta-policy network, the parameters of the meta-policy network are optimized by a model-independent meta-learning algorithm. When the meta-policy network faces a new task from the task set, it obtains a new weight allocation strategy through gradient update. S4023. During the deployment phase, when a new task is discovered, the meta-policy network fine-tunes its parameters based on online collected interactive data to achieve adaptive adjustment of the weight allocation strategy.

7. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The dual-modal control strategy switching mechanism based on fuzzy PID and model predictive control, wherein the dual-modal control strategy switching mechanism generates mode switching signals based on the dynamically weighted fusion features and the control calculation requirements include: S501. Perform multi-dimensional analysis on the dynamic weighted fusion features, extract task type identifiers, environmental sensitivity indicators and modal confidence weights, and generate mode switching signals through pattern recognition; S502. When the mode switching signal triggers the fuzzy PID control mode, the PID parameter library is dynamically adjusted based on the real-time weight coefficients in the dynamic weighted fusion feature to generate control calculation requirements. S503. When the mode switching signal triggers the model predictive control mode, a state space model is constructed based on the dynamic weighted fusion features. The state space model integrates the visual space constraints, tactile contact constraints, and force-mechanical constraints derived from the multimodal feature matrix enhanced by the physical information. An optimized control sequence is generated through a quadratic programming solver.

8. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The step of selecting the optimal control strategy for the task based on dynamic weighted fusion characteristics and control computation requirements, and generating corresponding control instructions through a pre-built control algorithm library, includes: S601. Based on the dynamic weighted fusion features and control calculation requirements, an initial control strategy framework is generated by matching the fuzzy PID algorithm or MPC algorithm in the pre-built control algorithm library with the pattern identifier. S602. Perform multi-dimensional analysis on the dynamic weighted fusion features, extract the spatial constraint parameters of visual texture features, the temporal fluctuation coefficient of tactile pressure distribution, and the mechanical stability index of force fluctuation, and align them with the constraint conditions in the control calculation requirements to form an optimized input vector; S603. Based on the initial control strategy framework and optimized input vector, control commands are generated through a fuzzy inference engine or a quadratic programming solver.

9. The adaptive control method and system for industrial robots based on multimodal sensor fusion according to claim 1, characterized in that, The process of using control commands to drive the robot to perform tasks, synchronously collecting control process data of the tasks, and encapsulating it into a shared, standardized experience data package to achieve knowledge transfer across robot platforms includes: S701. After the task is completed, record and encapsulate the key data of the current task to form a standardized experience data package. The key data includes: task description information, fine-tuned meta-policy network parameter increments, successful attention pattern characteristics, control parameter combinations, and final control performance indicators. S702. Upload the standardized experience data package to the cloud-based collaborative knowledge base; S703, The cloud-based collaborative knowledge base clusters, denoises, and extracts knowledge from experience data packages from multiple industrial robot terminals to form an optimized knowledge set with different task labels; S704. When any industrial robot terminal faces a new task, it provides its meta-policy network with optimized knowledge from the cloud-based collaborative knowledge base, which has been recommended through similarity matching, as prior initialization values ​​for network parameters to accelerate the adaptive process.

10. An adaptive control system for an industrial robot based on multimodal sensor fusion, used to implement the adaptive control method for an industrial robot based on multimodal sensor fusion as described in any one of claims 1-9, characterized in that, The system includes: The data acquisition module is used to acquire multimodal raw data and perform spatiotemporal alignment to obtain a spatiotemporally aligned multimodal data basis. The feature extraction module is used to capture the spatiotemporal semantic associations of visual texture, tactile pressure distribution and force fluctuation in multimodal data based on a multimodal data basis and through a multi-head attention mechanism guided by a physical model, and to perform spatiotemporal registration to obtain a multimodal feature matrix with enhanced physical information. The feature fusion module is used to extract visual texture features and temporal tactile features from the multimodal feature matrix with physical information enhancement using a CNN-LSTM hybrid model. It performs feature-level fusion through a cross-modal attention mechanism to generate a fused feature vector with physical consistency. The weighted allocation module is used to obtain real-time environmental parameters and use a meta-reinforcement learning-driven weight allocation model to dynamically allocate weights to the fusion feature vector with physical consistency, generating dynamically weighted fusion features. The mode switching module is used to construct a dual-mode control strategy switching mechanism based on fuzzy PID and model predictive control. The dual-mode control strategy switching mechanism generates mode switching signals and control calculation requirements based on the dynamic weighted fusion features. The instruction generation module is used to select the optimal control strategy for the job task based on the dynamic weighted fusion characteristics and control calculation requirements, and generate the corresponding control instructions through a pre-built control algorithm library. The transfer learning module is used to drive the robot to perform tasks using control commands, synchronously collect control process data of the tasks, and encapsulate it into a shared standardized experience data package to realize knowledge transfer across robot platforms.

Images