Control method and device based on master-slave cooperation, equipment and medium

Abstract

The invention relates to the technical field of robot control, and discloses a master-slave cooperation-based control method, device, equipment and medium, and the method comprises the steps: collecting a spatial pose, a contact force and a dual-view image, and constructing a multi-modal demonstration data set; executing time alignment and normalization to form a standardized input sequence; action features, force features and visual features are extracted through the action modal coding network, the mechanical modal coding network and the visual coding network; fusing to form a multi-modal tensor sequence, and inputting the multi-modal tensor sequence into a Transform decoder to generate a joint time sequence representation; outputting a training action prediction vector from the joint time sequence representation through an action predictor, constructing a supervision loss function in combination with expert demonstration annotation to update each network, and obtaining an optimization control model; and processing the real-time input control driving execution mechanism based on the optimization control model. According to the method, the control model is trained and optimized through multi-modal sensing and time sequence modeling, the joint control instruction is generated in deployment, and stable execution of a complex interaction task is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular to a control method and device based on master-slave cooperation, equipment and a storage medium. BACKGROUND

[0002] Robot imitation learning (IL) has gradually become an important path to realize intelligent control of complex tasks, and is widely used in intelligent manufacturing, remote control, medical examination and collaborative handling scenes. The existing mainstream methods mainly include behavior cloning (BC) based on supervised learning and inverse reinforcement learning (IRL) based on reward function backpropagation. These methods can learn strategies using expert demonstrations to some extent, but still have significant deficiencies in complex interactive environments and multi-modal information fusion.

[0003] In existing imitation learning systems, most researches only rely on position control to achieve task execution, and fail to effectively integrate feedback signals such as contact force. When the robot faces different hardness, surface material or unstructured environment, the control method lacking force information will lead to insufficient motion stability and decline of interaction quality. For example, in operations involving force contact such as ultrasonic exploration, wiping or assembly, it is difficult to obtain reliable results with only position trajectories.

[0004] In addition, the existing imitation learning methods generally have the problem of low control frequency. The running frequency of a large number of systems based on visual or text input is often only 1-10 Hz, which is much lower than the natural rhythm of human hand movements. In high-speed operation tasks, this low-frequency control will cause delay and instability, causing the robot motion to be unable to smoothly follow the changes in the environment, resulting in frequent corrections and control jitter.

[0005] The expression ability of existing time series modeling methods is also limited. Common networks such as LSTM are prone to problems such as gradient disappearance and difficulty in capturing long-term dependencies when processing long time series and multi-modal data, making it difficult to model complex nonlinear and coupled relationships in human operation processes. This makes the model lack generalization ability when dealing with unknown tasks or new objects, making it difficult to maintain stable performance in open scenarios.

[0006] Finally, the existing data collection method is too single, usually based on video imitation or remote trajectory recording, and fails to fully capture the interaction force information between humans and objects and the environment. This one-sided data input leads to a lack of real interaction perception support in the training process, making it difficult for the model to learn the tactile control strategies formed by humans in operation, limiting the applicability of imitation learning in practical applications. SUMMARY

[0007] The main purpose of the present application is to provide a master-slave collaborative control method, device, equipment and storage medium, aiming at solving the technical problem that the prior art lacks multi-modal fusion modeling of spatial pose, contact force and visual image, resulting in that the robot cannot guarantee the stability and generalization ability of the action at the same time in complex interaction tasks.

[0008] To achieve the above-mentioned purpose, the present application provides a master-slave collaborative control method, comprising: Establishing a master-slave collaborative control link, synchronously collecting spatial pose coordinates, contact force vectors and double-view images through a bilateral force-position closed-loop feedback mechanism, and constructing a multi-modal demonstration data set; Performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; Processing the action state vector in the standardized input sequence through an action modal encoding network to obtain an action feature, processing the contact force vector in the standardized input sequence through a mechanics modal encoding network to obtain a force feature, and processing the double-view image in the standardized input sequence through a visual encoding network to obtain a visual feature; Fusing the action feature, the force feature and the visual feature into a multi-modal tensor sequence and inputting the multi-modal tensor sequence into a Transformer decoder to generate a joint time sequence representation; Outputting a training action prediction vector through an action predictor based on the joint time sequence representation, constructing a supervised loss function based on the training action prediction vector and expert demonstration labels, updating the parameters of the action modal encoding network, the mechanics modal encoding network, the visual encoding network, the Transformer decoder and the action predictor, and obtaining an optimized control model; In the deployment stage, processing real-time action positions, contact forces and visual images input through the optimized control model to generate a control action vector, and converting the control action vector into a joint space control instruction to drive an execution mechanism.

[0009] Further, to achieve the above-mentioned purpose, the present application provides a master-slave collaborative control device, comprising: A multi-modal data acquisition module is configured to establish a master-slave collaborative control link, synchronously collect spatial pose coordinates, contact force vectors and double-view images through a bilateral force-position closed-loop feedback mechanism, and construct a multi-modal demonstration data set; A data preprocessing module is configured to perform time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; a modal encoding module, configured to process the action state vector in the standardized input sequence through an action modal encoding network to obtain action features, process the contact force vector in the standardized input sequence through a force modal encoding network to obtain force features, and process the dual-view image in the standardized input sequence through a visual encoding network to obtain visual features; a feature fusion decoding module, configured to fuse the action features, the force features and the visual features into a multi-modal tensor sequence and input the multi-modal tensor sequence into a Transformer decoder to generate a joint temporal representation; a model training optimization module, configured to output a training action prediction vector from the joint temporal representation through an action predictor, construct a supervision loss function based on the training action prediction vector and expert demonstration labels, update parameters of the action modal encoding network, the force modal encoding network, the visual encoding network, the Transformer decoder and the action predictor to obtain an optimized control model; a deployment execution module, configured to generate a control action vector by processing real-time action positions, contact forces and visual images through the optimized control model in a deployment stage, and convert the control action vector into joint space control instructions to drive an execution mechanism.

[0010] Further, to achieve the above object, the present application also provides a computer device, which comprises a memory, a processor and a master-slave collaborative control program stored in the memory and executable on the processor, and the master-slave collaborative control program, when executed by the processor, implements the steps of the master-slave collaborative control method.

[0011] Further, to achieve the above object, the present application also provides a computer readable storage medium, which stores a master-slave collaborative control program, and the master-slave collaborative control program, when executed by a processor, implements the steps of the master-slave collaborative control method.

[0012] Beneficial effects: The application relates to the technical field of robot control, and discloses a control method, device, equipment and medium based on master-slave cooperation, which comprises the following steps: a master-slave cooperative control link is established, a multi-modal demonstration data set is constructed by synchronously collecting spatial pose coordinates, a contact force vector and a double-view image through a bilateral force-position closed-loop feedback mechanism; time synchronization alignment and normalization processing are performed on the multi-modal demonstration data set to form a standardized input sequence; action state vectors, contact force vectors and double-view images in the standardized input sequence are respectively processed through an action modal encoding network, a mechanical modal encoding network and a visual encoding network to obtain action features, force features and visual features; the action features, the force features and the visual features are fused into a multi-modal tensor sequence and input into a Transformer decoder to generate joint time sequence representation; a trained action prediction vector is output by the joint time sequence representation through an action predictor, a supervision loss function is constructed based on the trained action prediction vector and expert demonstration labeling, parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor are updated, and an optimized control model is obtained; in the deployment stage, real-time action positions, contact forces and visual images are input into the optimized control model to generate a control action vector, and the control action vector is converted into joint space control instructions to drive an execution mechanism. In the data acquisition stage, the application introduces multi-modal synchronous perception of spatial pose, contact force and visual image, combines time alignment and normalization preprocessing, ensures the uniformity and time sequence consistency of input data, extracts action, force and visual features through the action modal encoding network, the mechanical modal encoding network and the visual encoding network respectively, and performs feature fusion and time sequence modeling in the Transformer decoder, so that the coupling relationship between different modalities can be fully captured, the effective learning of multi-modal features and model optimization are realized through the supervision constraint of the trained action prediction vector and the expert demonstration labeling, and joint space control instructions are generated and issued in real time by using the optimized control model in the deployment stage, so that the action stability and generalization ability of the robot in complex interactive tasks are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0013] The application will be further described below in combination with the drawings and embodiments, and the drawings show: Figure 1 It is an application environment schematic diagram of the control method based on master-slave cooperation in an embodiment of the application. Figure 2 It is a flow schematic diagram of the control method based on master-slave cooperation in an embodiment of the application. Figure 3 It is a functional module schematic diagram of the control device based on master-slave cooperation in a preferred embodiment of the application. Figure 4 It is a structural schematic diagram of a computer device in an embodiment of the application. Figure 5This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0014] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0015] The master-slave collaborative control method provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will now be described in detail through specific embodiments.

[0016] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the master-slave collaborative control method provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0017] like Figure 2 As shown, the master-slave collaborative control method proposed in this invention includes the following steps: S10: Establish a master-slave collaborative control link, and synchronously collect spatial pose coordinates, contact force vectors and dual-view images through a bilateral force-position closed-loop feedback mechanism to construct a multimodal demonstration dataset;

[0018] In this embodiment, establishing a master-slave collaborative control link requires building a low-latency bidirectional communication channel between the master control terminal and the slave terminal. This link is typically implemented using Ethernet or industrial fieldbus, but can also be achieved through high-bandwidth wireless transmission. Its purpose is to ensure that commands from the master control terminal and feedback data from the slave terminal can be transmitted and responded to within milliseconds. During this process, the communication protocol needs to support multi-channel synchronous transmission of pose data, force feedback data, and image streams to avoid delays or data loss between different modal information.

[0019] Synchronous acquisition of spatial pose coordinates via a bilateral force-position closed-loop feedback mechanism typically relies on high-precision position sensors and joint encoders. Spatial pose coordinates refer to the combination of position and orientation in three-dimensional space, generally described by Cartesian coordinates and Euler angles or quaternions. The encoder outputs joint angle data in real time, which, combined with the robot's kinematic model, allows calculation of the spatial pose of the end effector. The master control unit generates a position compensation signal based on the difference between this pose and the pose fed back from the slave end, thereby achieving motion mirroring.

[0020] The acquisition of the contact force vector relies on a six-dimensional force sensor installed at the end effector or joint, which includes force components along three-axis directions and torque components around three-axis. After analog-digital conversion and filtering, the signal of the force sensor can reflect the interaction between the robot and the environment in real time. The force vectors of the master and slave are compared, and the difference is used to generate a force compensation signal to achieve stable force feedback and compliant control.

[0021] The acquisition of the dual-view image is usually completed by two cameras arranged at different angles, which respectively collect image sequences of the target scene. The cameras need to be calibrated to ensure the synchronization of the images in the time axis and the geometric consistency in the space. The introduction of the dual-view image can enhance the depth perception of the scene and provide redundant information sources for subsequent multi-modal learning. The image acquisition process needs to ensure sufficient frame rate to meet the synchronization requirements with the pose and force signals.

[0022] The construction of the multi-modal demonstration dataset relies on the time-synchronous acquisition of the above-mentioned pose coordinates, contact force vectors, and dual-view images. All data are aligned according to a unified timestamp to form a continuous data stream. The organization form of the dataset is usually a time sequence sample, and each time point contains three types of information: pose, force vector, and image. This dataset not only retains the state information in the human-robot interaction process but also covers the whole process of action and environmental contact, providing a complete data basis for subsequent imitation learning.

[0023] In specific implementation, the master-slave collaborative control link can be built based on industrial Ethernet or high-speed wireless communication modules. In medical operations with high precision requirements, wired mode can be selected to reduce delay and interference, while in remote collaboration scenarios, wireless mode can be selected to improve the operation range. The acquisition of spatial pose coordinates can be achieved by combining joint encoders and inertial measurement units, or assisted by external optical positioning systems. The acquisition of the contact force vector can be completed by a six-dimensional force sensor, or indirectly derived through motor current and joint torque estimation algorithms. The acquisition of the dual-view image can use a binocular stereo camera, or a combination of two monocular cameras, and if high-precision depth information is required, data fusion can also be combined with structured light or laser radar.

[0024] This embodiment can ensure the real-time and consistency of data when acquiring spatial pose, contact force, and visual image by establishing a master-slave collaborative control link and introducing a force-position dual-closed-loop feedback mechanism. The multi-modal demonstration dataset constructed contains three types of complementary information: position, force sensation, and vision. Compared with data relying on only a single modality, it can more truly reflect the human-robot interaction and task execution process, thereby providing a complete and high-quality data basis for subsequent learning and control.

[0025] S20, performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; In this embodiment, in the process of performing time synchronization alignment on the multi-modal demonstration data set, first, the timestamps of the pose coordinates, the contact force vectors, and the dual-view images need to be extracted. Due to the sampling frequency difference of different acquisition devices, for example, the encoder and the force sensor can reach hundreds of hertz, while the camera frame rate is usually in the tens of hertz range, therefore, the timestamps must be resampled and interpolated based on a unified clock reference. The resampling process can be realized by linear interpolation, polynomial interpolation or Kalman filtering, etc., so that the different modal data are strictly aligned on a unified time axis, thereby avoiding the accumulation of errors caused by inconsistent information.

[0026] In the normalization processing link, the spatial pose coordinates need to be standardized and converted. The common method is to unify all data to the robot base coordinate system or the world coordinate system, eliminating the bias caused by different sensor installation positions or initial attitudes. The conversion usually relies on the homogeneous transformation matrix to map the data in the joint coordinate system to the target reference system, thereby ensuring that all position and attitude data have a unified geometric meaning.

[0027] The normalization of the contact force vector needs to be scaled for the sensor range, mapping the force and torque values output by different sensors to the same numerical interval, such as [-1, 1] or [0, 1]. This operation can avoid the inconsistency of amplitude caused by different sensor sensitivities, so that the subsequent model training will not be biased towards a certain type of signal. When normalizing, linear scaling is usually used, and standardization can also be combined with sensor calibration parameters to eliminate the effects of temperature drift or zero-point offset.

[0028] The standardization processing of the dual-view images includes channel normalization and size normalization. Channel normalization refers to mapping the pixel values of the RGB channels to a distribution with zero mean and unit variance by subtracting the mean and dividing by the standard deviation, so that the image data has better numerical stability. Size normalization adjusts images of different resolutions to a uniform size through interpolation, so that they can be batched into a deep learning model for subsequent processing. In order to ensure the geometric consistency across the views, the images usually need to be corrected for camera intrinsic parameters and distortion before normalization.

[0029] Finally, the pose coordinates, contact force vectors, and dual-view images that have undergone time alignment and normalization processing are encapsulated in time order to form a standardized input sequence. The input sequence is a time-ordered sample set, each time slice contains position, force, and image data, and has been numerically standardized and geometrically consistent, providing stable input conditions for subsequent feature extraction and time series modeling.

[0030] The embodiment can eliminate the timing difference and numerical inconsistency between different sampling devices by time alignment and normalization processing on the multi-modal demonstration data set, so that the pose, force and image information are input together on a unified time axis with a unified numerical scale to form a standardized input sequence. This preprocessing ensures the stability and accuracy of subsequent feature extraction and timing modeling, thereby avoiding model convergence difficulties or generalization performance degradation caused by different synchronization or amplitude between modalities.

[0031] S30, processing the action state vector in the standardized input sequence through an action modal encoding network to obtain an action feature, processing the contact force vector in the standardized input sequence through a mechanical modal encoding network to obtain a force feature, and processing the dual-view image in the standardized input sequence through a visual encoding network to obtain a visual feature; In the embodiment, when processing the action state vector in the standardized input sequence, a special action modal encoding network is needed to extract features that can represent dynamic motion patterns. The action state vector is usually composed of spatial position, attitude angle, velocity or acceleration, etc., and can be represented as a time series matrix in form. In order to extract discriminative representation from these high-dimensional continuous variables, the action modal encoding network can use a multi-layer perceptron to transform the input through fully connected layers and nonlinear activation functions layer by layer, so that the input action state vector is mapped to the action feature in the low-dimensional latent space. This process can capture the change rule of the pose over time and eliminate the difference in dimension and scale of the input data.

[0032] The processing of the contact force vector is completed by the mechanical modal encoding network. The contact force vector contains force along three directions and torque along three directions, which is the standard form of six-dimensional force sensing data. This kind of data is usually accompanied by noise and unstable fluctuations during acquisition, so the mechanical modal encoding network needs to have the ability of noise filtering and robust expression. A common implementation is to input the contact force vector into a fully connected neural network, or to capture the continuity of force and torque in the time dimension through a one-dimensional convolution network for local reception of time slices. The encoded output force feature and the action feature maintain the same dimension, so as to be uniformly processed in the subsequent multi-modal fusion.

[0033] The processing of the dual-view image needs to be completed by a visual encoding network. The visual encoding network can use a pre-trained convolutional neural network or a visual Transformer to extract deep features from the input dual-view image. The features of the two views are not simply superimposed, but the attention weights are calculated through a cross-view self-attention mechanism, and then the complementary information between different views is emphasized. For example, the occluded area in one view may be visible in another view. Through the weighted summation of cross-view attention, a more complete visual feature expression can be obtained. The finally output visual features, action features and force features are kept in a similar representation space, ensuring that they can be effectively spliced when multi-modal fusion is performed.

[0034] By encoding the action state vector, contact force vector and dual-view image into action features, force features and visual features respectively, the embodiment can make different modal data have a unified numerical representation and semantic dimension before entering the subsequent fusion module. This processing not only improves the matching between modalities, but also significantly enhances the representation ability of the model in complex interaction tasks, so that the robot can understand kinematic information, mechanical information and environmental visual information at the same time, thereby providing stable input conditions for subsequent time series modeling and control.

[0035] S40, the action features, the force features and the visual features are fused into a multi-modal tensor sequence and input into a Transformer decoder to generate a joint time series representation; In the embodiment, the action features, force features and visual features come from different modalities in numerical form, and have different dimensions and distributions, so they need to be fused before entering the unified modeling structure. The first step of fusion is usually to concatenate the action features, force features and visual features along the feature dimension, thereby obtaining a multi-modal tensor representation corresponding to a time. The concatenation method can be a simple vector-level connection, or the features of each modality can be dimensionally aligned through linear projection before being concatenated, to ensure that the tensor sequence has consistency in the numerical space. The multi-modal tensor sequence obtained by concatenation can contain kinematic, mechanical and visual information at the same time.

[0036] In order to let the sequence data retain the time sequence information, position encoding needs to be added to the multi-modal tensor sequence. The common method is to use sine and cosine functions to generate encoding signals of different frequencies, and add them to the original features element by element, so that the model can distinguish different time steps without introducing additional learnable parameters. The position encoding provides position information across time, so that the subsequent decoder can capture long-distance dependencies in the sequence.

[0037] After obtaining the multi-modal tensor sequence with position encoding, it needs to be input into the Transformer decoder. The Transformer decoder processes the input features through the multi-head attention mechanism. Each attention head generates query, key and value matrices on the input tensor, and calculates the attention weight through the dot product of query and key, and then weights the value matrix. The results of multiple attention heads are spliced together, which can simultaneously focus on different dependencies in the multi-modal sequence. The feedforward neural network in the decoder performs nonlinear transformation and dimension expansion on the fused results, layer normalization ensures the stability of training, and residual connection is used to avoid gradient disappearance and preserve the original information. After a series of operations, the final output is the joint temporal representation. This representation not only integrates the information of action, force and vision three modalities, but also has temporal continuity and cross-modal dependency.

[0038] In this embodiment, by fusing action features, force features and visual features into a unified multi-modal tensor sequence and inputting it into the Transformer decoder, three types of information can be retained in a single representation, and the multi-head attention mechanism can be used to capture cross-modal and cross-temporal dependencies. This processing method avoids the limitations of single modal control accuracy, enhances the perception and reasoning ability of the model for complex interaction tasks, and enables the joint temporal representation to provide a stable and rich information source for subsequent action prediction and control.

[0039] S50, outputting the joint temporal representation to the action predictor to obtain a training action prediction vector, constructing a supervised loss function based on the training action prediction vector and expert demonstration label, updating parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor to obtain an optimized control model;

[0040] In this embodiment, after the joint temporal representation enters the action predictor, it needs to map the high-dimensional temporal features to the specific action output space. The action predictor is usually composed of a linear mapping layer and a nonlinear activation function. The former compresses or expands the joint temporal representation to the target action dimension, and the latter introduces nonlinear expression ability to enhance the prediction accuracy. The prediction result is the training action prediction vector, which is consistent in dimension with the expert demonstration label and can be directly used for supervised learning. The expert demonstration label is usually derived from artificially collected trajectories or task execution results, which is formatted as an action vector sequence and used as a standard reference for training. The construction of the supervised loss function usually takes the mean square error as the core measure, that is, the sum of the element-wise square differences between the training action prediction vector and the expert demonstration label is calculated, and a regularization term is added as necessary to ensure temporal continuity or suppress overfitting.

[0041] During the training process, the supervised loss function is used as the optimization target, and the error signal is transmitted layer by layer to the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor through the back propagation algorithm, so as to calculate the gradient of each parameter. The parameter updating process generally adopts a gradient descent type optimization algorithm, such as Adam or RMSprop, uses a learning rate to control the updating amplitude, and iteratively optimizes until the loss function converges. The whole process realizes the learning of the model from the joint time sequence representation to the action prediction, so that the network can accurately output the action prediction vector conforming to the expert demonstration when inputting the multi-modal information. The finally obtained optimized control model can process real-time input and generate control actions in the deployment stage, providing a stable strategy for robot execution.

[0042] In different implementations, different loss function combination methods can be selected. For example, the mean square error can be used as the main loss function, and the first order difference or the second order difference of the action prediction vector in the time dimension can be added as a continuity constraint, so as to ensure the smoothness of the predicted trajectory. A weight factor can also be added to the mean square error to give higher weight to the prediction error at critical moments in the training, so as to improve the learning effect of high-risk actions. In the optimization method selection, the standard Adam optimizer can be used, or the stochastic gradient descent with momentum can be used to adapt to different training scales and data distributions. If the training data contains noise, the parameter range can be constrained by a regularization term in the loss function to reduce the risk of overfitting.

[0043] When adapting to different scenarios, if the sampling frequency of the expert demonstration annotation is high, interpolation or downsampling can be performed on the prediction results and the annotation sequence before loss calculation to ensure time alignment. If it is necessary to improve the prediction ability of the model for complex actions, multiple nonlinear structures can be added to the action predictor instead of using only a single linear layer. For training data containing different tasks, task weight parameters can be introduced into the loss function to balance the learning effect between different tasks.

[0044] The embodiment can establish a direct mapping relationship from multi-modal input to action output by inputting the joint time sequence representation into the action predictor and training it in a supervised comparison manner with the expert demonstration annotation, so as to realize accurate learning of the model for human demonstration behavior. The back propagation and parameter updating process enables the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor to be optimized cooperatively, so as to improve the accuracy and stability of action prediction, and the finally obtained optimized control model has the ability to generalize and execute tasks in complex scenarios.

[0045] S60, in the deployment phase, the optimization control model processes real-time action position, contact force and visual image input to generate a control action vector, and converts the control action vector into joint space control instructions to drive the actuator.

[0046] In this embodiment, in the deployment phase, the optimization control model undertakes the task of real-time reasoning, and the input includes three types of data including action position, contact force and visual image. The action position is generally obtained through the robot joint encoder, external positioning system or inertial measurement unit, and is usually expressed in Cartesian space coordinates and attitude angle; the contact force is derived from a six-dimensional force sensor or torque sensor, and the value includes the action force and action torque; the visual image is collected by a binocular or multi-view camera, providing visual information of the environment and objects. These data are calculated through multi-modal fusion of the optimization control model, and a control action vector is output. The control action vector contains the target displacement, attitude adjustment and force control reference of the robot end effector, and is a key intermediate quantity connecting perception and execution.

[0047] The control action vector needs to be further converted into joint space control instructions to drive the actuator. The conversion process usually relies on an inverse kinematics solver to decompose the end pose and force control target into joint angle, velocity or torque instructions. In actual implementation, inverse kinematics solving can be realized by analytical method for high-speed mapping, or numerical iterative method for improved adaptability to complex mechanisms. The joint space control instructions obtained are input to the joint actuator, which then adjusts the motor torque or speed according to the input to drive the actuator to output actions, completing the entire perception-decision-execution closed loop.

[0048] In different embodiments, different data acquisition and control instruction generation methods can be used. For example, a task planning based on Cartesian space can be used to directly convert the control action vector into end pose, and then obtain joint angle instructions through inverse kinematics calculation; a direct prediction based on joint space can also be used, so that the control action vector itself outputs joint angle or velocity, thereby omitting the inverse kinematics link to improve real-time performance. In terms of force control, impedance control strategy can be used to add expected force component in the control action vector, so that the actuator has compliant characteristics when interacting with the environment; force-position hybrid control can also be used to control the execution force of part of the degrees of freedom, and the remaining degrees of freedom remain in position control, to adapt to scenarios requiring flexible contact.

[0049] The embodiment can realize real-time perception and adaptive control of the robot in a complex dynamic environment by processing real-time action positions, contact forces and visual image data using the optimized control model in the deployment stage and converting the generated control action vector into joint space control instructions. This mechanism ensures that the actuator can complete smooth and accurate task actions under the guidance of multi-modal input, making the robot have higher operation flexibility and safety in uncertain environments.

[0050] In one embodiment, the above step S10 comprises: S101, establishing a physical connection channel between the master control end and the slave robot end; S102, collecting real-time position state and contact force information of the master control end to obtain a first spatial pose coordinate and a first contact force vector; S103, receiving real-time position state and contact force information feedback from the slave robot end to obtain a second spatial pose coordinate and a second contact force vector; S104, calculating the position error of the first spatial pose coordinate and the second spatial pose coordinate; S105, calculating the contact force error of the first contact force vector and the second contact force vector; S106, generating a position compensation control signal based on the position error; S107, generating a force compensation control signal based on the contact force error; S108, driving the slave robot end to realize action mirroring through the position compensation control signal and the force compensation control signal; S109, synchronously collecting spatial pose coordinates, contact force vectors and double-view images during the mirroring action process; S110, constructing a multi-modal demonstration data set based on the spatial pose coordinates, contact force vectors and double-view images.

[0051] In this embodiment, the master-slave cooperative control link establishes a stable and low-latency communication and control coupling relationship for the two end entities, the master control end undertakes human-computer interaction and expected action generation, and the slave robot end executes the action and returns the state. The link implementation includes hardwiring or industrial Ethernet, time synchronization mechanism, data packaging and verification, and exception recovery process. Time synchronization can be achieved through unified clock source or timestamp alignment, so that the master end sampling sequence and the slave end feedback sequence correspond on the same time axis, and the synchronization accuracy directly affects the error calculation and closed-loop effect. The data channel distinguishes between control channels and observation channels, the control channel transmits position compensation control signals and force compensation control signals, and the observation channel carries the first spatial pose coordinates, the first contact force vector, the second spatial pose coordinates, the second contact force vector and the dual-view image. To ensure robustness, the communication layer introduces packet retransmission, sliding window and message sequence number management to avoid pseudo errors caused by out-of-order data.

[0052] The bilateral force-position closed-loop feedback mechanism parallelly couples the position loop and the force loop to form bidirectional energy interaction. The position loop takes spatial pose coordinates as the object, forms a position error through the difference between the master end expectation and the slave end feedback, and then generates a position compensation control signal after gain shaping, amplitude limiting and filtering. The force loop takes the contact force vector as the object, forms a contact force error through the difference between the master end haptic expectation and the slave end haptic feedback, and generates a force compensation control signal after impedance or compliance mapping. The two loops share a unified sampling and time delay compensation strategy, and when necessary, introduce scattering transformation or passivity layer to ensure stability and energy consistency of bilateral interaction under communication delay and environmental uncertainty. To eliminate high-frequency noise, a bandpass or lowpass filter is configured at the front end of the force loop, and a differential smoothing and acceleration limiting is introduced in the position loop channel to prevent the compensation signal from exciting structural resonance.

[0053] The spatial pose coordinates use six-dimensional representation, including three-dimensional position and three-dimensional attitude, and the coordinate system definition uses the robot base coordinate system and the end effector coordinate system, which are related through the external parameter matrix obtained by calibration. The acquisition implementation relies on the human-computer interface device of the master control end and the encoder and external parameter solving module of the slave robot end, which respectively output the first spatial pose coordinates and the second spatial pose coordinates. To ensure consistency across devices, the pose data is unified in units and parameterized in attitude before being sent, and the common practice is to unify Euler angles into quaternions or quaternions into Euler angles, and maintain a single representation in the pipeline. The pose timestamp is written in alignment with the hardware clock to eliminate the false displacement caused by jitter.

[0054] The contact force vector is represented by a six-dimensional force torque, including three-axis force and three-axis torque. The master control end can give the first contact force vector through the force handle or the six-dimensional force sensor, and the slave robot end can give the second contact force vector through the end flange or joint side sensor. The sensor link must be zero offset corrected, temperature drift compensated and range calibrated, and online self-calibration is used when necessary to suppress long-term drift. In order to improve the contact quality evaluation ability, the contact stability index and the friction cone consistency index can be calculated in parallel with the force vector, which are used for tactile anomaly detection and safety limiting. The force vector and the pose together constitute the state description of the interaction with the environment, providing complete geometric and mechanical information for motion mirroring and data acquisition.

[0055] The physical connection channel covers mechanical fixation, power supply and signal transmission. Mechanical fixation ensures that the master control end and the sensor coordinate system do not drift, the power supply module prevents transient voltage drop from affecting sensing accuracy, and signal transmission uses shielded twisted pair or optical fiber to reduce electromagnetic interference. After the connection channel is established, link self-checking is immediately performed, including loopback test, redundant channel switching and delay measurement, and connection credentials are generated for subsequent data record tracing.

[0056] The first spatial pose coordinates and the first contact force vector come from the master control end, representing the expected motion and expected touch; the second spatial pose coordinates and the second contact force vector come from the slave robot end, representing the real execution state and real touch. The four types of quantities are paired on a unified time axis to form the input for error calculation. Position error is obtained by corresponding component difference, and attitude component difference uses Lie group or quaternion difference to avoid gimbal singularity. Contact force error is obtained by the same dimension difference, and different gains can be applied in different axial directions according to task weight. Two types of errors are subjected to outlier rejection and sliding window smoothing before being sent to control shaping to prevent false triggering caused by instantaneous impact or image occlusion.

[0057] The position compensation control signal is derived from the position error and generated by proportional or proportional-integral-derivative law, and the gain matrix can be decoupled according to the axis or automatically shaped according to the task. To improve the adaptability to unknown compliant environment, an adaptive term based on environment stiffness estimation can be added to the position loop to make the compensation produce different displacement responses to soft and hard environments. The force compensation control signal is derived from the contact force error, which can be converted into displacement or velocity command through impedance mapping or compliance mapping, or directly added to the motor torque layer through torque feedforward. The two compensation signals are coordinate-consistent and saturated before being sent to the slave robot end to ensure that the execution mechanism does not enter dangerous working conditions.

[0058] Motion mirroring refers to the slave robot end replicating the master control end hand movement and tactile force exertion under bilateral force-position closed loop. The execution process first fuses the position compensation control signal and the force compensation control signal to obtain the end position and contact force, and then solves the joint target through inverse kinematics and kinematics constraints. The joint controller adopts a speed or torque inner loop plus position outer loop structure to obtain fast and controllable dynamic response. To ensure mirror stability, the system continuously monitors the end energy flow and contact stability indicators, and automatically adjusts the compensation weight when detecting overload or slipping trend to achieve force-position redistribution.

[0059] Synchronous acquisition covers spatial pose coordinates, contact force vectors, and dual-view image data. Acquisition triggering is completed by a unified clock or a hard trigger pulse. Dual-view images come from imaging devices fixed at different angles, and internal and external parameters are obtained through calibration to ensure spatiotemporal registrability of the two views. Image data, pose, and force data are concatenated through a unified data recorder, using lossless or light compression encoding, recording timestamps, frame numbers, device identifiers, and verification information. To improve training availability, the recorder generates an acquisition summary online, including coarse labels of effective contact segments, non-contact segments, and occlusion segments, facilitating subsequent filtering.

[0060] The construction of the multi-modal demonstration dataset is based on the three-channel data collected on site. The data organization uses time slices as the main index, and under a single time slice, it hangs the spatial pose coordinates, contact force vectors, and dual-view images. It also saves the first spatial pose coordinates, the first contact force vector, the second spatial pose coordinates, the second contact force vector, and the compensation signal, forming a complete master-slave interaction trace. The dataset performs consistency checking during construction, including temporal continuity, force-position physical consistency, and image exposure consistency. If abnormalities are found, it rolls back to the last consistent snapshot to prevent mixing of unusable samples. To support subsequent labeling and retrieval, the dataset is accompanied by task metadata and device configuration lists to ensure traceable and reproducible experimental conditions across batches.

[0061] This embodiment couples the human hand expectation and the robot execution in the same time axis and the same physical quantity space by establishing a master-slave collaborative control link and introducing a bilateral force-position closed loop. Position error and contact force error are independently shaped to drive motion mirroring. The spatial pose coordinates, contact force vectors, and dual-view images collected are synchronized under the same tactile context. The multi-modal demonstration dataset obtained contains three types of complementary information: geometric trajectory, tactile interaction, and visual perception. Compared with position-only or vision-only recording, it more completely covers the interaction characteristics of humans and the environment. The control constraints of data consistency and energy consistency reduce the influence of noise and lag, reduce invalid samples and pseudo-labels, and directly support the convergence speed and generalization robustness of subsequent imitation learning.

[0062] In one embodiment, the above step S20 comprises: S201, extract timestamp information in the multi-modal demonstration data set; S202, align the spatial pose coordinates, contact force vectors and dual-view images to a unified time axis according to the timestamp information; S203, perform coordinate system standardization conversion on the spatial pose coordinates to obtain standardized spatial pose coordinates; S204, perform force sensor range normalization on the contact force vectors to obtain normalized contact force vectors; S205, perform channel standardization processing on the dual-view images to obtain standardized dual-view images; S206, encapsulate the standardized spatial pose coordinates, the normalized contact force vectors and the standardized dual-view images into a standardized input sequence in time sequence.

[0063] In this embodiment, the extraction of timestamp information is carried out around three data channels, which are spatial pose coordinates, contact force vectors and dual-view images. Each channel carries a hardware clock or system clock label during the acquisition stage. The timestamp source can be a master clock broadcast synchronization, a PTP hardware time synchronization or a recording terminal unified time base. To ensure consistency across devices, first perform monotonicity check and jitter removal on the timestamp, remove the reverse order and repeated labels; for the channel with time base drift, use the synchronization pulse or known event sequence to perform linear or segmented affine correction to generate a high-precision timestamp sequence consistent with the reference time base.

[0064] Based on the timestamp, the alignment to the unified time axis is completed. The unified time axis is constructed with fixed sampling intervals, and the interval depends on the target frequency of the control and learning loop. The spatial pose coordinates and the contact force vectors are interpolated and resampled at the target time point. The quaternion spherical linear or Lie group exponential mapping is used for pose interpolation, and the cubic spline or constrained spline is used for position component to suppress overshoot; the contact force vector is smoothed in a short window before interpolation to weaken high-frequency noise. The dual-view images use nearest neighbor time matching and set time window constraints. If there is a small phase difference between the two views, use an external participant and a motion model to compensate for the time difference before matching. For the case of missing frames and missing samples, use a time mask to record the unavailable positions and limit the loss contribution of the subsequent learning stage; if the continuous loss exceeds the threshold, it is marked as an unalignable segment and does not enter the encapsulation link.

[0065] The coordinate system standardization conversion of spatial pose coordinates aims to unify the reference coordinates. The calibration of robot base coordinates, end coordinates, main control end coordinates and two-way camera coordinates is completed in advance to obtain the rigid body transformation of each to the reference coordinates. The pose sequence is first unified in units and dimensions, and then the coordinate transformation is completed by left multiplication of the external parameter matrix; the attitude representation is normalized by quaternion to eliminate numerical cumulative error, and the non-singular axis sequence is selected under the demand of Euler angle representation. To improve the consistency across devices, soft / hard iron compensation or tool coordinate automatic estimation can be introduced, and in the presence of end tool, the end pose is further corrected by tool calibration matrix.

[0066] The range normalization of contact force vectors handles the differences in sensor range and sensitivity. First, zero drift estimation and zero point rollback are performed within each collection batch, and the bias is estimated using static segments and subtracted from the full sequence; then the original readings are mapped to physical quantities according to the calibration table, and the three-axis force and three-axis moment are scaled respectively. To obtain the normalized expression, map the six-dimensional quantity to the symmetric interval according to the range interval, while preserving the original dimension for traceability; introduce soft limiting and inversion correction for saturation and shear distortion, and record the label without directly discarding. On the spectrum level, low-pass or zero-phase filtering is used to suppress quantization noise and power ripple, and the transient state is preserved at the contact switching edge to facilitate learning of the tactile transition.

[0067] Channel standardization of dual-view images to reduce brightness and color distribution differences. The image is first de-warped and cropped to make the imaging and external parameters consistent; then mean removal and variance scaling are performed on the RGB channels, the mean and variance can be based on the entire library statistics, or based on the collection batch statistics; in environments with large changes in illumination, adaptive histogram equalization can be added to stabilize the contrast. The resolution is unified to a predetermined size, and the interpolation algorithm preserves the edges to reduce blurring; the color space remains consistent to prevent bias introduced by white balance differences across devices. The time and geometry are consistent between the two views, and the parallax damage caused by single-view independent cropping is prohibited, and if necessary, the epipolar constraint is used to review the registration quality.

[0068] The encapsulation of standardized input sequences constructs data records around the time step as the main index. Each time step contains standardized spatial pose coordinates, normalized contact force vectors, standardized dual-view images, and time masks and availability markers. To adapt to batch training, sliding window or fixed length window is used to generate sample segments, and when the sequence length is insufficient, the tail or head is padded, and the mask is used to distinguish between real data and padded values. The sequence retains the source identification, external parameters and gain version number to ensure that the experimental conditions can be reproduced and the data processing chain can be traced. Consistency checks are performed during the encapsulation phase, including time continuity, cross-modal synchronization error threshold and geometric-tactile physical consistency, and segments that do not meet the constraints are automatically excluded or rolled back to the last available snapshot.

[0069] This embodiment transforms raw multi-source asynchronous data into a learnable sequence under the same time axis, reference coordinates, dimensions, and visual statistics through a combination of time synchronization, geometric and mechanical unification, image channel consistency, and structured encapsulation. Spatial pose coordinates are unified and orientation standardized to avoid singular interpolation; contact force vectors are zero-drift corrected and range normalized to improve cross-device comparability; dual-view images are standardized through channel normalization and geometric consistency to preserve cross-view constraints; and unified time axis and mask management reduce the impact of frame drops and latency on learning stability. The resulting standardized input sequence is more suitable for downstream encoding and temporal modeling in terms of numerical scale, statistical distribution, and temporal alignment, reducing invalid variance and cross-modal bias, improving the convergence speed and generalization robustness of parameter learning, and suppressing noise and drift while preserving details and tactile transitions.

[0070] In one embodiment, step 30 above includes: S301, input the action state vector in the standardized input sequence into a multilayer perceptron network, perform feature transformation through a fully connected layer and a nonlinear activation function, and output action features; S302, the contact force vector in the standardized input sequence is input into a fully connected neural network, feature signals are extracted through a linear transformation layer, and force features of the same dimension as the action features are output; S303, Input the dual-view images in the standardized input sequence into the pre-trained visual model, and extract the deep features of the first view and the deep features of the second view; S304, perform cross-view self-attention weight calculation on the deep features of the first viewpoint and the deep features of the second viewpoint to obtain cross-view attention weight; S305, based on the cross-view attention weight, perform a weighted summation of the deep features of the first view and the deep features of the second view, and output the weighted summation result as a visual feature.

[0071] In this embodiment, the action state vector in the standardized input sequence enters the action modality coding network (AMC) using time steps as indices. The vector contains a combination of position, velocity, and pose parameters and their derived values. It first undergoes a linear mapping layer to achieve dimensionality regularization and numerical scale alignment, and then enters a multilayer perceptron for nonlinear sequential transformation. Each layer employs a cascaded structure with a fully connected weight matrix and element-wise activation. A common configuration is a stack of two to four layers with residual bypass and layer normalization, used to stabilize gradient propagation and suppress internal covariate shifts. For temporally incomplete data points, a validity mask carried with the sequence is used to mask the gradient and forward contribution of the corresponding channel, preventing outliers from affecting parameter convergence. The output uses linear projection to map the latent space features to a unified dimension, forming action features.

[0072] The contact force vector in the standardized input sequence enters the mechanical modal encoding network. The six-dimensional force and moment components have completed zero drift compensation and range normalization before entering the encoding, and further short window whitening is performed here to eliminate quantization steps and carrier frequency ripples. The network structure is mainly composed of fully connected layers, and the first layer of weights is initialized according to the force and moment groups to reflect the different physical dimensions. The subsequent layers introduce gating units to suppress the abnormal response of saturated segments. In order to maintain isomorphism with the action features in the multi-modal fusion stage, the terminal linear layer aligns the output dimension to the same size as the action features; during the alignment process, the amplitude statistics are consistent, and the cross-modal energy balance is achieved through the learnable calibration of the scale parameter and the bias parameter. If there is a high-frequency impact or sliding contact, an additional short path is added in parallel to extract high-frequency residuals, which are then aggregated with the main branch to improve the expression ability of transient tactile events, and finally the force features are obtained.

[0073] The dual-view images in the standardized input sequence enter the visual encoding network. Each view image first undergoes geometric distortion correction and resolution regularization, and then extracts hierarchical features through a pre-trained convolutional or Transformer-based visual backbone. The backbone network outputs one or more global semantic vectors at the last layer, which can obtain view-level deep features through class label vectors or global average pooling, denoted as first view deep feature and second view deep feature, respectively. In order to adaptively allocate information weights between the two views, a cross-view self-attention weight calculation module is constructed. First, project the two features into a shared comparison space, calculate the similarity matrix or correlation score, and then generate non-negative weights through soft normalization. To avoid bias caused by lighting, occlusion or motion blur, confidence gating is introduced in the weight calculation. The confidence can be estimated by image clarity, edge density or detected occlusion indicators. The gating factor is multiplied by the attention weight and then normalized to obtain the cross-view attention weight. Then, linearly weight and sum the two deep features to obtain the visual feature. In order to participate in subsequent cross-modal fusion, the visual feature is mapped to the same or splicable dimension as the action feature through a linear reshaping layer; if the visual backbone outputs a multi-token structure, first perform weighted aggregation and then complete dimension reshaping.

[0074] The three-way encoding process is executed in parallel at the same time step and inherits the time mask and synchronization label in the standardized input sequence. The encoding output maintains a one-to-one index relationship in the time dimension, ensuring that the subsequent fusion stage can be spliced or attention calculated at minimal cost. To improve consistency across batches and across acquisition batches, a statistical normalization layer can be added at the end of the three-way network to linearly calibrate the output with the mean and variance aggregated during training, while maintaining stability during inference. Joint training or phased freezing strategies can be used for parameter update, and the current explanation focuses on the structure and data flow of the feature extraction path without limiting the specific training schedule.

[0075] The nonlinear embedding of the action state vector through the multilayer perceptron can preserve the coupling relationship between trajectory dynamics and posture in a unified dimension, the expression of the contact force vector obtained through the special fully connected and gated structure is sensitive to both transient tactile and steady pressing force, and the visual feature obtained through the pre-trained backbone and cross-view self-attention weight processing of the dual-view image can adaptively focus on key information under the conditions of occlusion and parallax change. The consistency of the three-way output in dimension and time alignment provides stable input for subsequent fusion and time series modeling, reduces the estimation bias caused by cross-modal scale mismatch and asynchronous sampling, and improves the learning convergence speed and generalization robustness in complex interaction scenarios.

[0076] In one embodiment, the above step S40 comprises: S401, concatenating the action feature, the force feature and the visual feature along the feature dimension in the Transformer decoder to form a multi-modal tensor sequence; S402, adding sinusoidal position encoding information to the multi-modal tensor sequence; S403, dividing the multi-modal tensor sequence with added sinusoidal position encoding information into multi-head attention subspaces; S404, calculating a query matrix, a key matrix and a value matrix in each attention subspace in the multi-head attention subspace; S405, calculating an attention weight distribution based on the similarity of the query matrix and the key matrix; S406, obtaining a fused feature by weighted fusion of the value matrix according to the attention weight distribution; S407, inputting the fused feature into a feedforward neural network to update the feature representation through nonlinear transformation, and obtaining an updated feature; S408, performing layer normalization operation on the updated feature to obtain a normalized feature; S409, integrating the multi-modal tensor sequence with added sinusoidal position encoding information and the normalized feature through residual connection in the Transformer decoder to obtain a residual integrated feature; S410, outputting the residual integrated feature as a joint time series representation.

[0077] In this embodiment, the action feature, the force feature, and the visual feature are concatenated in the feature dimension after being aligned with the same time index to form a multi-modal tensor sequence. Assuming that the batch size is B, the time length is T, the action feature dimension is d_a, the force feature dimension is d_f, the visual feature dimension is d_v, and the concatenated feature dimension D is equal to d_a plus d_f plus d_v, the shape of the multi-modal tensor sequence is B times T times D. To avoid the distribution breakpoint formed by the numerical scale and statistical difference of different modalities at the concatenation position, the outputs of each modality before concatenation are subjected to linear shaping and amplitude calibration, so that the mean and variance of each dimension after concatenation are in a controllable range that can be learned, thereby ensuring the numerical stability of subsequent attention calculation.

[0078] The multi-modal tensor sequence is superimposed with sinusoidal position encoding information to explicitly inject the time sequence. For any time index pos and even-odd dimension index i, the position encoding is constructed according to the sin and cos functions, the sin term corresponds to the even dimension, the cos term corresponds to the odd dimension, the frequency covers the full dimension according to the exponential scale, and the encoding tensor and the multi-modal tensor are added element by element to obtain a position enhanced sequence. The use of deterministic trigonometric function construction ensures that the model still retains phase continuity when extrapolating time steps, which is suitable for alignment and comparison across sequence lengths. When T is not fixed, the encoding is generated by slicing according to the current length, avoiding the introduction of invalid phases by padding bits.

[0079] The position enhanced sequence is divided into multiple attention subspaces. The number of attention heads H and the key query dimension d_k and value dimension d_vh of each head are selected so that H times d_vh is equal to D, ensuring that the concatenation of parallel heads still returns to the original dimension. For each attention head, an independent linear projection matrix is used to generate a query matrix Q, a key matrix K, and a value matrix V, with shapes of B times T times d_k, B times T times d_k, and B times T times d_vh, respectively. If there are invalid time bits (generated by the time synchronization stage), a synchronization time mask is transmitted, which is broadcast to the corresponding positions of the attention score matrix and shielded with negative infinity to avoid invalid bits participating in weight normalization.

[0080] The scaled dot-product attention is calculated within each attention head. First, the transpose product of Q and K is calculated to obtain the score matrix, which is scaled by the square root of d_k to suppress the distribution variance expansion, and then the exponential normalization is performed in combination with the time mask and the optional causal mask to obtain the attention weight distribution. The weight distribution is multiplied by the corresponding value matrix to obtain the weighted output of each head, the outputs of all heads are concatenated in the feature dimension and mapped back to the dimension D through linear mapping to obtain the fused feature. If the sequence contains time segments with significant differences in cross-modal semantics, a gating factor can be introduced at the projection layer, which is generated from the confidence estimates of each modality and acts on the value matrix or the head output, thereby adaptively suppressing low-confidence channels in the fused feature.

[0081] The fused features enter a feed-forward neural network to complete the time-step-wise nonlinear reconstruction, which includes two linear mappings with intermediate activations, and the hidden layer dimension can be r times D (r is an amplification factor), the first linear layer improves the expression capacity, the activation function introduces nonlinear separability, and the second linear layer restores the dimension D. To enhance numerical stability, the output of the first linear layer can use a higher precision data type or range clipping to avoid extreme values causing statistical distortion in the subsequent normalization stage. The output of the feed-forward network is the updated feature.

[0082] The updated feature is subjected to layer normalization in the feature dimension to obtain normalized features. The normalization is calibrated by a learnable affine transformation of the mean and variance, and a stabilizing term is added in the denominator to avoid instability caused by near-zero variance. The layer normalization is calculated independently in the time dimension and position by position, which adapts to the distribution differences of different time steps while maintaining batch consistency.

[0083] The position-enhanced sequence and the normalized features are integrated through a residual connection to form a residual integrated feature. To ensure element-wise additivity, the shapes of the two are checked and ensured to be consistent before integration, and if the feed-forward path has undergone dimension transformation, a matching linear projection is introduced in the residual branch. The residual operation preserves the original time position information carried by the position-enhanced sequence and the concatenated multi-modal straight-through feature, while superimposing the reconstruction components generated by attention and feed-forward transformation, forming a more discriminative temporal expression.

[0084] The residual integrated feature is output as the joint temporal representation. This representation preserves the index structure consistent with the input in the time dimension, and can be directly connected to subsequent action generation or policy inference modules. To improve engineering usability, optional dropout or noise suppression operators can be added at the output end to improve generalization, or multi-scale versions can be derived to be compatible with different downstream decoding window lengths. The calculation of the joint temporal representation is parallel in the entire batch, and the mask is throughout the QK score, weight normalization, and residual convergence to ensure stable output in variable length and missing sampling conditions.

[0085] This embodiment explicitly binds the temporal order and modal semantics in the same embedding space through the combination of multi-modal tensor sequences and sinusoidal position encoding information, the partition of multi-head attention subspaces allows the temporal dependence and cross-modal association to be modeled in different subspaces, the scaled dot product and mask mechanism improve numerical stability and robustness to missing positions, the feed-forward reconstruction provides nonlinear mixing capability for each time position and expands the expression dimension, the layer normalization and residual connection introduce transformed reconstruction information while preserving the original position enhancement information, alleviating the gradient decay and distribution drift problems of deep networks. The joint temporal representation obtained in this way still maintains stable relevance coding and separability under long time span, asynchronous perception, and modal confidence fluctuation conditions, providing higher learnability and better generalization ability for subsequent action prediction and control generation.

[0086] In one embodiment, the step S50 described above comprises: S501, inputting the joint time sequence representation into a linear mapping layer of the action predictor, processing by an activation function, and outputting a training action prediction vector; S502, calculating the mean square error between the training action prediction vector and the expert demonstration label as an action prediction loss term; S503, calculating a time continuity constraint term of the adjacent time step training action prediction vector; S504, adding the action prediction loss term and the time continuity constraint term to construct a supervised loss function; S505, calculating the gradient of the supervised loss function with respect to the parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor by using a back propagation algorithm; S506, updating the parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor based on the gradient of the parameters using an optimization algorithm; S507, iteratively performing parameter updating until a preset stopping condition is met to obtain an optimized control model.

[0087] In the embodiment, the joint time sequence representation is a time-sequenced high-dimensional vector sequence, which is derived from the previous fusion and decoding process and carries the correlation encoding of the action, force, and visual semantics on the time axis. The sequence maintains the same index structure in the batch dimension and the time dimension as in the data acquisition and alignment stage to ensure that the time-by-time pairing with the expert demonstration label can correspond one by one. To enter the training link, the joint time sequence representation is first sent to the mapping path of the action predictor according to the time position. The action predictor uses a linear mapping layer to connect a selectable nonlinear unit to realize the action space compression and boundary constraint. The weight matrix dimension of the linear mapping layer is projected from the representation dimension D to the action control dimension d_a, and the bias vector is consistent with the dimension after projection; the nonlinear unit selects a saturation type or a piecewise linear type according to the control interface to limit the output range, for example, the joint speed or torque is limited by the physical upper limit, and the saturation type is used to avoid numerical overflow. The mapping operation is independently executed at each time position to obtain a training action prediction vector sequence; if the strategy design is one-step look-ahead prediction, the output at the t position is aligned with the label at the t+1 time, and the data loader aligns the input and the label by one time when building the batch and removes the boundary positions at the beginning and end of the sequence that are not aligned by using a mask.

[0088] The deviation between the training action prediction vector and the expert demonstration label constitutes the measure of the imitation term. The expert demonstration label refers to the reference action sequence recorded through the master-slave collaborative link and processed through alignment and normalization, organized according to the same time index as the joint time representation. To construct the action prediction loss term, the mean square error is calculated in the batch dimension, time dimension, and action dimension. The mask is used to exclude invalid items when there are padding positions, ensuring that the gradient only comes from real observations. This error term measures the degree of point-by-point approximation in the action space, reflecting the static registration capability but not directly constraining the change rate of adjacent time.

[0089] The time continuity constraint term is used to suppress high-frequency jitter and improve trajectory executability. The two-norm square sum of the difference between adjacent times is constructed for the training action prediction vector, and the difference operation is limited to the position pairs allowed by the mask to avoid using out-of-bound or padding values. When the control interface requires strict smoothness, different smoothness weights are assigned to different action dimensions. The position-related components and the posture-related components can use independent weight coefficients, thereby imposing stronger constraints on the dimensions that affect execution stability. If there are different control periods on the deployment side, the amplitude correction is performed on the time interval of adjacent positions according to the target control period before calculating the difference, so that the continuity term is consistent with the real time scale.

[0090] The supervised loss function is obtained by weighting and superimposing the action prediction loss term and the time continuity constraint term. The weight coefficient λ maintains dimensionless consistency with the unit, and the proportion of continuity constraint can be appropriately increased at the beginning of training to stabilize convergence, and gradually reduced in the middle and later stages to release the expression ability. This scheduling can be achieved through piecewise constants or cosine annealing, etc. The loss function is averaged in the batch dimension and the time dimension to obtain a scalar target, which facilitates the optimizer to maintain the comparability of gradient amplitude under different batch sizes and sequence lengths.

[0091] Backpropagation is unfolded from bottom to top on the computation graph. The linear mapping layer of the action predictor obtains the gradient from the loss function and returns to the weights and biases. The activation unit participates in chain differentiation and truncates or disposes of out-of-bound gradients. The gradient continues to pass through the residual, normalization, feedforward, and attention paths of the Transformer decoder, back to the projection matrix and attention weight, and then maps back to the action modal encoding network, mechanical modal encoding network, and visual encoding network through cross-modal splicing. To avoid unstable accumulation, amplitude clipping is applied during gradient backpropagation, limiting the global norm to a preset threshold. Numerical stability terms are preserved at layer normalization and attention scaling nodes to prevent gradient explosion or disappearance caused by extremely small variance or extremely large score. The mask also clears the gradient at the same position in the backward direction while shielding in the forward direction, ensuring that invalid time positions do not affect parameter updating.

[0092] The parameter update adopts an optimization algorithm based on adaptive moment estimation or a momentum type stochastic gradient method. Taking the adaptive moment estimation algorithm as an example, first, the exponential moving average of the first and second moments of the gradient estimation is calculated, and after bias correction, the learning rate is normalized and updated; different sub-networks can be divided into independent parameter groups and learning rate groups, and the action predictor and the decoder use a relatively higher base learning rate to accelerate convergence, and the encoding network uses a lower learning rate to protect the learned perceptual representation. Weight decay is applied to parameters other than normalization and bias to enhance generalization. The learning rate scheduling can adopt a warm-up and decay strategy, the learning rate is linearly increased in the warm-up stage to establish stable gradient statistics, and then decreases with the training step number to approach zero in a cosine or step manner. After each training step, the gradient cache is emptied to avoid the cumulative effect of historical gradients on subsequent steps; when a numerical anomaly is encountered, a rollback is triggered and the learning rate is reduced for retry.

[0093] The stop condition is determined based on the validation set indicator, the training loss plateau or the maximum round. The validation set uses the same time mask and alignment strategy as the training to calculate the weighted sum of the action prediction loss term and the continuity constraint term as the early stopping monitoring signal; if no significant improvement is triggered in multiple consecutive evaluation periods, the training is terminated in advance; it can also be combined with the two signals that the gradient norm is long-term lower than the threshold and the learning rate is reduced to the lower limit to determine together. After meeting the stop condition, the parameter snapshots of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor are derived to form an optimal control model for deployment and calling; the normalization statistics and position encoding configuration are recorded when the derivation is performed to ensure consistency in the inference stage.

[0094] The embodiment jointly drives the supervised learning of the time sequence representation to minimize the registration error of the action space and the smoothness cost of the adjacent time, so that the network can ensure the imitation accuracy while suppressing high-frequency oscillation, thereby obtaining a control trajectory that is easier for the actuator to stably track; the multi-component gradient is propagated end-to-end through the decoding and encoding paths, so that the action predictor, the Transformer decoder and the three types of encoding networks converge collaboratively under the same target, and the cross-modal correlation is continuously strengthened; the adaptive learning rate and the grouping update mechanism provide matching optimization steps for different sub-networks, and layer normalization, residual and gradient clipping suppress training instability, early stopping and weight decay reduce the risk of overfitting. The optimal control model obtained after training balances between action accuracy and time sequence smoothness, retaining both detail response capability and deployment-side execution stability and generalization margin.

[0095] In one embodiment, the above step S60 includes: S601, collecting real-time action position coordinates, contact force vectors and double-view visual images of the robot; S602, generate a control action vector by fusing the real-time action position coordinates, contact force vector and double-view visual images through the optimized control model; S603, obtain a current spatial pose coordinate of the robot; S604, calculate a displacement difference value of the control action vector and the current spatial pose coordinate; S605, calculate an attitude angle change amount of the control action vector and the current spatial pose coordinate; S606, convert the displacement difference value and the attitude angle change amount into a joint space speed control instruction through an inverse kinematics algorithm; S607, execute the joint space speed control instruction through a joint driver to drive an execution mechanism of the robot to complete a corresponding task action.

[0096] In the embodiment, the real-time action position coordinates refer to six-dimensional pose data read at a determined frequency during running, including three-dimensional position components and three-dimensional attitude components. The data can be obtained by joint encoders and forward kinematics, or external calibration signals can be provided by optical or electromagnetic tracking systems, and the selection of the source depends on the sensing configuration of the execution mechanism. To be consistent with the training stage, the pose is expressed in the robot base coordinate system, the acquisition end writes a high-precision timestamp according to the sensor time reference, and the attitude is internally expressed by using a quaternion or a rotation matrix to avoid the singularity of Euler angles, and then mapped to an Euler angle sequence according to the needs at the output interface. The acquisition process includes temperature drift compensation, hard limit check and outlier rejection, the original discrete sequence is filtered to suppress high-frequency noise with low delay, and the filter window and phase compensation are set according to the control period and the execution mechanism bandwidth. The contact force vector refers to six-dimensional force and torque data from a six-axis force sensor at the end or a joint torque observer. Before acquisition, a bias baseline is established under zero load and the temperature coefficient is saved, and during running, the zero-point drift is deducted in real time, and the amplitude saturation protection is performed on the force and torque channels respectively, and the band-stop or low-pass combined filter is used to filter out the structural resonance frequency band signal. The double-view visual image refers to an image pair collected by two synchronous cameras at the same timestamp, the hardware trigger and clock synchronization circuit ensure the time consistency across devices, the lens intrinsic parameters and extrinsic parameters are obtained by offline calibration, and the online processing includes distortion removal, binocular epipolar line correction and gray or color normalization, and the resolution and format are completely consistent with the training stage. The three sources of data are aligned by timestamp in the aggregator, if a source frame is missing, interpolation or trigger degradation strategy is used according to the adjacent frame, and consistent form input package is ensured for each control period.

[0097] The optimized control model receives the aligned real-time action position coordinates, contact force vectors, and dual-view visual images at the inference end, applies the same numerical standardization and channel arrangement as in the training phase, and maps the three types of inputs into control action vectors. The control action vector is the desired action representation in the task space, which can be expressed in incremental form or end velocity or force-position hybrid instruction form. The specific form has been fixed in the training phase and remains consistent during deployment. Inference is performed in a fixed batch size and single-step or limited sliding window manner, with a limited calculation delay within the control cycle budget. When the visual frame rate and control cycle are inconsistent, the nearest neighbor holding or linear interpolation is used for time domain alignment of visual features. To improve robustness, the inference module sets an abnormality detector for the contact force vector, which reduces the weight of the force channel and maintains the continuity of the action when there is a short-term strong impact or significant saturation.

[0098] The current spatial pose coordinates are given by the robot state manager within the control cycle. The state manager uses sensor data and kinematic models to ensure the same coordinate system and pose representation as the aforementioned real-time action position coordinates. The displacement difference value is obtained by interpreting the control action vector as a desired pose or pose increment, and then taking the vector difference with the current spatial pose coordinates in the same reference frame to obtain the position component difference. The attitude angle change is obtained by rotation difference. The calculation process can use quaternion left multiplication inverse to obtain the relative rotation, and then convert it to axis-angle or Euler angle increment for subsequent control. If Euler angles are used, unwrap the jump across ±π to maintain continuity. The difference result is subjected to amplitude limiting and rate limiting before entering the downstream. The limiting interval is set according to the safety speed of the actuator and the size of the working space.

[0099] Inverse kinematics algorithm converts displacement difference value and attitude angle change into joint space velocity control instruction. Numerical implementation can use damped least squares Jacobian pseudo-inverse to solve joint velocity, and damping factor is adaptively adjusted according to task space error to balance convergence speed and noise amplification. Redundant articulated actuators can introduce zero space projection to guide the joint solution to areas away from singular positions or maintain consistency with historical poses while meeting end task requirements. The zero space weight is dynamically adjusted according to the joint limit margin. The contact force vector can be used to construct a task space weight matrix, which applies strong suppression to the normal velocity component of the contact and maintains flexibility to the tangential component, thereby balancing stable contact and task advancement in contact scenarios. The obtained joint space velocity control instruction is subjected to rate limiting, jerk constraint, and discrete reconstruction before entering the communication layer. The communication layer is packaged according to the real-time bus protocol, including sequence number and timestamp to support packet loss detection.

[0100] Joint driver receives joint space velocity control command and performs cascade regulation of speed loop and position loop at the drive layer, inner loop current or torque control is completed locally by the driver, and outer loop speed and position regulation is implemented at the controller side or the driver side. To form a double closed loop of position and force, the controller simultaneously reads the contact force vector, adds impedance or compliance adjustment term to the task space speed based on the force error, and then maps it back to the joint space through Jacobian, achieving compliance to contact direction and fast following to non-contact direction. During drive execution, the consistency of command and feedback is monitored in real time, and if there is a deviation, communication packet loss or sensor disconnection, the gradual stop or stop logic is triggered and diagnostic information is recorded. The action completion condition can be determined by the target pose convergence threshold, force stability threshold and time window. When multiple consecutive cycles meet the threshold constraint, it is considered that the corresponding task action is completed.

[0101] For example, real-time acquisition of position and force data of Leader end Acquire the current state of the Follower end .

[0102] Calculate error: = - . = -

[0103] According to the error, design the position and force feedback control law (such as PID controller or impedance control), respectively adjust the joint control signal and force output of the Follower end, and ensure that the error converges to zero.

[0104]

[0105] Adjust the joint control and force output of the Follower end by calculating the position error and force error, and ensure that the error converges to zero.

[0106] The Leader end adjusts the operation feeling according to the force signal feedback by the Follower end, and realizes natural force sensation feedback; The Follower end executes corresponding control according to the action and force command of the Leader end, and completes the action imitation.

[0107] In the demonstration process, the operator executes the task action through the Leader end, and the Follower end mirrors the master action and its contact force feedback in real time. At the same time, the system synchronously collects relevant position information, contact force information and visual images, providing multi-modal data support for imitation learning.

[0108] Multi-modal data acquisition module: this module is based on = 100Hz of the sampling frequency real-time acquisition of the following multi-dimensional state information: Six degrees of freedom spatial pose coordinates: the pose of the robot end effector under the base coordinates is represented as a vector: wherein, represents the position coordinates of the end in three-dimensional space, represents the Euler angle attitude, and all data is collected at time t.

[0109] Contact force data: the contact force data of the robot end: Reflect the actual force state of the robot during operation, including linear force and torque Multi-view images: a set of high-resolution images collected synchronously from multiple cameras: The image collected by the Nth camera at time t is taken as N, and our design takes N as 2, which can be increased according to the task difficulty to increase the robustness and effect of imitation learning. The multi-modal data sequence provides rich input information for the imitation learning model, supports the mapping learning from state to action, and realizes high-precision and real-time robot control.

[0110] Imitation learning model module: the imitation learning model module is designed based on the Transformer architecture, which can effectively fuse multi-modal input information and realize high-precision prediction and control of complex actions. The module mainly includes an action input processing subnetwork, a contact force input processing subnetwork, a visual feature extraction module, and a multi-layer attention mechanism decoder. The specific structure and process are as follows: Action input processing: the action state vector of the robot at the current time is a six-dimensional vector, indicating the current position as the action input

[0111] The vector is formalized as a function mapping by a pre-neural network (action neural network) as follows: represents the action pre-network, which is an MLP layer, which maps the action input to a high-dimensional feature space for subsequent feature fusion, which is Contact force input processing: uses as the contact force vector input at the current time, which also passes through an independent pre-neural network (force pre-neural network), which is a multi-layer fully connected layer in this invention. It is represented as a function mapping as follows: Force Precedence Network, multi-layer fully connected layers, to extract force input features and ensure the same dimension as action.

[0112] Visual feature extraction: the synchronous images collected by the dual-view camera for visual feature extraction are: Deep features are extracted by pre-trained visual models (Swin Transformer, etc.), represented as: . Where, is the visual extraction network, outputting high-dimensional feature vectors for each view. The two-view features are fused to form a single visual Token, is the self-attention weighted fusion, and the obtained .

[0113] Multi-modal feature fusion and action prediction: As a sequence input Transformer multi-layer Decoder, the action feature , mechanical feature , visual feature TransformerDecoder contains multiple layers of self-attention (Self-Attention) and feed-forward network (Feed-Forward Network) to calculate the correlation between modalities and time sequences through multi-head attention mechanism: . Through linear layers and activation functions, the output of the Decoder is mapped to the next time action prediction vector: is the weight, is the bias, is the activation function ReLU, is the next action of the robot predicted by the model.

[0114] The embodiment synchronously collects real-time action position coordinates, contact force vectors and double-view visual images on the deployment side, and fuses and infers through an optimized control model to obtain a control action vector that is consistent with the training expression, thereby providing a stable target for subsequent pose difference and attitude change solving; the difference result is converted into joint space speed control instructions through a damping inverse kinematics transformation, and redundancy allocation and weight adjustment enable the solution to maintain executability and safety margin near singularity and in contact working conditions; the joint driver cooperates in a speed loop as the main loop and a position and force double closed loop to track the target trajectory and suppress oscillation in the contact direction. The above link closes the perception, inference, geometric transformation and servo control in each control period, and is based on time-consistent alignment input and coordinate-consistent error measurement, thereby substantially reducing accumulated errors caused by sensing lag and model mismatch, and thus obtaining smoother action following, more stable contact process and more controllable abnormal handling execution effect.

[0115] In an embodiment, a master-slave collaborative control device is provided, which corresponds to the master-slave collaborative control method in the above embodiment. Referring to Figure 3 , Figure 3 A functional module schematic diagram of a preferred embodiment of the master-slave collaborative control device of the present application. The multi-modal data acquisition module 10, the data preprocessing module 20, the modal encoding module 30, the feature fusion decoding module 40, the model training optimization module 50 and the deployment execution module 60. The detailed description of each functional module is as follows: The multi-modal data acquisition module 10 is used to establish a master-slave collaborative control link, and synchronously acquires spatial pose coordinates, contact force vectors and double-view images through a bilateral force-position closed-loop feedback mechanism to construct a multi-modal demonstration data set; The data preprocessing module 20 is used to perform time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; The modal encoding module 30 is used to obtain action features by processing action state vectors in the standardized input sequence through an action modal encoding network, obtain force features by processing contact force vectors in the standardized input sequence through a mechanics modal encoding network, and obtain visual features by processing double-view images in the standardized input sequence through a visual encoding network; The feature fusion decoding module 40 is used to fuse the action features, the force features and the visual features into a multi-modal tensor sequence and input a Transformer decoder to generate a joint time series representation; The model training optimization module 50 is configured to train the joint time sequence representation into a motion prediction vector by the motion predictor, construct a supervision loss function based on the motion prediction vector and expert demonstration labels, update parameters of the motion modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the motion predictor, and obtain an optimized control model. The deployment execution module 60 is configured to process real-time motion position, contact force and visual image inputs into a control motion vector by the optimized control model in a deployment stage, and convert the control motion vector into joint space control instructions to drive an execution mechanism.

[0116] In one embodiment, a computer device is provided, which can be a server, and an internal structure diagram thereof can be as shown in FIG. 1. Figure 4 The computer device includes a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external user terminal through a network connection. The computer program is executed by the processor to implement functions or steps of a master-slave collaborative control method on the server side.

[0117] In one embodiment, a computer device is provided, which can be a user terminal, and an internal structure diagram thereof can be as shown in FIG. 2. Figure 5 The computer device includes a processor, a memory, a network interface, a display screen and an input device connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external server through a network connection. The computer program is executed by the processor to implement functions or steps of a master-slave collaborative control method on the user terminal side.

[0118] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program: A master-slave collaborative control link is established, spatial pose coordinates, contact force vectors and double-view images are synchronously collected through a bilateral force-position closed-loop feedback mechanism, and a multi-modal demonstration dataset is constructed. performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; processing the action state vector in the standardized input sequence through an action modal encoding network to obtain an action feature, processing the contact force vector in the standardized input sequence through a mechanical modal encoding network to obtain a force feature, and processing the dual-view image in the standardized input sequence through a visual encoding network to obtain a visual feature; fusing the action feature, the force feature, and the visual feature into a multi-modal tensor sequence and inputting the multi-modal tensor sequence into a Transformer decoder to generate a joint time sequence representation; outputting a training action prediction vector from the joint time sequence representation through an action predictor, constructing a supervised loss function based on the training action prediction vector and expert demonstration labeling, updating parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor to obtain an optimized control model; in a deployment phase, processing real-time action position, contact force, and visual image input through the optimized control model to generate a control action vector, and converting the control action vector into a joint space control instruction to drive an execution mechanism.

[0119] In one embodiment, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the following steps are implemented: establishing a master-slave collaborative control link, synchronously collecting spatial pose coordinates, contact force vectors, and dual-view images through a bilateral force-position closed-loop feedback mechanism, and constructing a multi-modal demonstration data set; performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; processing the action state vector in the standardized input sequence through an action modal encoding network to obtain an action feature, processing the contact force vector in the standardized input sequence through a mechanical modal encoding network to obtain a force feature, and processing the dual-view image in the standardized input sequence through a visual encoding network to obtain a visual feature; fusing the action feature, the force feature, and the visual feature into a multi-modal tensor sequence and inputting the multi-modal tensor sequence into a Transformer decoder to generate a joint time sequence representation; outputting a training action prediction vector from the joint time sequence representation through an action predictor, constructing a supervised loss function based on the training action prediction vector and expert demonstration labeling, updating parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor to obtain an optimized control model;

[0120] In the deployment stage, the real-time action position, contact force and visual image input are processed by the optimization control model to generate a control action vector, and the control action vector is converted into joint space control instructions to drive the actuator.

[0121] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the server side and the user side in the foregoing method embodiments, and to avoid repetition, they will not be described one by one here.

Claims

1. A control method based on master-slave cooperation, characterized by, The method comprises the following steps: establishing a master-slave collaborative control link, synchronously collecting spatial pose coordinates, contact force vectors and double-view images through a bilateral force-position closed-loop feedback mechanism, and constructing a multi-modal demonstration data set; performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; processing the action state vector in the standardized input sequence through an action modal encoding network to obtain action features, processing the contact force vector in the standardized input sequence through a mechanics modal encoding network to obtain force features, and processing the double-view image in the standardized input sequence through a visual encoding network to obtain visual features; fusing the action features, the force features and the visual features into a multi-modal tensor sequence and inputting the multi-modal tensor sequence into a Transformer decoder to generate a joint time sequence representation; outputting a training action prediction vector from the joint time sequence representation through an action predictor, constructing a supervised loss function based on the training action prediction vector and expert demonstration labels, updating parameters of the action modal encoding network, the mechanics modal encoding network, the visual encoding network, the Transformer decoder and the action predictor to obtain an optimized control model; in the deployment stage, processing real-time action positions, contact forces and visual images input through the optimized control model to generate a control action vector, and converting the control action vector into joint space control instructions to drive an execution mechanism.

2. The master-slave collaboration based control method of claim 1, wherein, establishing a master-slave collaborative control link, synchronously collecting spatial pose coordinates, contact force vectors and double-view images through a bilateral force-position closed-loop feedback mechanism, and constructing a multi-modal demonstration data set, comprising: establishing a physical connection channel between a master control end and a slave robot end; collecting real-time position states and contact force information of the master control end to obtain first spatial pose coordinates and first contact force vectors; receiving real-time position states and contact force information feedback from the slave robot end to obtain second spatial pose coordinates and second contact force vectors; calculating position errors of the first spatial pose coordinates and the second spatial pose coordinates; calculating contact force errors of the first contact force vectors and the second contact force vectors; generating a position compensation control signal based on the position errors; generating a force compensation control signal based on the contact force errors; driving the slave robot end to realize action mirroring through the position compensation control signal and the force compensation control signal; synchronously collecting spatial pose coordinates, contact force vectors and double-view images during the mirroring action process; constructing a multi-modal demonstration data set based on the spatial pose coordinates, the contact force vectors and the double-view images.

3. The master-slave collaboration based control method of claim 1, wherein, performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence, comprising: extracting timestamp information in the multi-modal demonstration data set; aligning the spatial pose coordinates, the contact force vectors and the double-view images to a unified time axis according to the timestamp information; performing coordinate system standardization conversion on the spatial pose coordinates to obtain standardized spatial pose coordinates; performing force sensor range normalization on the contact force vectors to obtain normalized contact force vectors; performing channel standardization processing on the double-view images to obtain standardized double-view images; The standardized space pose coordinates, the normalized contact force vectors, and the standardized dual-view images are packaged in time sequence as a standardized input sequence.

4. The master-slave collaboration based control method of claim 1, wherein, The action state vectors in the standardized input sequence are processed by an action modal encoding network to obtain action features, the contact force vectors in the standardized input sequence are processed by a mechanics modal encoding network to obtain force features, and the dual-view images in the standardized input sequence are processed by a visual encoding network to obtain visual features, including: The action state vectors in the standardized input sequence are input into a multi-layer perception network, and feature transformation is performed through a fully connected layer and a nonlinear activation function to output action features. The contact force vectors in the standardized input sequence are input into a fully connected neural network, and feature signals are extracted through a linear transformation layer to output force features of the same dimension as the action features. The dual-view images in the standardized input sequence are input into a pre-trained visual model to extract first-view deep features and second-view deep features. Cross-view self-attention weight calculation is performed on the first-view deep features and the second-view deep features to obtain cross-view attention weights. The first-view deep features and the second-view deep features are weighted and summed based on the cross-view attention weights, and the weighted summation result is output as visual features.

5. The master-slave collaboration based control method of claim 1, wherein, The action features, the force features, and the visual features are fused into a multi-modal tensor sequence and input into a Transformer decoder to generate joint time sequence representation, including: The action features, the force features, and the visual features are concatenated along the feature dimension in the Transformer decoder to form a multi-modal tensor sequence. Sinusoidal position encoding information is added to the multi-modal tensor sequence. The multi-modal tensor sequence with added sinusoidal position encoding information is divided into multiple attention subspaces. Query matrices, key matrices, and value matrices are calculated in each attention subspace in the multiple attention subspaces. Attention weight distribution is calculated based on the similarity of the query matrices and the key matrices. The value matrices are weighted and fused according to the attention weight distribution to obtain fused features. The fused features are input into a feedforward neural network to update the feature representation through nonlinear transformation to obtain updated features. Layer normalization is performed on the updated features to obtain normalized features. The multi-modal tensor sequence with added sinusoidal position encoding information and the normalized features are integrated through residual connection in the Transformer decoder to obtain residual integration features. The residual integration features are output as joint time sequence representation.

6. The master-slave collaboration based control method of claim 1, wherein, The joint time sequence representation is output by an action predictor to obtain a training action prediction vector, a supervision loss function is constructed based on the training action prediction vector and expert demonstration labels, and the parameters of the action modal encoding network, the mechanics modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor are updated to obtain an optimized control model, including: The joint time sequence representation is input into a linear mapping layer of the action predictor, processed by an activation function, and output as a training action prediction vector. Calculate the mean square error of the training action prediction vector and the expert demonstration label as an action prediction loss term; Calculate the time continuity constraint term of the adjacent time step training action prediction vector; Add the action prediction loss term and the time continuity constraint term to construct a supervised loss function; Calculate the gradient of the supervised loss function with respect to the parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor using the backpropagation algorithm; Based on the gradient of the parameters, update the parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder, and the action predictor using an optimization algorithm; Iteratively perform parameter updates until a pre-set stopping condition is met to obtain an optimized control model.

7. The master-slave collaboration based control method of claim 1, wherein, In the deployment phase, the real-time action position, contact force, and visual image input are processed by the optimized control model to generate a control action vector, which is converted into a joint space control command to drive the actuator, including: Collecting real-time action position coordinates, contact force vectors, and dual-view visual images of the robot; Fusing and processing the real-time action position coordinates, contact force vectors, and dual-view visual images through the optimized control model to generate a control action vector; Obtaining the current spatial pose coordinates of the robot; Calculating the displacement difference value of the control action vector and the current spatial pose coordinates; Calculating the attitude angle change amount of the control action vector and the current spatial pose coordinates; Converting the displacement difference value and the attitude angle change amount into joint space speed control commands through inverse kinematics algorithm; Executing the joint space speed control commands through the joint drive to drive the robot's actuator to complete the corresponding task action.

8. A master-slave cooperative-based control device characterized by comprising: The control device based on master-slave collaboration includes: A multi-modal data acquisition module for establishing a master-slave collaborative control link and synchronously acquiring spatial pose coordinates, contact force vectors, and dual-view images through a bilateral force-position closed-loop feedback mechanism to construct a multi-modal demonstration data set; A data preprocessing module for performing time synchronization alignment and normalization processing on the multi-modal demonstration data set to form a standardized input sequence; A modal encoding module for processing the action state vector in the standardized input sequence through an action modal encoding network to obtain action features, processing the contact force vector in the standardized input sequence through a mechanical modal encoding network to obtain force features, and processing the dual-view image in the standardized input sequence through a visual encoding network to obtain visual features; A feature fusion decoding module for fusing the action features, force features, and visual features into a multi-modal tensor sequence and inputting them into a Transformer decoder to generate a joint time series representation. The model training optimization module is configured to output a training action prediction vector by the joint time sequence characteristic via the action predictor, construct a supervision loss function based on the training action prediction vector and expert demonstration labeling, update parameters of the action modal encoding network, the mechanical modal encoding network, the visual encoding network, the Transformer decoder and the action predictor, and obtain an optimized control model. The deployment execution module is configured to generate a control action vector by processing real-time action positions, contact forces and visual image inputs via the optimized control model in a deployment stage, and convert the control action vector into joint space control instructions to drive an execution mechanism.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a master-slave collaborative control program stored on the memory and executable on the processor, and the master-slave collaborative control program, when executed by the processor, implements the steps of the master-slave collaborative control method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a master-slave collaborative control program, and the master-slave collaborative control program, when executed by the processor, implements the steps of the master-slave collaborative control method according to any one of claims 1-7.

Images