Upper limb skill transfer method based on multi-modal fusion Transformer

Abstract

The invention discloses an upper limb skill transfer method based on multi-modal fusion Transform, and relates to the technical field of robotics.The method comprises the steps that firstly, an IMU sensor and upper limb motion characteristics are combined, and an upper limb position calculation model is established; secondly, an upper limb rigidity transfer frame is established by combining the electromyographic sensor and the rigidity characteristic of the flexible arm; then, a sample collection scene is established, and a sample set is constructed; then, a joint angle sequence prediction model based on the multi-modal fusion Transform is constructed; and finally, completing model training based on the sample set, and verifying the test set by using the trained prediction model. According to the method, the transfer from the upper limb rigidity to the flexible arm rigidity is realized by combining the characteristics of the flexible arm; through multi-scale feature fusion, two modal data features are efficiently extracted and spliced, and in combination with a Transform multi-head attention mechanism, the sequence prediction precision of the model can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, and in particular to a method for transferring upper limb skills based on multi-modal fusion Transformer. BACKGROUND

[0002] The human upper limb is a biological system similar to a mechanical arm, which has rich motion expression information, such as kinematic information and biological information, which can be measured by various sensors. However, how to combine the motion information of the human upper limb with the control logic of the robot to endow the flexible robot with human skills is still a problem worth studying.

[0003] The existing upper limb skill transfer method has the following deficiencies: (1) Most of the previous studies use EMG signals alone to realize the transfer of upper limb stiffness to the stiffness parameter of the rigid arm impedance controller. Since the flexible arm itself is relatively special, there are few human skill transfer methods for such mechanical arms, and the target task is relatively single.

[0004] (2) Single modal method only relies on a single signal source, which is difficult to fully reflect complex upper limb movements, which may limit the prediction accuracy of the model. Compared with the multi-modal method which utilizes the complementary features of multi-modal data for training and learning, the single modal method has limitations. SUMMARY

[0005] In order to overcome the defects in the prior art, the present application provides a method for transferring upper limb skills based on multi-modal fusion Transformer, which transfers the upper limb stiffness adjustment strategy to the flexible arm and endows the flexible arm with human-like skills.

[0006] To achieve the above purpose, the present application adopts the following technical solutions, comprising: A method for transferring upper limb skills based on multi-modal fusion Transformer, specifically comprising the following steps: S1, combining IMU sensors and upper limb motion characteristics to establish an upper limb position solution model; S2, combining electromyography sensors and flexible arm stiffness characteristics to establish an upper limb stiffness transfer framework; S3, building a sample collection scene and constructing a sample set; S4, constructing a joint angle sequence prediction model based on multi-modal fusion Transformer; S5, based on the sample set, completing model training, and using the trained model to verify the test set.

[0007] Preferably, step S1 comprises the following steps: S11, the human upper limb is simplified as a simple multi-link robot arm, and coordinate systems are established at the shoulder, elbow and wrist respectively 、 、 , are the shoulder point, elbow point and wrist point respectively; three IMU sensors are placed at the shoulder point, elbow point and wrist point respectively, and the local coordinate systems thereof are consistent with the directions 、 、 ; the shoulder coordinate system is selected as the reference coordinate system, and the elbow coordinate system and the wrist coordinate system are moving; the wrist point is selected as the upper limb position calculation point; S12, the upper limb position is solved through the forward kinematics coordinate system transformation relationship, and the position of the wrist point in the reference coordinate system is obtained: ; wherein the homogeneous transformation matrix is composed of a rotation transformation matrix and a translation transformation matrix ; is determined by the three-axis attitude angles of the reference coordinate system and the elbow coordinate system ; the homogeneous transformation matrix is composed of a rotation transformation matrix and a translation transformation matrix ; is determined by the three-axis attitude angles of the elbow coordinate system and the wrist coordinate system .

[0008] Preferably, step S2 comprises the following steps: S21, wearing an electromyography sensor with eight surface electromyography signal sampling channels at the elbow, and using a moving window average method to extract the original electromyography signal envelope; S22, obtaining a muscle activation coefficient by calculating the absolute value of the amplitude of the eight-channel original electromyography signal envelope; S23, limiting the muscle activation coefficient to a specified range through an exponential function-based operation expression, and obtaining an upper limb stiffness mapping coefficient : ; wherein and are parameters to be estimated; S24, calculating the stiffness by using a mathematical model derived on the basis of a statics model; after the external environment and the structure of the flexible arm are determined, the stiffness is determined by the joint angle and the rope; when the stiffness in a certain direction is mapped, the stiffness matrix of the flexible arm corresponding component and upper limb stiffness mapping coefficient The relationship is as follows, the increase of upper limb muscle activation intensity determines the change of stiffness component index in the direction of increase: ; wherein, and respectively represent the index values in the direction of stiffness matrix of flexible arm before and after upper limb stiffness mapping x ; Step S25, since the results of inverse kinematics of redundant flexible arm have many groups, and the above stiffness component index is a function of flexible arm joint angle , the method of null space stiffness optimization is used to solve, and the flexible arm joint angle after the change of initial stiffness to expected stiffness is obtained.

[0009] Preferably, in step S21, the moving window average method is used to extract the original electromyogram envelope: ; wherein, denotes the size of the window function, which is set to 50; denotes the instantaneous amplitude of the original electromyogram, and respectively represent time and sampling points; denotes the amplitude of the original electromyogram envelope; In step S22, the calculation of muscle activation coefficient is as follows: .

[0010] Preferably, step S3 is specifically as follows: Wear three IMU sensors at the shoulder point, elbow point and wrist point respectively, wear the electromyogram sensor at the elbow, transmit the upper limb position to the end of the flexible arm, and solve the joint angle of the control mechanical arm through the geometric iteration method; the single sample collection process is to control the upper limb to be stationary in the set shape, the palm slowly changes from the open state to the tight state within the set time, the muscle activation intensity of the arm is controlled to be larger, and the IMU data, EMG data and joint angle data are saved; the average value of the joint angle data collected in each sample is taken as the initial arm shape angle, and the flexible arm joint angle after stiffness adjustment is obtained through the null space stiffness optimization method.

[0011] Preferably, in step S4, the target task of the model is to train the flexible arm joint angle sequence corresponding to different muscle activation degrees under different initial upper limb configurations, so as to be able to transfer the upper limb skills to the flexible arm; The model can accurately predict the flexible arm joint angle sequence corresponding to different muscle activation degrees under different initial upper limb configurations according to the current given kinematic data The biological signal data corresponding to the muscle activation coefficient of the previous step Predict the joint angle sequence of the next step : ; wherein, represents the predicted flexible arm joint angle sequence; represents the model; represents the model parameters.

[0012] Preferably, in step S4, multi-scale feature fusion is used for IMU data and EMG data, feature alignment of two modal data is realized, the feature representation vectors of the processed IMU data and EMG data are aligned and spliced, and are added with position Embedding, arranged in order from small to large according to the stiffness mapping coefficient, to obtain an input representation matrix of the sample, which is simultaneously used as the input of the Transformer Encoder and the Transformer Decoder, to complete the supervised learning sequence-to-sequence prediction task.

[0013] The application provides a readable storage medium, which has a computer program stored thereon, and the computer program is executed to realize the upper limb skill transfer method based on multi-modal fusion Transformer.

[0014] The application provides an electronic device, which includes a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the upper limb skill transfer method based on multi-modal fusion Transformer.

[0015] The application provides a computer program product, which includes computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the upper limb skill transfer method based on multi-modal fusion Transformer.

[0016] The application has the following advantages: (1) The kinematic information and biological information of the human upper limb are fully considered, and the stiffness characteristics of the flexible robot are combined, so that the human upper limb can control the position and stiffness of the flexible robot, and the upper limb multi-skill transfer is realized.

[0017] (2) Multi-scale feature fusion is used for low-dimensional IMU sequence signals and EMG space-time matrix signals, efficient fusion and feature alignment of two modal data are realized, and the advantages of the Transformer multi-head attention mechanism in capturing long-distance dependency relationships are combined, so that the accuracy of the model sequence-to-sequence prediction is improved.

[0018] (3) The method combines the characteristics of the flexible arm itself to realize the transfer of the upper limb stiffness to the stiffness of the flexible arm; through multi-scale feature fusion, the two modal data features are efficiently extracted and spliced, and combined with the multi-head attention mechanism of the Transformer, the sequence prediction accuracy of the model can be effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a flowchart of an upper limb skill transfer method based on multi-modal fusion Transformer according to an embodiment of the present application.

[0020] Figure 2 is an upper limb kinematic model according to an embodiment of the present application.

[0021] Figure 3 is a schematic diagram of an upper limb skill transfer method based on multi-modal fusion Transformer according to an embodiment of the present application.

[0022] Figure 4 is a model prediction effect diagram according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0024] Embodiment 1 As shown in Figure 1 and Figure 3 , an upper limb skill transfer method based on multi-modal fusion Transformer of the present application is used to transfer the upper limb stiffness adjustment strategy to the flexible arm and endow the flexible arm with human-like skills, and the specific steps are as follows: S1, combine the IMU sensor and the upper limb motion characteristics to establish an upper limb position calculation model.

[0025] S2, combine the electromyography sensor and the stiffness characteristics of the flexible arm to establish an upper limb stiffness transfer framework.

[0026] S3, build a sample collection scene and construct a sample set.

[0027] S4, construct a joint angle sequence prediction model based on multi-modal fusion Transformer (a deep learning model).

[0028] S5, complete model training based on the sample set, and verify the test set using the trained joint angle sequence prediction model.

[0029] In step S1, the specific steps of establishing the upper limb position solution model include: S11, as shown in the figure, the human upper limb is simplified as a simple multi-link mechanical arm, and coordinate systems are established at the shoulder, elbow and wrist respectively Figure 2 are the shoulder point, elbow point and wrist point respectively; and are the lengths of the connecting rods connecting two joint points and , i.e. the lengths of the upper arm and the lower arm.

[0030] Three IMU sensors (inertial measurement unit, measuring motion state and spatial attitude) are placed at the shoulder point, elbow point and wrist point respectively, and their local coordinate systems are consistent with the direction. The shoulder coordinate system is selected as the reference coordinate system, and the elbow coordinate system and the wrist coordinate system are moving. The wrist point is selected as the upper limb position solution point.

[0031] S12, the upper limb position is solved by the forward kinematics coordinate system transformation relationship. The transformation from one coordinate system to another coordinate system can be decomposed into two steps of rotation and translation. The position of the wrist point in the reference coordinate system is obtained: ; wherein the homogeneous transformation matrix is composed of the rotation transformation matrix and the translation transformation matrix ; which is determined by the three-axis attitude angles of the reference coordinate system and the elbow coordinate system. The homogeneous transformation matrix is composed of the rotation transformation matrix and the translation transformation matrix ; which is determined by the three-axis attitude angles of the elbow coordinate system and the wrist coordinate system; The three-axis measurement angles of the shoulder and elbow IMU are calculated as follows: ; In the above formula, and are the three-axis angle values measured by the IMU of the shoulder and the elbow respectively, and ​​​​​​The static error values of the IMU measurements of the shoulder and elbow, respectively (eliminate static error, make the straight arm state correspond to the original arm shape of the flexible arm).

[0032] The three-axis measurement angles of the elbow and wrist IMU are calculated as follows: ; In the above formula, and are the three-axis angle values measured by the IMU of the elbow and wrist, respectively, and are the static error values of the IMU measurements of the elbow and wrist, respectively (eliminate static error, make the straight arm state correspond to the original arm shape of the flexible arm).

[0033] In step S2, the specific steps of establishing the upper limb stiffness transfer framework include: S21, wear the electromyographic sensor at the elbow, in this invention, the electromyographic sensor uses MYO Band equipment, which has eight surface electromyographic signal sampling channels, a sampling frequency of 50Hz and a built-in filtering algorithm. The original electromyographic signal envelope is extracted using the moving window average method: ; where, represents the size of the window function, set to 50; represents the instantaneous amplitude of the original electromyographic signal, and represent time and sampling points, respectively; represents the amplitude of the sEMG signal (original electromyographic signal) envelope.

[0034] S22, obtain the muscle activation coefficient by calculating the absolute value of the envelope amplitude of the eight-channel sEMG signal , as shown in the following formula: ; S23, limit the muscle activation coefficient to a specified range through an exponential function-based operation expression, to obtain the upper limb stiffness mapping coefficient : ; where, and are normal parameters to be estimated, which are set to 1 and 0.01, respectively, through testing, so as to limit the upper limb stiffness mapping coefficient to the range of (1, 2).

[0035] S24, The stiffness of a flexible arm typically refers to its ability to resist deformation when the end effector is subjected to external forces. Because the flexible arm moves slowly and is approximately in equilibrium during each small displacement, its stiffness can be calculated using a mathematical model derived from a static model. ; in, and These are the position Jacobian matrix and the rope length Jacobian matrix, respectively. , and These are the Hessian matrix representing the position, rope length, and height. Let be the diagonal matrix of the rope's stiffness; This is the stiffness matrix of the flexible arm.

[0036] Once the external environment and the structure of the flexible arm itself are determined, its stiffness is mainly determined by the joint angles and the cables. For example, when mapping the stiffness in a certain direction, the stiffness matrix of the flexible arm... The corresponding component and upper limb stiffness mapping coefficient The relationship is as follows: an increase in upper limb muscle activation intensity determines that this component index changes in the direction of increase: ; in, and Represent the stiffness matrices of the flexible arm before and after upper limb stiffness mapping, respectively. x The index value in the direction.

[0037] S25, since there are many sets of results for the inverse kinematics solution of the redundant flexible arm, and the above stiffness component indices are related to the joint angles of the flexible arm. Therefore, the joint angles after the change from the initial stiffness to the expected stiffness can be obtained by using the zero-space stiffness optimization method. .

[0038] In step S3, the sample collection scenario is set up and the sample set is constructed, as shown below: The sample acquisition stage is a scene built using the Unity virtual engine. Three IMU sensors and an EMG sensor are fitted to a fixed position on the arm, and a script for data acquisition and storage is written. The upper limb position is transmitted to the flexible arm's end effector, and the ten joint angles of the robotic arm are calculated using a geometric iteration method. During a single sample acquisition process, the upper limb is kept stationary in a specific shape, and the palm gradually changes from an open to a clenched state over 10 seconds, increasing the activation intensity of the arm muscles. The IMU, EMG, joint angle, and calculated upper limb stiffness mapping coefficients are then saved. Data. The average value of the joint angle data collected for each sample is taken as the initial arm shape angle, and the stiffness-adjusted flexible arm ten joint angles are obtained by zero-space stiffness optimization method In a single sample, IMU data , EMG data .

[0039] In step S4, a joint angle sequence prediction model based on a multi-modal fusion Transformer (a deep learning model) is constructed, and the specific process is as follows: Traditional sequence learning usually uses recurrent neural networks (RNNs), but it has the problems of gradient disappearance and explosion, and cannot handle long-distance dependencies. The long short-term memory (LSTM) network introduces a memory unit and a gating mechanism to solve the former problem, but has the problems of long training time and poor scalability. The Transformer can parallelly process all positions in the sequence by virtue of the self-attention mechanism, greatly accelerating the training process, and can be more accurate when capturing long-distance dependencies, without the problem of reduced information transmission efficiency.

[0040] The target task of the application is to train a flexible arm joint angle sequence that can accurately predict different muscle activation levels under different initial upper limb configurations, so as to be able to transfer upper limb skills to the flexible arm. For sensor data, we construct a dataset containing kinematic data and biological signal data . Among them, the kinematic data with a shape of 1x6 indicates the three-axis attitude angle calculated by the IMU at the shoulder, elbow and wrist. The biological signal data with a shape of 8x50 indicates the eight-channel electromyographic data measured by the electromyographic sensor worn on the elbow. The model can predict the next step joint angle sequence according to the current given kinematic data and the biological signal data corresponding to the muscle activation coefficient of the previous step : ; Among them, indicates the predicted flexible arm joint angle sequence; indicates the whole model; indicates all related parameters of the model.

[0041] In order to enhance the feature extraction effect of the input data and improve the accuracy and generalization ability of the model, the IMU data ​The processing is as follows: after first layer one-dimensional convolution, the input channel and the output channel are 1x16, and the basic features are extracted from the original data; in turn, the one-dimensional convolution layers with the input channel and the output channel being 9x27, 27x32 and 32x54 are sequentially passed, the feature extraction is further deepened, and higher-level features are obtained; finally, the one-dimensional convolution layers with the input channel and the output channel being 54x27 and 27x9 are sequentially passed, the number of feature map channels is reduced, and important features are combined. The convolution kernel size of each layer is 2, and then batch normalization layer and ReLU activation function are passed.

[0042] The EMG data The processing is as follows: after passing through a two-dimensional convolution layer with an input channel and an output channel being 1x16 and a convolution kernel size being 3x3, batch normalization and adaptive average pooling layers are passed.

[0043] The feature representation vectors of the processed IMU data and EMG data are aligned and spliced, and are added to the position Embedding, are arranged in order from small to large according to the stiffness mapping coefficient, and an input representation matrix of the sample is obtained. The input representation matrix is simultaneously input into the Transformer Encoder and the Transformer Decoder, and the task of supervised learning sequence to sequence prediction is completed.

[0044] S5, based on the sample set, the training of the model is completed, and the trained joint angle sequence prediction model is used to verify the test set.

[0045] In the experiment, the learning rate of the Adam optimizer is set to 0.0001, the weight decay is 0.001, and the batch size is 15. A total of 300 samples are collected, and the training set and the verification set are divided according to the ratio of 4:1. The tensor shape of the IMU data is , the tensor shape of the EMG data is , and the tensor shape of the Angles data is , and the epochs is set to 1000.

[0046] The number of samples of the test set is 50, different samples correspond to different initial arm shapes, but the data collected by a single sample is arranged in order according to the muscle activation degree. For the flexible arm, each sample corresponds to the case that the stiffness of the flexible arm is adjusted in the direction of increasing under a certain initial configuration. The present application tests the prediction effect of the model on the test set under different initial arm shapes and a certain muscle activation degree, as shown in Figure 4 It can be seen that the model can accurately predict the samples of different initial arm shapes.

[0047] Embodiment 2 An electronic device includes a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor implements the method of the upper limb skill transfer based on the multi-modal fusion Transformer according to the embodiment 1 when executing the computer program.

[0048] The electronic device of the embodiments of the present application can be the mobile device itself or a single device independent of the mobile device, which can communicate with the mobile device to receive the collected input signals therefrom and send the selected target decision behaviors thereto.

[0049] The electronic device includes one or more processors and a memory. The processor can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions. The memory can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor can execute the program instructions to implement the decision behavior decision method of the embodiments of the present application described above and / or other desired functions.

[0050] The electronic device can further include an input device and an output device.

[0051] Embodiment 3 In addition to the above method and device, the embodiments of the present application can also be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the decision behavior decision method according to the embodiments of the present application described in the above embodiment 1.

[0052] The computer program product can be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, and conventional procedural programming languages, such as "C" programming language or similar programming languages, to perform the operations of the embodiments of the present application. The program code can be executed entirely on the user computing device, partially on the user device, as a standalone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0053] Example 4 The embodiments of the present application can also be computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform the steps described in the above embodiments 1 of the decision-making method according to various embodiments of the present application.

[0054] The computer readable storage medium can be any combination of one or more computer readable medium(s). The computer readable medium can be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0055] The above merely describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-modal fusion Transformer-based upper limb skill transfer method, characterized in that, Specifically comprising the following steps: S1, combine IMU sensors and upper limb movement characteristics to establish an upper limb position solution model; S2, combine electromyography sensors and flexible arm stiffness characteristics to establish an upper limb stiffness transfer framework; S3, build a sample collection scene and construct a sample set; S4, construct a joint angle sequence prediction model based on multi-modal fusion Transformer; S5, complete model training based on the sample set and verify the test set using the trained model.

2. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, Step S1 includes the following steps: S11, the human upper limb is simplified as a simple multi-link robot arm, and coordinate systems are respectively established at the shoulder, elbow and wrist , , , are respectively a shoulder point, an elbow point and a wrist point; three IMU sensors are respectively placed at the shoulder point, the elbow point and the wrist point, and the local coordinate systems thereof are consistent with the directions , , The shoulder coordinate system is selected as a reference coordinate system, the elbow coordinate system and the wrist coordinate system are moving, and the wrist point is selected as an upper limb position calculation point; S12, solve the upper limb position by forward kinematics coordinate system transformation relationship, get the wrist point In the position of the reference coordinate system : wherein the homogeneous transformation matrix is composed of a rotation transformation matrix and a translation transformation matrix . is determined by three-axis attitude angles of the reference coordinate system and the elbow coordinate system . The homogeneous transformation matrix is composed of a rotation transformation matrix and a translation transformation matrix . is determined by three-axis attitude angles of the elbow coordinate system and the wrist coordinate system .

3. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, Step S2 includes the following steps: S21, wear an electromyography sensor with eight surface electromyography signal sampling channels at the elbow, and use a moving window average method to extract the original electromyography signal envelope; S22, obtain muscle activation coefficient by calculating absolute value of envelope amplitude of eight-channel original electromyography signal ; S23, expressing the muscle activation coefficient by an operation expression based on an exponential function limiting to a specified range, obtaining the upper limb stiffness mapping coefficient : wherein and are parameters to be estimated; S24, the stiffness calculated by the mathematical model derived on the basis of the static model, the external environment and the structure of the flexible arm itself are determined, the stiffness is determined by the joint angle and the rope; when mapping the stiffness in a certain direction, the stiffness matrix of the flexible arm The corresponding component and the upper limb stiffness mapping coefficient The relationship is as follows, the increase of the upper limb muscle activation intensity determines the change of the stiffness component index in the direction of increase: wherein, and respectively represent index values in the direction of the stiffness matrix of the flexible arm before and after the upper limb stiffness mapping x} Step S25, since the results of inverse kinematics solving of the redundant flexible arm have many groups, and the above stiffness component index is a function of the joint angle of the flexible arm , the method of zero space stiffness optimization is solved, and the joint angle of the flexible arm after the change of the initial stiffness to the expected stiffness is obtained .

4. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, In step S21, the moving window average method is used to extract the original electromyography signal envelope: wherein, represents the size of the window function, set to 50; represents the instantaneous amplitude of the raw myoelectric signal, and represent time and sample points, respectively; represents the amplitude of the raw myoelectric signal envelope; In step S22, the muscle activation coefficient is calculated as follows: 。 5. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, Step S3 is specifically as follows: Wear three IMU sensors at the shoulder point, elbow point and wrist point respectively, wear an electromyography sensor at the elbow, transmit the upper limb position to the flexible arm end, and solve the joint angle of the control mechanical arm through a geometric iteration method; The single sample collection process is to control the upper limb to be stationary in a set shape, the palm to slowly change from an open state to a tight state within a set time, and the hand muscle activation intensity to be large, and save the IMU data, EMG data and joint angle data; The average value of the joint angle data collected for each sample is used as the initial arm shape angle, and the stiffness adjusted flexible arm joint angle is obtained through the null space stiffness optimization method.

6. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, In step S4, the target task of the model is to train a flexible arm joint angle sequence that can accurately predict different muscle activation levels under different initial upper limb configurations, so as to be able to transfer upper limb skills to the flexible arm; The model is capable of predicting a next sequence of joint angles from a current given kinematic data and biological signal data corresponding to the muscle activation coefficients of the previous step :​ wherein, represents a predicted sequence of joint angles of the flexible arm; represents a model; represents a model parameter.

7. The upper limb skill transfer method based on multi-modal fusion Transformer according to claim 1, characterized in that, In step S4, multi-scale feature fusion is used for IMU data and EMG data to realize feature alignment of the two modal data, the feature representation vectors of the processed IMU data and EMG data are aligned and spliced, and are added with position Embedding, arranged in order from small to large according to the stiffness mapping coefficient, and an input representation matrix of the sample is obtained., which is used as the input of the Transformer Encoder and the Transformer Decoder, to complete the task of supervised learning sequence to sequence prediction.

8. A readable storage medium, characterized by, It has a computer program stored thereon, which is executed to realize the upper limb skill transfer method based on multi-modal fusion Transformer of any one of claims 1-7.

9. An electronic device, comprising: It includes a processor, a memory, and a computer program stored on the memory and executable on the processor, and the processor executes the computer program to realize the upper limb skill transfer method based on multi-modal fusion Transformer of any one of claims 1-7.

10. A computer program product, characterised in that, It includes a computer program / instruction, which is executed by a processor to realize the upper limb skill transfer method based on multi-modal fusion Transformer of any one of claims 1-7.

Images