Multi-modal data fusion method, flexible exoskeleton robot and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a multi-modal data fusion method and device, a flexible exoskeleton robot and a storage medium, and the method comprises the steps: obtaining I MU data and an RGB-D image collected by the flexible exoskeleton robot; prediction is carried out based on the I MU data, the RGB-D image and a trained pre-fusion time sequence vision multi-modal model, a first prediction result of the gait phase and a first classification result of the human body motion intention are obtained, and the pre-fusion time sequence vision multi-modal model is obtained by carrying out model training based on a Transform encoder. According to the method, the I MU data and the RGB-D image can be effectively fused, the control precision and the response speed of the flexible exoskeleton robot in a complex outdoor environment are improved, the multi-modal model is remarkably superior to a traditional single-modal model in the aspects of gait phase prediction and motion intention recognition, and the method has high practical application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multimodal data fusion method, a flexible exoskeleton robot and a storage medium. Background Art

[0002] With the increasing global aging population and the increase in the number of disabled people, the importance of assistive exoskeleton systems in improving human movement ability has become increasingly prominent. The application of exoskeleton technology has gradually expanded from laboratories to complex outdoor environments, which has put forward higher requirements for real-time control of exoskeletons, especially in gait phase prediction and movement intention recognition. The exoskeleton system needs to be able to cope with the changing outdoor terrain and recognize the user's movement intention in real time to provide effective assistance support.

[0003] Currently, exoskeleton control methods based on single-modal data (such as using only IMU or RGB depth cameras) perform well in specific scenarios, but their accuracy and robustness are limited in complex and diverse outdoor environments. Traditional gait analysis usually relies on data collected under controlled laboratory conditions, which is difficult to adapt to dynamic terrain changes in actual applications. Although some studies have begun to explore multi-sensor fusion techniques, such as combining muscle electrodes (sEMG), stretch sensors, and IMU sensors to improve the control performance of exoskeletons, most of these studies focus on indoor environments and do not fully utilize the complementarity between multi-modal data. Summary of the invention

[0004] Based on this, it is necessary to propose a multimodal data fusion method, device, flexible exoskeleton robot and storage medium to address the technical problems of poor gait phase prediction and motion intention recognition in the existing technology.

[0005] In a first aspect, a multimodal data fusion method is provided, the method comprising:

[0006] Obtain IMU data and RGB-D images collected by the flexible exoskeleton robot;

[0007] Prediction is performed based on IMU data, RGB-D images and a trained front-fusion temporal visual multimodal model to obtain a first prediction result of gait phase and a first classification result of human motion intention, wherein the front-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

[0008] In a second aspect, a flexible exoskeleton robot is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned multimodal data fusion method when executing the computer program.

[0009] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the multimodal data fusion method are implemented.

[0010] The multimodal data fusion method proposed in the present invention obtains IMU data and RGB-D images collected by the flexible exoskeleton robot, and finally predicts based on the IMU data, RGB-D images and the trained pre-fusion temporal visual multimodal model to obtain the first prediction result of the gait phase and the first classification result of the human body's motion intention, wherein the pre-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder. The present invention can effectively fuse IMU data and RGB-D images, improve the control accuracy and response speed of the flexible exoskeleton robot in complex outdoor environments, and the multimodal model is significantly superior to the traditional single-modal model in gait phase prediction and motion intention recognition, and has strong practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0012] in:

[0013] Figure 1 is an application environment diagram of a multimodal data fusion method in one embodiment;

[0014] Figure 2 The figure is a flowchart of a multimodal data fusion method in one embodiment. DETAILED DESCRIPTION

[0015] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0016] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] The multimodal data fusion method provided by the embodiment of the present invention can be applied in Figure 1 In the flexible exoskeleton robot, the flexible exoskeleton robot consists of three main parts: sensor system, control system and execution system. Through the multimodal data fusion model, the system realizes the real-time recognition of human gait phase and motion intention, provides key input information for the lower-level control, and thus optimizes the exoskeleton's assistance strategy. Among them, the sensor system includes MU sensors, whose functions are to collect six-axis kinematic data (acceleration, angular velocity) during user movement and provide key human motion time series information for gait phase prediction. Implementation: Six IMU sensors are installed in the flexible exoskeleton robot, each IMU collects acceleration in three directions and angular velocity in three directions, and collects data at a frequency of 100Hz. All data are synchronized through a FIFO (first in, first out) queue and transmitted to the control system for processing. The sensor system also includes an RGB-D camera, whose functions are to capture RGB images and depth information of the user's environment, and provide visual data support for terrain recognition and motion intention recognition. Implementation: The RGB-D camera captures environmental images at a frequency of 30Hz and synchronizes them to the edge computing device (NVIDIA Xavier NX), which is then fused with the IMU data in the control system. The sensor system includes load sensors. Function: Real-time monitoring of the load of the exoskeleton to ensure the rationality of the assistance strategy. Implementation: The load sensors are installed at key parts of the exoskeleton, and the sensor data is transmitted to the execution system through the control board to achieve dynamic adjustment.

[0019] The control system includes an edge computing device (NVIDIA Xavier NX), which has the following functions: running the pre-fusion temporal visual multimodal model and the post-fusion temporal visual multimodal model, processing IMU and RGB-D images in real time, and generating gait phase and motion intention prediction results. Implementation: The edge computing device receives real-time data from the sensor system, standardizes the data through the preprocessing module, and then inputs it into the pre-trained multimodal fusion model. The model performs forward propagation calculation based on the data input to generate prediction results.

[0020] The control system includes a control board (STM32). Function: Select and adjust the exoskeleton's power-assistance strategy based on the prediction results output by the model. Implementation: The control board receives the gait phase and motion intention prediction results from the edge computing device, transmits them to the execution system through the CAN bus, selects a suitable power-assistance curve based on the user's motion state, and dynamically adjusts the power-assistance level.

[0021] The actuator system includes a brushless DC motor (M2006). Function: Provide power support for the user's hip joint, reducing the user's muscle activation needs. Implementation: Two brushless DC motors transmit torque to the hip joint through Bowden cables to produce flexion. The motor dynamically adjusts the torque output according to the instructions of the control board to ensure smooth and effective assistance.

[0022] See also Figure 2 As shown, Figure 2 A flowchart of a multimodal data fusion method provided by an embodiment of the present invention is provided. The multimodal data fusion method is applied to a flexible exoskeleton robot. The multimodal data fusion method comprises the following steps:

[0023] Step S101: Acquire IMU data and RGB-D images collected by the flexible exoskeleton robot;

[0024] Step S102: Prediction is performed based on the IMU data, the RGB-D image, and the trained front-fusion temporal visual multimodal model to obtain a first prediction result of the gait phase and a first classification result of the human motion intention, wherein the front-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

[0025] In one embodiment, the step of performing prediction based on the IMU data, the RGB-D image, and the trained front-fusion temporal visual multimodal model to obtain a first prediction result of the gait phase and a first classification result of the human motion intention includes:

[0026] Step S1021: inputting the IMU data and the RGB-D image into the front fusion temporal visual multimodal model, wherein the front fusion temporal visual multimodal model is used to add position coding based on the RGB-D image to obtain visual data.

[0027] Specifically, the RGB-D image is divided into several small patches, which are expanded and projected into the embedding space. Each image generates projected image data through a linear projection matrix and a bias term. A learnable token is concatenated with the projected image data and a positional encoding is added to form the final image input tensor, which is the visual data.

[0028] Step S1022: adding position coding based on the IMU data to obtain time series data,

[0029] Specifically, the IMU time series data is also divided into small blocks and projected into the embedding space. The projected time series data is combined with a learnable token and position encoding to generate the final time series input tensor as the time series data.

[0030] Step S1023: adding modality-specific codes to the visual data and the time series data respectively, inputting the coded visual data and the time series data into the Transformer encoder to obtain coded data output by the Transformer encoder,

[0031] Specifically, modality-specific encodings are added to the visual data and time series data to help the model distinguish different types of inputs, where the modality-specific encodings refer to learnable vectors. Finally, the encoded visual data and time series data are input into the Transformer encoder to obtain the encoded data output by the Transformer encoder.

[0032] Step S1024: extracting global features of visual and temporal modalities from the encoded data, and integrating the global features into a unified feature vector.

[0033] Step S1025: Based on the feature vector and the cross attention mechanism, the target vector is obtained.

[0034] In this embodiment, a cross-attention mechanism is applied to the fused feature vector to further enhance the interaction between the modalities. This mechanism uses the learnable feature vector and the extracted feature vector in the Transformer encoder to perform attention calculations in the Transformer encoder, so that the model can focus on the key features in each modality according to the current task.

[0035] As an example, the extracted feature vectors include tokens from the image and tokens of M time channels (the M time channels may come from different sensors, such as human muscle electrodes, stretch sensors, imus, etc.). A learnable vector (an arbitrarily initialized vector) is used as the query input of the cross-attention. These tokens (extracted feature vectors) are used as the input of the key and value, and the cross-attention mechanism is calculated. Finally, the encoding vector corresponding to q is extracted.

[0036] Step S1026: input the target vector into a multilayer perceptron to obtain a first prediction result of the gait phase and a first classification result of the human body movement intention.

[0037] In this embodiment, the target vector is processed by a multi-layer perceptron (MLP) to generate a prediction result of the gait phase as a first prediction result and a classification result of the human body's motion intention as a first classification result, which are used for downstream task control of the exoskeleton.

[0038] In one embodiment, the multimodal data fusion method further includes:

[0039] Step S301: Prediction is performed based on the IMU data, the RGB-D image, and the trained post-fusion temporal visual multimodal model to obtain a second prediction result of the gait phase and a second classification result of the human motion intention, wherein the post-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

[0040] In one embodiment, the step S301 further includes:

[0041] Step S401: inputting the IMU data and RGB-D image into a trained post-fusion temporal visual multimodal model for prediction, wherein the post-fusion temporal visual multimodal model is used to extract visual features based on the RGB-D image and the visual Transformer encoder.

[0042] In this embodiment, the visual encoder independently processes the RGB-D image to extract visual features.

[0043] Step S402: Each IMU data channel is divided into small blocks and projected to obtain each first token, and each token is feature-connected with its corresponding single-channel global tag to obtain a second token, wherein the single-channel global tag is obtained by multi-layer attention calculation based on the token.

[0044] In this embodiment, each IMU data channel is divided into small blocks and projected separately. The model introduces single-channel global labeling and summarizes the information in the channel through the self-attention mechanism.

[0045] For example, there is time series data with 100 points, which is patched into a 10*10 time series patch unit, and then a 1*10 token is concatenated at the head to become an 11*10 input.

[0046] Step S403: Perform cross-attention calculation on the single-channel global label of each second token and the preset token to obtain the global label of all channels.

[0047] In this embodiment, assuming there are M channels, there are M channel global labels. Now initialize a q, which is an initialized token, to perform cross-attention calculation with the global labels of these M channels to obtain an integrated global label of all channels (covering the information of all temporal channels).

[0048] Step S404: Perform cross-attention calculation on the global label of all channels and the visual features to obtain the first vector,

[0049] In this embodiment, visual and temporal features are fused through a cross-attention mechanism, allowing the model to exploit the complementary information of the two modalities, i.e., the full-channel global tag (from the temporal signal) and the token from the visual features, also performing cross-attention calculations.

[0050] Step S405: Based on the first vector and the cross attention mechanism, a second vector is obtained.

[0051] In this embodiment, the first vector is processed by another cross-attention mechanism to further enhance the interaction between the modalities. The weights in the cross-attention can know the importance of each channel data.

[0052] Step S406: input the second vector into a multilayer perceptron to obtain a second prediction result of the gait phase and a second classification result of the human body movement intention.

[0053] In one embodiment, a flexible exoskeleton robot is provided, which can be a server. The flexible exoskeleton robot includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the flexible exoskeleton robot is used to provide computing and control capabilities. The memory of the flexible exoskeleton robot includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the flexible exoskeleton robot is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it implements the functions or steps of a multimodal data fusion method server side.

[0054] In one embodiment, a flexible exoskeleton robot is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0055] Obtain IMU data and RGB-D images collected by the flexible exoskeleton robot;

[0056] Prediction is performed based on IMU data, RGB-D images and a trained front-fusion temporal visual multimodal model to obtain a first prediction result of gait phase and a first classification result of human motion intention, wherein the front-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

[0057] The multimodal data fusion method proposed in the present invention obtains IMU data and RGB-D images collected by the flexible exoskeleton robot, and finally predicts based on the IMU data, RGB-D images and the trained pre-fusion temporal visual multimodal model to obtain the first prediction result of the gait phase and the first classification result of the human body's motion intention, wherein the pre-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder. The present invention can effectively fuse IMU data and RGB-D images, improve the control accuracy and response speed of the flexible exoskeleton robot in complex outdoor environments, and the multimodal model is significantly superior to the traditional single-modal model in gait phase prediction and motion intention recognition, and has strong practical application value.

[0058] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0059] Obtain IMU data and RGB-D images collected by the flexible exoskeleton robot;

[0060] Prediction is performed based on IMU data, RGB-D images and a trained front-fusion temporal visual multimodal model to obtain a first prediction result of gait phase and a first classification result of human motion intention, wherein the front-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

[0061] The multimodal data fusion method proposed in the present invention obtains IMU data and RGB-D images collected by the flexible exoskeleton robot, and finally predicts based on the IMU data, RGB-D images and the trained pre-fusion temporal visual multimodal model to obtain the first prediction result of the gait phase and the first classification result of the human body's motion intention, wherein the pre-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder. The present invention can effectively fuse IMU data and RGB-D images, improve the control accuracy and response speed of the flexible exoskeleton robot in complex outdoor environments, and the multimodal model is significantly superior to the traditional single-modal model in gait phase prediction and motion intention recognition, and has strong practical application value.

[0062] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or flexible exoskeleton robot can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0063] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0064] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0065] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A multimodal data fusion method, characterized in that: Applied to a flexible exoskeleton robot, the multimodal data fusion method includes: Obtain IMU data and RGB-D images collected by the flexible exoskeleton robot; Prediction is performed based on IMU data, RGB-D images and a trained front-fusion temporal visual multimodal model to obtain a first prediction result of gait phase and a first classification result of human motion intention, wherein the front-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

2. The multimodal data fusion method according to claim 1, characterized in that: The step of performing prediction based on the IMU data, the RGB-D image and the trained front-fusion temporal visual multimodal model to obtain a first prediction result of the gait phase and a first classification result of the human body movement intention includes: The IMU data and the RGB-D image are input into the front fusion temporal visual multimodal model, wherein the front fusion temporal visual multimodal model is used to add position coding based on the RGB-D image to obtain visual data, Based on the IMU data, position coding is added to obtain time series data. Adding modality-specific codes to the visual data and the time series data respectively, inputting the encoded visual data and the time series data into a Transformer encoder to obtain encoded data output by the Transformer encoder, Extract the global features of visual and temporal modalities from the encoded data and integrate them into a unified feature vector. Based on the feature vector and the cross attention mechanism, the target vector is obtained. The target vector is input into a multilayer perceptron to obtain a first prediction result of the gait phase and a first classification result of the human body movement intention.

3. The multimodal data fusion method according to claim 1, characterized in that: The multimodal data fusion method further includes: Prediction is performed based on IMU data, RGB-D images and a trained post-fusion temporal visual multimodal model to obtain a second prediction result of the gait phase and a second classification result of the human motion intention, wherein the post-fusion temporal visual multimodal model is obtained by model training based on the Transformer encoder.

4. The multimodal data fusion method according to claim 3, characterized in that: The step of performing prediction based on the IMU data, the RGB-D image and the trained post-fusion temporal visual multimodal model to obtain a second prediction result of the gait phase and a second classification result of the human body movement intention comprises: The IMU data and RGB-D images are input into a trained post-fusion temporal visual multimodal model for prediction, wherein the post-fusion temporal visual multimodal model is used to extract visual features based on the RGB-D image and the visual Transformer encoder, Each IMU data channel is divided into small blocks and projected to obtain the first tokens. Each token is feature-connected with the corresponding single-channel global tag to obtain the second token. The single-channel global tag is obtained by multi-layer attention calculation based on the token. Perform cross-attention calculation on the single-channel global tag of each second token and the preset token to obtain the full-channel global tag. The full-channel global mark and the visual features are cross-attention calculated to obtain the first vector, Based on the first vector and the cross attention mechanism, the second vector is obtained. The second vector is input into a multilayer perceptron to obtain a second prediction result of the gait phase and a second classification result of the human body movement intention.

5. A flexible exoskeleton robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multimodal data fusion method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the multimodal data fusion method according to any one of claims 1 to 4 are implemented.

Images