A method, device and apparatus for analyzing multimodal signals

Abstract

The present application discloses a method, device and equipment for analyzing multimodal signals, which collects video data of a first user's target part in motion, extracts the positions of key points of the human body from the data, and obtains a first muscle mechanical curve corresponding to the target part that changes over time. At the same time, the first EEG signal of the first user is collected and input into an EEG analysis model to obtain a classification result corresponding to the first EEG signal that changes over time. A mapping relationship is established between the first muscle mechanical curve and the classification result corresponding to the first EEG signal. The second EEG signal of the second user when imagining the target part in motion is collected and input into the EEG analysis model to obtain a classification result corresponding to the second EEG signal that changes over time. Based on the mapping relationship and the classification result corresponding to the second EEG signal that changes over time, a second muscle mechanical curve corresponding to the target part that changes over time is obtained. Quantitative analysis of muscle strength is generated through motor imagination, providing theoretical support for how to conduct exercise training for the target part.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device and equipment for analyzing multimodal signals. Background Art

[0002] Stroke is an acute cerebrovascular disease caused by sudden rupture of a brain blood vessel or blockage of a blood vessel, resulting in brain damage. It is generally categorized as ischemic or hemorrhagic. Symptoms vary depending on the location of the stroke, but generally, stroke causes damage to the central nervous system.

[0003] Stroke patients need rehabilitation training to help rebuild and reshape central nervous system functions. However, currently, doctors generally guide patients through rehabilitation training based on their experience, lacking theoretical analysis to scientifically guide rehabilitation training. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method, apparatus, and device for analyzing multimodal signals to provide theoretical support for rehabilitation training.

[0005] To solve the above problems, the technical solutions provided in the embodiments of the present application are as follows:

[0006] A method for analyzing a multimodal signal, the method comprising:

[0007] Collecting video data of a target part of a first user in motion, extracting positions of key points of the human body from the video data, and obtaining a first muscle mechanics curve corresponding to the target part that changes over time based on the positions of the key points of the human body;

[0008] collecting a first EEG signal of the first user when the target part of the user moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time;

[0009] Establishing a mapping relationship between the first muscle mechanics curve that changes with time and the classification result that changes with time corresponding to the first EEG signal;

[0010] collecting a second EEG signal of a second user when imagining movement of the target part, inputting the second EEG signal into the EEG analysis model, and obtaining a classification result corresponding to the second EEG signal that changes over time;

[0011] A second muscle mechanics curve corresponding to the target part that changes with time is obtained according to the mapping relationship and the classification result corresponding to the second EEG signal that changes with time.

[0012] In a possible implementation, the method further includes:

[0013] A muscle electrical stimulation program for the target site is obtained according to the second muscle mechanical curve.

[0014] In a possible implementation, the method further includes:

[0015] A simulated video of the target part movement is generated according to the classification result corresponding to the second EEG signal that changes over time.

[0016] In one possible implementation, the EEG analysis model includes a data conversion module and a classification module, the classification module includes a connected first convolutional layer, at least one first processing module, a first mathematical operation module, a first activation layer and a fully connected layer, and the first processing module includes a connected DEG module and a connection operation module;

[0017] The collecting of a first EEG signal of the first user when the target part of the user moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time includes:

[0018] collecting a first EEG signal of the first user when the target part moves, inputting the first EEG signal into the data conversion module, converting the first EEG signal into an initial frequency domain EEG signal, and taking the amplitude of a non-negative frequency segment in the initial frequency domain EEG signal to obtain a frequency domain EEG signal;

[0019] The frequency domain EEG signal is input into the classification module, the first convolution layer is used to perform convolution calculation on the frequency domain EEG signal, the DEG module is used to decompose the signal input to the DEG module second by second and perform calculations, the connection operation module is used to perform connection operation on the output result of the DEG module, the first mathematical operation module is used to perform target mathematical operation on the output result of the first processing module, and the classification result corresponding to the first EEG signal that changes over time is output through the first activation layer and the fully connected layer.

[0020] In one possible implementation, the EEG analysis model also includes a self-supervised learning module connected between the data conversion module and the classification module, the self-supervised learning module includes an encoder, a long short-term memory artificial neural network LSTM module and a decoder, and the self-supervised learning module is used to perform self-supervised learning on the frequency domain EEG signal to recover the frequency domain EEG signal.

[0021] In one possible implementation, the encoder includes a connected second convolutional layer and at least one first processing module, the decoder includes at least one second processing module and a third convolutional layer, and the second processing module includes an inverse DEG module and a connection operation module.

[0022] In one possible implementation, the EEG analysis model also includes an EEG signal graph structure establishment module connected between the self-supervised learning module and the classification module; the EEG signal graph structure establishment module is used to convert frequency domain EEG signals into a graph neural network.

[0023] In one possible implementation, the DEG module includes at least one connected third processing module and a second mathematical operation module, the third processing module includes a DMG module and a second activation layer; the second mathematical operation module is used to perform weighted multiplication on the output results of each of the third processing modules.

[0024] In a possible implementation, the method further includes:

[0025] Inputting the first EEG signal into an EEG analysis model to obtain a brain region activation location map corresponding to the first EEG signal;

[0026] and / or,

[0027] The second EEG signal is input into the EEG analysis model to obtain a brain region activation position map corresponding to the second EEG signal.

[0028] A multimodal signal analysis device, comprising:

[0029] a first acquisition unit, configured to acquire video data of a first user's target part in motion, extract positions of key points of the human body from the video data, and obtain a first muscle mechanics curve corresponding to the target part that changes over time based on the positions of the key points of the human body;

[0030] a second acquisition unit, configured to acquire a first EEG signal of the first user when the target part of the user moves, input the first EEG signal into an EEG analysis model, and obtain a classification result corresponding to the first EEG signal that changes over time;

[0031] an establishing unit, configured to establish a mapping relationship between the first muscle mechanics curve that varies with time and the classification result that varies with time corresponding to the first EEG signal;

[0032] a third acquisition unit, configured to acquire a second EEG signal of the second user when the user imagines movement of the target part, input the second EEG signal into the EEG analysis model, and obtain a classification result corresponding to the second EEG signal that changes over time;

[0033] The first obtaining unit is configured to obtain a second muscle mechanics curve corresponding to the target part that varies with time according to the mapping relationship and the classification result corresponding to the second EEG signal that varies with time.

[0034] A multimodal signal analysis device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multimodal signal analysis method described above is implemented.

[0035] A computer-readable storage medium stores instructions, which, when executed on a terminal device, enable the terminal device to execute the multimodal signal analysis method as described above.

[0036] It can be seen that the embodiments of the present application have the following beneficial effects:

[0037] The embodiment of the present application collects video data of the target part of the first user when it moves. The positions of the key points of the human body can be extracted from the video data, thereby obtaining a first muscle mechanics curve corresponding to the target part that changes over time. While the target part of the first user moves, the first EEG signal of the first user is collected, and the first EEG signal is input into the EEG analysis model to obtain a classification result corresponding to the first EEG signal that changes over time. A mapping relationship is established between the first muscle mechanics curve that changes over time and the classification result corresponding to the first EEG signal. Then, a second EEG signal is collected when the second user imagines the target part moving, and the second EEG signal is also input into the EEG analysis model to obtain a classification result corresponding to the second EEG signal that changes over time. Based on the established mapping relationship and the classification result corresponding to the second EEG signal that changes over time, a second muscle mechanics curve corresponding to the target part that changes over time can be obtained. Through motion imagination, a quantitative analysis of muscle force mechanics can be generated, thereby providing theoretical support for how to conduct exercise training for the target part. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of an exemplary application scenario provided in an embodiment of the present application;

[0039] Figure 2 A flowchart of a multimodal signal analysis method provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of key points of the human body provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of an EEG analysis model provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a classification module provided in an embodiment of the present application;

[0043] Figure 6A schematic diagram of another EEG analysis model provided in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of a self-supervised learning module provided in an embodiment of the present application;

[0045] Figure 8 A schematic diagram of the specific structure of the self-supervised learning module provided in an embodiment of the present application;

[0046] Figure 9 A schematic diagram of a DEG module provided in an embodiment of the present application;

[0047] Figure 10 A schematic diagram of a DMG module provided in an embodiment of the present application;

[0048] Figure 11 A schematic diagram of a multimodal signal analysis device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0050] In order to facilitate understanding and explanation of the technical solutions provided by the embodiments of the present application, the background technology of the present application will be described below.

[0051] Stroke is an acute cerebrovascular disease caused by sudden rupture of a cerebral blood vessel or blockage of a blood vessel, resulting in brain damage. It is generally categorized as ischemic or hemorrhagic. Symptoms vary depending on the location of the stroke, but generally, stroke can lead to damage to the central nervous system, such as hemiplegia.

[0052] Stroke patients need rehabilitation training to help rebuild and reshape central nervous system functions. However, currently, doctors generally guide patients through rehabilitation training based on their experience, lacking theoretical analysis to scientifically guide rehabilitation training.

[0053] Based on this, the present invention provides an analysis method based on multimodal signals such as EEG and video. By establishing a mapping relationship between these signals, a quantitative analysis of human movement and muscle strength can be obtained based on motor imagery. This provides theoretical support for how patients should conduct exercise training, and has high accuracy, robustness, and practicality.

[0054] In order to facilitate understanding of the multimodal signal analysis method provided in the embodiment of the present application, the following Figure 1 See the example scenario shown. Figure 1As shown in the figure, this figure is a schematic diagram of an exemplary application scenario provided by an embodiment of the present application.

[0055] First, the first user performs movement of the target part, such as extending the fingers and making a fist of the hand, and collects video data of the movement of the target part through a video acquisition device, and simultaneously collects the first EEG signal through a brain-computer interface device.

[0056] Video data is analyzed to obtain the three-dimensional coordinates (x, y, z) of key points on the human body (such as joints). Dynamic analysis is performed on the key points' positional information to generate the time-varying primary muscle force curves corresponding to the target areas. For example, the time-varying primary muscle force curves for the extensor and flexor muscles of the hand are shown.

[0057] At the same time, the first EEG signal generated by the first user during actual movement behavior enters the EEG analysis model provided in the embodiment of the present application to obtain the classification results corresponding to the first EEG signal that change over time, such as the binary EEG analysis results of clenching a fist or extending a finger.

[0058] The first muscle mechanics curve that changes with time is mapped to the classification result that changes with time corresponding to the first EEG signal to obtain a mapping relationship.

[0059] The second user then performs motor imagery, such as imagining themselves extending their fingers and making a fist, while simultaneously collecting the corresponding second EEG signal. The second EEG signal is input into the EEG analysis model to obtain a classification result corresponding to the second EEG signal that changes over time. Then, combined with the mapping relationship, a second muscle mechanics curve corresponding to the target area that changes over time can be obtained. In addition, the second muscle mechanics curve can also be used to obtain a muscle electrical stimulation plan for the target area. These analysis results can be used to provide active training plans for rehabilitation training for patients with central nervous system damage after stroke.

[0060] Those skilled in the art will understand that Figure 1 The framework diagram shown is only an example in which the embodiments of the present application can be implemented. The scope of application of the embodiments of the present application is not limited by any aspect of the framework.

[0061] To facilitate understanding of the present application, a multimodal signal analysis method provided in an embodiment of the present application is described below with reference to the accompanying drawings.

[0062] See also Figure 2 As shown in FIG, this figure is a flow chart of a multimodal signal analysis method provided in an embodiment of the present application, as shown in FIG. Figure 2 As shown, the method may include S201-S205:

[0063] S201: Collect video data of a target part of a first user in motion, extract positions of key points of the human body from the video data, and obtain a first muscle mechanics curve corresponding to the target part that changes over time based on the positions of the key points of the human body.

[0064] When the target part of the first user moves, the video data of the target part movement is collected through the video capture device. For example, the target part is the hand, and the target part movement is the alternating movement of extending the fingers and clenching the fist. Then, the collected video data is analyzed using MediaPipe to obtain the three-dimensional coordinates (x, y, z) of the position of each key point of the human body (such as a joint point). MediaPipe is a machine learning framework built by Google for processing time series data such as video and audio. For example, Figure 3 As shown, a schematic diagram of the positions of key points of the hand body is shown.

[0065] Then, the OpenSim tool is used to perform a dynamic analysis of the key points of the human body. The input is the coordinate data of the position of the key points of the human body that changes over time and the skeletal muscle model of the target part (for example, the skeletal muscle model of the hand). First, the data format is converted to convert the motion trajectory data of the key points of the human body that changes over time into data that can be parsed by OpenSim. Then, the skeletal muscle model is scaled. The general skeletal muscle model is scaled according to the position of the sampling point to generate a skeletal muscle model that is adapted to the first user's target part. Then, kinematic fitting, i.e., inverse kinematics calculation, is performed. The motion trajectory is subjected to inverse kinematics simulation analysis, and then residual calculation and dynamic fitting analysis are performed based on the inverse kinematics simulation results. Finally, forward dynamics simulation analysis is performed to obtain the muscle force and muscle activation state related to the target part, and a first muscle mechanical curve corresponding to the target part that changes over time is generated. The first muscle mechanical curve represents the muscle force value of the corresponding muscle corresponding to the target part that changes over time. For example, a first muscle mechanical curve of the extensor and flexor muscles of the hand that changes over time is generated.

[0066] S202: Collecting a first EEG signal of the first user when the target part of the user moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time.

[0067] When the first user performs a target body part movement, a first EEG signal is synchronously collected. The first EEG signal is input into the EEG analysis model to obtain a classification result corresponding to the first EEG signal that varies over time, such as a binary classification result of fist clenching or finger extension corresponding to the first EEG signal that varies over time.

[0068] The embodiment of the present application provides an EEG analysis model. The specific structure of the EEG analysis model and the specific implementation of this step can be found in the subsequent embodiments.

[0069] S203: Establishing a mapping relationship between the first muscle mechanics curve that changes with time and the classification result that changes with time corresponding to the first EEG signal.

[0070] Since the video data and the first EEG signal are collected synchronously, the classification results corresponding to the first muscle mechanics curve and the first EEG signal can also change over time based on the same time axis. A mapping relationship can be established between the first muscle mechanics curve that changes over time and the classification results corresponding to the first EEG signal. For example, the muscle force value of the first muscle mechanics curve corresponding to the first moment is in the first position, and the classification result corresponding to the first EEG signal at the first moment is fist clenching, and the muscle force value of the first muscle mechanics curve corresponding to the second moment is in the second position, and the classification result corresponding to the first EEG signal at the second moment is finger extension.

[0071] S204: collecting a second EEG signal of the second user when he / she imagines the target part moving, inputting the second EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the second EEG signal that changes over time.

[0072] In this embodiment, the second user performs motor imagination, that is, imagines the target part to move, but the actual target part does not move, for example, imagines the hand to perform alternating movements of extending the fingers and making a fist. At this time, the second EEG signal of the second user is collected. The second EEG signal is input into the EEG analysis model, and the classification results corresponding to the second EEG signal that change over time can be obtained, such as the binary classification results of making a fist or extending the fingers that change over time corresponding to the second EEG signal. It can be understood that the second user can be the same as the first user or different from the first user. For example, the second user can be a patient who cannot move the target part.

[0073] In addition, the specific implementation process of inputting the second EEG signal into the EEG analysis model to obtain the classification result is consistent with the specific implementation process of inputting the first EEG signal into the EEG analysis model to obtain the classification result, and will not be further described in order to avoid repetition.

[0074] S205: Obtaining a second muscle mechanics curve corresponding to the target part that varies with time based on the mapping relationship and the classification result corresponding to the second EEG signal that varies with time.

[0075] Based on the established mapping relationship and the classification result corresponding to the second EEG signal that changes over time, a second muscle mechanics curve corresponding to the target part that changes over time can be obtained when the second user performs the target part exercise. For example, the muscle force value of the first muscle mechanics curve corresponding to the first moment is in the first position, and at the same time, the classification result corresponding to the first EEG signal at the first moment is fist clenching. If the classification result corresponding to the second EEG signal at the first moment is also fist clenching, that is, the second user has imagined fist clenching, then theoretically, if the second user performs the fist clenching action, the muscle force value corresponding to the first moment should be in the first position. In this way, the muscle force value of the second muscle mechanics curve corresponding to the first moment can be obtained to be in the first position.

[0076] In a possible implementation, the method may further include obtaining a muscle electrical stimulation program for the target site according to the second muscle mechanics curve.

[0077] The second muscle mechanics curve can be used to obtain the theoretical muscle strength value of the corresponding muscle corresponding to the target part when the second user is performing motor imagination, thereby providing theoretical support for how the corresponding muscle corresponding to the target part should move. Based on the second muscle mechanics curve, a muscle electrical stimulation scheme for the target part is obtained. If the muscle electrical stimulation scheme is executed on the second user, the corresponding muscle strength value can be generated for the muscle at the target part, and the muscle can be trained in a feedback-type manner, thereby repeatedly stimulating the cerebral cortex under a certain condition through feedback. After long-term conditional repeated training, a corresponding conditioned reflex can be formed, and an excitation focus can be formed in the corresponding part of the cerebral cortex, which is helpful for the reconstruction and remodeling of the central nervous system function. It should be noted that the embodiment of the present application only generates a muscle electrical stimulation scheme for the doctor's reference. How to execute the muscle electrical stimulation scheme needs to be set by the doctor according to the actual situation of the patient, and does not belong to the execution part of the embodiment of the present application.

[0078] In a possible implementation, the method may further include generating a simulated video of the target part movement according to the classification result corresponding to the second EEG signal that changes over time.

[0079] In an embodiment of the present application, a simulation video of the movement of the target part can also be generated based on the classification result of the second EEG signal. For example, the Unity tool is used to generate a visual simulation video of clenching a fist and extending a finger. If the simulation video of the movement of the target part is played to the second user to provide visual feedback to the second user, an exercise mechanism for the patient can also be established through visual feedback. Through continuous visual feedback, the main motor cortex of the human brain is stimulated, the electrical activity and excitability of the cortex are affected, brain function remodeling is promoted, and motor function recovery is induced. It should be noted that the embodiment of the present application only generates a simulation video of the movement of the target part for the doctor's reference. How to play the simulation video of the movement of the target part needs to be set by the doctor according to the actual situation of the patient, and does not belong to the execution part of the embodiment of the present application.

[0080] Based on the description of S201-S205, the embodiment of the present application collects video data of the target part of the first user when it moves, and the positions of the key points of the human body can be extracted from the video data, thereby obtaining the first muscle mechanics curve corresponding to the target part that changes over time. While the target part of the first user moves, the first EEG signal of the first user is collected, and the first EEG signal is input into the EEG analysis model to obtain the classification result corresponding to the first EEG signal that changes over time. A mapping relationship is established between the first muscle mechanics curve that changes over time and the classification result corresponding to the first EEG signal. Then, the second EEG signal of the second user is collected when he imagines the target part moving, and the second EEG signal is also input into the EEG analysis model to obtain the classification result corresponding to the second EEG signal that changes over time. Based on the established mapping relationship and the classification result corresponding to the second EEG signal that changes over time, the second muscle mechanics curve corresponding to the target part that changes over time can be obtained. Through motion imagination, a quantitative analysis of muscle force mechanics can be generated, thereby providing theoretical support for how to conduct exercise training for the target part.

[0081] The present application embodiment provides an EEG analysis model, see Figure 4 As shown, the EEG analysis model includes a data conversion module 401 and a classification module 402, wherein the classification module includes a connected first convolution layer, at least one first processing module, a first mathematical operation module, a first activation layer and a fully connected layer, and the first processing module includes a connected DEG (Direct Edge Graphic) module and a connection operation module. Figure 5 As shown, a schematic diagram of a classification module is shown, and an example in which the classification module includes two first processing modules is used for explanation.

[0082] The specific implementation of S202 collecting a first EEG signal of the first user when the target part moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time may include:

[0083] A1: Collect a first EEG signal of the first user when the target part moves, input the first EEG signal into a data conversion module, convert the first EEG signal into an initial frequency domain EEG signal, and take the amplitude of the non-negative frequency segment in the initial frequency domain EEG signal to obtain a frequency domain EEG signal.

[0084] Because EEG signals are closely related to electrical activity signals within a certain frequency band, it is believed that short-term frequency-domain input signals contain more information than time-domain signals. Therefore, the data conversion module first performs a Fourier transform on the input signal (e.g., the first EEG signal) to obtain the transformed initial frequency-domain EEG signal. It then takes a non-negative frequency band and uses its amplitude as the frequency-domain EEG signal. This resulting frequency-domain EEG signal is both half the length, reducing training or application time and improving efficiency, and contains more frequency-domain information, making it ideal for classification tasks involving EEG signals, which are closely related to frequency.

[0085] A2: Input the frequency domain EEG signal into the classification module, use the first convolution layer to perform convolution calculation on the frequency domain EEG signal, use the DEG module to decompose the signal input to the DEG module second by second and perform calculations, use the connection operation module to connect the output results of the DEG module, use the first mathematical operation module to perform target mathematical operations on the output results of the first processing module, and output the classification results corresponding to the first EEG signal that change over time through the first activation layer and the fully connected layer.

[0086] The frequency domain EEG signal is input into the classification module, and the frequency domain EEG signal is convolutionally calculated using the first convolutional layer. The output of the first convolutional layer is decomposed second by second and entered into the DEG module for calculation, and the output result is connected. If the number of the first processing modules is 2, the calculation process of the DEG module and the connection operation module is passed twice. After that, the output result of the first processing module is input into the first mathematical operation module for a series of mathematical operations, such as splitting the output result of the first processing module into two parts, left and right, and then calculating using the formula right+(1-right)*left. The output result of the first mathematical operation module enters the activation function, and then passes through 2 fully connected layers to obtain the classification result corresponding to the first EEG signal that changes over time, such as obtaining a binary classification result of extending a finger or making a fist. The calculation process of the DEG module can be found in the subsequent embodiments.

[0087] In one possible implementation, see Figure 6 As shown, the EEG analysis model may further include a self-supervised learning module 403 connected between the data conversion module and the classification module. Figure 7 As shown, the self-supervised learning module may include an encoder, an LSTM (Long Short-Term Memory, long short-term memory artificial neural network) module and a decoder. The self-supervised learning module is used to perform self-supervised learning on the frequency domain EEG signal to regain the frequency domain EEG signal.

[0088] The EEG analysis model of the embodiment of the present application may also include a self-supervised learning module. The self-supervised learning module has the ability to let a sample of length n predict the next sample of length n. Adding a self-supervised learning module to the EEG analysis model can allow the EEG analysis model to have the ability to predict future EEG signal information, which can improve the accuracy of the EEG analysis model in classifying EEG signals. The training of the self-supervised learning module encodes the input EEG signal of length n through an encoder, performs LSTM operation, and then decodes it through a decoder to obtain an output signal, which is compared and fitted with the next EEG signal of length n, and iteratively optimized according to the loss to obtain the self-supervised learning module. In the process of using the self-supervised learning module, the frequency domain EEG signal is encoded, LSTM operated and decoded, and the frequency domain EEG signal can be obtained again.

[0089] See also Figure 8 FIG2 shows a schematic diagram of a specific implementation of a self-supervised learning module, wherein the encoder includes a connected second convolutional layer and at least one first processing module, the first processing module includes a connected DEG module and a connection operation module. The decoder includes at least one second processing module and a third convolutional layer, the second processing module includes an inverse DEG module and a connection operation module. The number of first processing modules and second processing modules is the same, and the number of first processing modules and second processing modules is 2 for illustration.

[0090] In an embodiment of the present application, the frequency domain EEG signal is decomposed second by second (the signal format per second is the number of EEG electrodes * signal length), enters the DEG module for operation, and the output result is connected. If the number of the first processing modules is 2, the calculation process of the DEG module and the connection operation module is passed twice to complete the operation of the encoder part. The output result of the encoder is first subjected to LSTM operation and memory learning, and then the inverse DEG module performs the reverse DEG operation operation and performs the connection operation. The calculation process of the same inverse DEG module and the connection operation module is passed twice. Finally, after the convolution operation, the output result with the same size as the frequency domain EEG signal is obtained, and the output result is reused as the frequency domain EEG signal.

[0091] In this way, the EEG analysis model has the ability to predict the EEG signals of the next stage, making the EEG analysis model more accurate in classification.

[0092] In one possible implementation, see Figure 6 As shown, the EEG analysis model also includes an EEG signal graph structure establishment module 404 connected between the self-supervised learning module and the classification module. The EEG signal graph structure establishment module is used to convert the frequency domain EEG signals into a graph neural network. Simultaneously, the EEG signal graph structure establishment module is also used to generate the adjacency matrix of the graph neural network.

[0093] The graph structure of EEG signals can be divided into two forms: one is an anatomical approximation based on the natural geometric shape of EEG signal nodes, and the other is a functional approximation based on the dynamic links of EEG signal nodes in the brain. The graph neural network established in the embodiment of the present application is a functional approximation based on the dynamic links of EEG signal nodes in the brain. The EEG signal is represented in the form of G = {V, E, A}, which is a graph neural network, where V represents the EEG signal node, E represents the edge representing the connection relationship between each signal node, and A represents the adjacency matrix.

[0094] In the specific implementation, the cross-correlation coefficients of the frequency-domain EEG signals from different signal nodes at time T are calculated to form the connection coefficient relationship solution set between the edges of each signal node, and then the adjacency matrix A is generated (the size of the adjacency matrix is ​​the number of signal nodes * the number of signal nodes, and the value of each position in the matrix is ​​the corresponding cross-correlation coefficient). Then the following operation is performed: the degree matrix d of the adjacency matrix is ​​calculated, and then the inverse of each element of the degree matrix is ​​taken. Then a diagonal matrix is ​​generated from this newly generated matrix, that is, the elements in the new matrix D can be expressed as Then use this D and A to do matrix product, and then transpose the result to get the adjacency matrix to be solved.

[0095] See also Figure 9 As shown, the DEG module in the above embodiment is described. The DEG module includes at least one connected third processing module and a second mathematical operation module. The third processing module includes a DMG (Distributed Matrix Graphic) module and a second activation layer. The second mathematical operation module is used to perform weighted multiplication on the output results of each third processing module. The number of third processing modules is 2 as an example for description.

[0096] The signal input to the DEG module first enters the DMG module, and then the output result enters the second activation layer, and then enters the DMG module again, and the output result enters the second activation layer again. The first activation output result and the second activation output result are passed through the second mathematical operation module (for example, weighted multiplication, weight*activation1*activation2, where weight is the weight, activation1 is the first activation output result, and activation2 is the second activation output result) to obtain the final output result of the DEG module.

[0097] Among them, see Figure 10As shown, the DMG module can first perform a matrix product operation on the input signal and the adjacency matrix to obtain a first output result. The first output result is then subjected to two branch operations respectively. The first branch is a matrix product operation with the adjacency matrix to obtain a second output result. The second branch is first convolutionalized by the fourth convolution layer, and then connected with the second output result through a connection operation module. The output result of the connection operation module is then convolutionalized by the fourth convolution layer, and the result of the matrix product of the output result of the fourth convolution layer, the second output result and the adjacency matrix is ​​connected through a connection operation module, and then connected with the output of the input signal after the full connection layer through a connection operation module. Finally, the output result of the connection operation module is subjected to a mathematical operation by the third mathematical operation module to obtain the output result of the DMG module. In addition, if the EEG analysis model does not include an EEG signal graph structure establishment module, the adjacency matrix is ​​not involved in the calculation in the above calculation process of the DMG module.

[0098] The EEG analysis model based on the above structure can make the classification results more accurate when classifying EEG signals.

[0099] In addition, based on the EEG signal graph structure establishment module in the EEG analysis model, the first EEG signal can be input into the EEG analysis model to obtain the brain area activation position map corresponding to the first EEG signal; and / or, the second EEG signal can be input into the EEG analysis model to obtain the brain area activation position map corresponding to the second EEG signal.

[0100] Since the EEG analysis model can generate a graph neural network based on EEG signals, it can display the positional relationship of activated nodes and edges and the corresponding weight relationship. Therefore, the output results of the EEG analysis model can display the activation position and activation degree of the brain area, and can quantitatively evaluate which specific brain area has been activated and the degree of activation. In this way, the degree of activation of the central nervous system can be quantitatively evaluated.

[0101] Based on the multimodal signal analysis method provided in the above method embodiment, the embodiment of the present application also provides a multimodal signal analysis device, which will be described below with reference to the accompanying drawings.

[0102] See also Figure 11 As shown in FIG, this figure is a structural diagram of a multimodal signal analysis device provided in an embodiment of the present application. Figure 11 As shown, the multimodal signal analysis device includes:

[0103] The first acquisition unit 1101 is configured to acquire video data of a first user's target part in motion, extract positions of key points of the human body from the video data, and obtain a first muscle mechanics curve corresponding to the target part that changes over time based on the positions of the key points of the human body;

[0104] The second collecting unit 1102 is configured to collect a first EEG signal of the first user when the target part of the user moves, input the first EEG signal into an EEG analysis model, and obtain a classification result corresponding to the first EEG signal that changes over time;

[0105] An establishing unit 1103 is configured to establish a mapping relationship between the first muscle mechanics curve that varies with time and the classification result that varies with time corresponding to the first EEG signal;

[0106] The third collecting unit 1104 is configured to collect a second EEG signal of the second user when the user imagines movement of the target part, input the second EEG signal into the EEG analysis model, and obtain a classification result corresponding to the second EEG signal that changes over time;

[0107] The first obtaining unit 1105 is configured to obtain a second muscle mechanics curve corresponding to the target part that varies with time according to the mapping relationship and the classification result corresponding to the second EEG signal that varies with time.

[0108] In a possible implementation, the apparatus further includes:

[0109] The second obtaining unit is used to obtain a muscle electrical stimulation program for the target part according to the second muscle mechanical curve.

[0110] In a possible implementation, the method further includes:

[0111] A generating unit is configured to generate a simulated video of the movement of the target part according to the classification result corresponding to the second EEG signal that changes over time.

[0112] In one possible implementation, the EEG analysis model includes a data conversion module and a classification module, the classification module includes a connected first convolutional layer, at least one first processing module, a first mathematical operation module, a first activation layer and a fully connected layer, and the first processing module includes a connected DEG module and a connection operation module;

[0113] The second acquisition unit includes:

[0114] an acquisition subunit, configured to acquire a first EEG signal of the first user when the target part moves, input the first EEG signal into the data conversion module, convert the first EEG signal into an initial frequency-domain EEG signal, and obtain a frequency-domain EEG signal by taking the amplitude of a non-negative frequency segment of the initial frequency-domain EEG signal;

[0115] The classification subunit is used to input the frequency domain EEG signal into the classification module, use the first convolution layer to perform convolution calculation on the frequency domain EEG signal, use the DEG module to decompose the signal input to the DEG module second by second and perform calculations, use the connection operation module to perform connection operation on the output result of the DEG module, use the first mathematical operation module to perform target mathematical operation on the output result of the first processing module, and output the classification result corresponding to the first EEG signal that changes over time through the first activation layer and the fully connected layer.

[0116] In one possible implementation, the EEG analysis model also includes a self-supervised learning module connected between the data conversion module and the classification module, the self-supervised learning module includes an encoder, a long short-term memory artificial neural network LSTM module and a decoder, and the self-supervised learning module is used to perform self-supervised learning on the frequency domain EEG signal to recover the frequency domain EEG signal.

[0117] In one possible implementation, the encoder includes a connected second convolutional layer and at least one first processing module, the decoder includes at least one second processing module and a third convolutional layer, and the second processing module includes an inverse DEG module and a connection operation module.

[0118] In one possible implementation, the EEG analysis model also includes an EEG signal graph structure establishment module connected between the self-supervised learning module and the classification module; the EEG signal graph structure establishment module is used to convert frequency domain EEG signals into a graph neural network.

[0119] In one possible implementation, the DEG module includes at least one connected third processing module and a second mathematical operation module, the third processing module includes a DMG module and a second activation layer; the second mathematical operation module is used to perform weighted multiplication on the output results of each of the third processing modules.

[0120] In a possible implementation, the apparatus further includes:

[0121] a third obtaining unit, configured to input the first EEG signal into an EEG analysis model to obtain a brain region activation position map corresponding to the first EEG signal;

[0122] and / or,

[0123] The fourth obtaining unit is used to input the second EEG signal into the EEG analysis model to obtain a brain area activation position map corresponding to the second EEG signal.

[0124] In addition, an embodiment of the present application also provides a multimodal signal analysis device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the multimodal signal analysis method as described in any one of the above items is implemented.

[0125] In addition, an embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes the multimodal signal analysis method as described in any one of the above items.

[0126] In this way, the embodiment of the present application collects video data of the target part of the first user when it moves, and the positions of the key points of the human body can be extracted from the video data, thereby obtaining the first muscle mechanics curve corresponding to the target part that changes over time. While the target part of the first user moves, the first EEG signal of the first user is collected, and the first EEG signal is input into the EEG analysis model to obtain the classification result corresponding to the first EEG signal that changes over time. A mapping relationship is established between the first muscle mechanics curve that changes over time and the classification result corresponding to the first EEG signal. Then the second EEG signal of the second user is collected when he imagines the target part moving, and the second EEG signal is also input into the EEG analysis model to obtain the classification result corresponding to the second EEG signal that changes over time. Based on the established mapping relationship and the classification result corresponding to the second EEG signal that changes over time, the second muscle mechanics curve corresponding to the target part that changes over time can be obtained. Through motion imagination, quantitative analysis of muscle force mechanics can be generated, thereby providing theoretical support for how to conduct exercise training for the target part.

[0127] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0128] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0129] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0130] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0131] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for analyzing multimodal signals, characterized in that: The method comprises: Collecting video data of a target part of a first user in motion, extracting positions of key points of the human body from the video data, and obtaining a first muscle mechanics curve corresponding to the target part that changes over time based on the positions of the key points of the human body; the first muscle mechanics curve representing a muscle force value of a corresponding muscle corresponding to the target part that changes over time; collecting a first EEG signal of the first user when the target part of the user moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time; Establishing a mapping relationship between the first muscle mechanics curve that changes with time and the classification result that changes with time corresponding to the first EEG signal; collecting a second EEG signal of a second user when imagining movement of the target part, inputting the second EEG signal into the EEG analysis model, and obtaining a classification result corresponding to the second EEG signal that changes over time; Obtaining a second muscle mechanics curve corresponding to the target part that varies with time based on the mapping relationship and the classification result corresponding to the second EEG signal that varies with time; the second muscle mechanics curve represents a theoretical muscle force value of the corresponding muscle corresponding to the target part that varies with time; The EEG analysis model includes a data conversion module and a classification module, the classification module includes a connected first convolution layer, at least one first processing module, a first mathematical operation module, a first activation layer and a fully connected layer, and the first processing module includes a connected DEG module and a connection operation module; The collecting of a first EEG signal of the first user when the target part of the user moves, inputting the first EEG signal into an EEG analysis model, and obtaining a classification result corresponding to the first EEG signal that changes over time includes: collecting a first EEG signal of the first user when the target part moves, inputting the first EEG signal into the data conversion module, converting the first EEG signal into an initial frequency domain EEG signal, and taking the amplitude of a non-negative frequency segment in the initial frequency domain EEG signal to obtain a frequency domain EEG signal; Input the frequency domain EEG signal into the classification module, use the first convolution layer to perform convolution calculation on the frequency domain EEG signal, use the DEG module to decompose the signal input to the DEG module second by second and perform calculation, use the connection operation module to perform connection operation on the output result of the DEG module, use the first mathematical operation module to perform target mathematical operation on the output result of the first processing module, and output the classification result corresponding to the first EEG signal that changes over time through the first activation layer and the fully connected layer; The EEG analysis model also includes a self-supervised learning module connected between the data conversion module and the classification module. The self-supervised learning module includes an encoder, a long short-term memory artificial neural network LSTM module and a decoder. The self-supervised learning module is used to perform self-supervised learning on the frequency domain EEG signal to obtain the frequency domain EEG signal again.

2. The method according to claim 1, characterized in that The method further comprises: A muscle electrical stimulation program for the target site is obtained according to the second muscle mechanical curve.

3. The method according to claim 1, characterized in that The method further comprises: A simulated video of the target part movement is generated according to the classification result corresponding to the second EEG signal that changes over time.

4. The method according to claim 1, wherein The encoder includes a connected second convolution layer and at least one first processing module, the decoder includes at least one second processing module and a third convolution layer, and the second processing module includes an inverse DEG module and a connection operation module.

5. The method according to claim 1, characterized in that The EEG analysis model also includes an EEG signal graph structure establishment module connected between the self-supervised learning module and the classification module; the EEG signal graph structure establishment module is used to convert frequency domain EEG signals into a graph neural network.

6. The method according to any one of claims 1, 4-5, characterized in that: The DEG module includes at least one connected third processing module and a second mathematical operation module, the third processing module includes a DMG module and a second activation layer; the second mathematical operation module is used to perform weighted multiplication on the output results of each of the third processing modules.

7. The method according to claim 5, characterized in that The method further comprises: Inputting the first EEG signal into an EEG analysis model to obtain a brain region activation location map corresponding to the first EEG signal; and / or, The second EEG signal is input into the EEG analysis model to obtain a brain region activation position map corresponding to the second EEG signal.

8. A multimodal signal analysis device, characterized in that: The device comprises: a first acquisition unit configured to acquire video data of a target part of a first user in motion, extract positions of key points of the human body from the video data, and obtain a first muscle mechanics curve corresponding to the target part that varies over time based on the positions of the key points of the human body; the first muscle mechanics curve representing a muscle force value of a corresponding muscle corresponding to the target part that varies over time; a second acquisition unit, configured to acquire a first EEG signal of the first user when the target part of the user moves, input the first EEG signal into an EEG analysis model, and obtain a classification result corresponding to the first EEG signal that changes over time; an establishing unit, configured to establish a mapping relationship between the first muscle mechanics curve that varies with time and the classification result that varies with time corresponding to the first EEG signal; a third acquisition unit, configured to acquire a second EEG signal of the second user when the user imagines movement of the target part, input the second EEG signal into the EEG analysis model, and obtain a classification result corresponding to the second EEG signal that changes over time; a first obtaining unit, configured to obtain, based on the mapping relationship and the classification result corresponding to the second EEG signal that varies over time, a second muscle mechanical curve corresponding to the target part that varies over time; wherein the second muscle mechanical curve represents a theoretical muscle force value of the corresponding muscle corresponding to the target part that varies over time; The EEG analysis model includes a data conversion module and a classification module, the classification module includes a connected first convolution layer, at least one first processing module, a first mathematical operation module, a first activation layer and a fully connected layer, and the first processing module includes a connected DEG module and a connection operation module; The second acquisition unit includes: an acquisition subunit, configured to acquire a first EEG signal of the first user when the target part moves, input the first EEG signal into the data conversion module, convert the first EEG signal into an initial frequency-domain EEG signal, and obtain a frequency-domain EEG signal by taking the amplitude of a non-negative frequency segment of the initial frequency-domain EEG signal; A classification subunit is configured to input the frequency domain EEG signal into a classification module, perform convolution calculation on the frequency domain EEG signal using the first convolution layer, decompose the signal input to the DEG module second by second using the DEG module, perform a connection operation on the output result of the DEG module using the connection operation module, perform a target mathematical operation on the output result of the first processing module using the first mathematical operation module, and output the classification result corresponding to the first EEG signal that changes over time through the first activation layer and the fully connected layer; The EEG analysis model also includes a self-supervised learning module connected between the data conversion module and the classification module. The self-supervised learning module includes an encoder, a long short-term memory artificial neural network LSTM module and a decoder. The self-supervised learning module is used to perform self-supervised learning on the frequency domain EEG signal to obtain the frequency domain EEG signal again.

9. A multimodal signal analysis device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for analyzing a multimodal signal according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the multimodal signal analysis method according to any one of claims 1 to 7.

Images