A multimodal interaction method and system based on metaverse and brain-computer interface

Abstract

The present invention discloses a multimodal interaction method and system based on metaverse and brain-computer interface, wherein the method comprises: inputting a real-time grayscale image into a convolutional neural network model, and outputting an image recognition result; obtaining multimodal signal data based on a brain-computer interface under the display of the image recognition result; performing feature extraction and feature classification on all signal data of different modes in batches, selecting different layers for feature fusion based on different batches of data processing processes, and obtaining recognition and classification results; converting the generation conditions of the control commands set by the algorithm based on the recognition and classification results into control commands; and the metaverse image manipulation module changes the state of the virtual character and the real-time grayscale image of the metaverse according to the control commands. In the present invention, individualized signal features are identified and their weights are adjusted to reduce or directly eliminate the influence of the individualized signal features on the signal processing process, taking into account individual differences and improving the accuracy of individual motivation analysis.

Description

Technical Field

[0001] The present invention relates to the field of human-computer interaction technology, and in particular to a multimodal interaction method and system based on metaverse and brain-computer interface. Background Art

[0002] A brain-computer interface is a direct connection between the human or animal brain and an external computing device, used for computers to receive instructions from the brain in the form of electroencephalogram signals. The metaverse is a virtual world built by humans using digital technology, which maps or transcends the real world and can interact with the real world. It is a digital living space with a new social system. The brain-computer interface provides basic technical support for the interaction between humans and the virtual world. The brain-computer interface needs to be connected to the brain in real time to realize data collection and command issuance, so as to record brain signals, and convert the brain signals into instructions through a series of processing, learning, and training, thereby controlling the actions of virtual people in the metaverse.

[0003] Traditional single-modality brain-computer interfaces still have certain limitations in performance. In order to improve the accuracy and reliability of brain-computer interfaces, existing technologies use multimodal brain-computer interface technology to fuse multiple signals such as EEG signals, eye movement signals, and electromyography signals, and perform signal fusion and signal decoding on multimodal signals. Among them, signal fusion uses feature-level fusion and decision-level fusion methods to fuse multiple signals, feature-level fusion that fuses the features of different signals, and decision-level fusion that fuses the decision results of different signals, respectively, to improve the signal discrimination and accuracy, and the fused signals are converted into corresponding control commands.

[0004] The existing multimodal brain-computer interface technology usually uses multiple signals such as EEG signals, eye movement signals, and electromyography signals. In addition, in order to perceive the individual's own movements, the movement state of the brain can be collected through a gyroscope, and different control commands can be converted according to the movement laws of the brain. Based on this, the multimodal brain-computer interface perceives the user's movement intention by identifying the changes in the brain itself. However, there are individual differences in the multimodal data processing process of the brain-computer interface. Under the specific performance of some individuals, the habitual signal characteristics of an individual may be mistakenly analyzed as an erroneous control signal, resulting in a large difference in the accuracy of the brain-computer interface interaction results when used by different individuals. Summary of the invention

[0005] To this end, the present invention provides a multimodal interaction method and system based on the metaverse and brain-computer interface, which effectively solves the technical problem that there are individual differences in the multimodal data processing process of the brain-computer interface in the prior art, and the habitual signal characteristics of an individual may be mistakenly analyzed as an erroneous control signal under the specific performance of some individuals, resulting in a large difference in the accuracy of the brain-computer interface interaction results when used by different individuals.

[0006] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: a multimodal interaction method based on metaverse and brain-computer interface, comprising the following steps:

[0007] Step 100, construct a convolutional neural network model, obtain a real-time grayscale image of the metaverse, input the real-time grayscale image into the convolutional neural network model, and output the image recognition result;

[0008] Step 200, establishing a multimodal signal system, and acquiring multimodal signal data based on a brain-computer interface under the display of image recognition results;

[0009] Step 300, all signal data of different modes are divided into batches for feature extraction and feature classification, different layers are selected for feature fusion based on different batch data processing processes, and recognition and classification results are obtained;

[0010] Step 400, converting the recognition and classification results into control commands according to the generation conditions of the control commands set by the algorithm;

[0011] Step 500, the Metaverse image manipulation module receives the control command and changes the state of the virtual character and the real-time grayscale image of the Metaverse according to the control command;

[0012] In one of the batches, the signal data of each mode are subjected to feature extraction, feature classification, and decision-making in turn to obtain preliminary decision results. The signal data of several individuals are obtained and compared horizontally to identify individualized signal features that are different from other individuals. The weights of the individualized signal features are adjusted, and then the preliminary decision results are substituted according to the weighted summation fusion rule to calculate the recognition and classification results.

[0013] Further,

[0014] The process of determining the individualized signal characteristics specifically includes the following methods:

[0015] Acquire signal data of several individuals and compare them horizontally to identify individualized signal characteristics that are different from other individuals and the expression mechanism of individualized signal characteristics;

[0016] Conduct staged sampling studies on individuals to obtain the manifestations of individualized signal characteristics at different stages, and analyze the uncorrelated manifestation mechanism of individualized signal characteristics through longitudinal comparison;

[0017] Among them, all possible mechanisms of the individualized signal characteristics are composed of the manifestation mechanism of the individualized signal characteristics and the unrelated manifestation mechanism.

[0018] Further,

[0019] The weight adjustment process based on at least one individualized signal feature specifically includes the following methods:

[0020] Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism.

[0021] Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted to 0;

[0022] Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights.

[0023] Further,

[0024] The weight adjustment process based on at least one individualized signal feature specifically includes the following methods:

[0025] Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism.

[0026] Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted from a% to na%;

[0027] Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights;

[0028] Here, n is greater than 0 and less than 1.

[0029] Further,

[0030] Get several individual samples as test sets;

[0031] Initially set an n value, perform feature extraction and feature classification on the signal data of multiple individuals in the test set in batches, and select different layers for feature fusion based on the processing of different batches of data to obtain simulated classification results;

[0032] Compare the simulated classification results with the actual identification and classification results corresponding to the individual, and calculate the coincidence rate;

[0033] Adjust the n value and repeat the above steps to calculate the coincidence rate between the simulated classification results and the actual recognition classification results obtained under different n values;

[0034] The n value corresponding to the simulation classification result with the highest coincidence rate is selected and used as the proportion standard for weight adjustment of individualized signal features;

[0035] The data in the sample contains signal data of each mode and the actual recognition and classification results adapted thereto.

[0036] Further,

[0037] The process of obtaining preliminary decision results from signal data of each mode through different batches includes the following methods:

[0038] The signal data of each mode are integrated at the data layer, and then feature extraction and feature classification are performed to output the preliminary decision results;

[0039] Extract features from the signal data of each mode, perform data fusion at the feature layer, and then perform feature classification to output preliminary decision results;

[0040] The signal data of each mode is subjected to feature extraction and feature classification decision in turn to obtain a preliminary decision result. At the decision layer, the preliminary decision result is substituted into the weighted sum fusion rule to calculate the recognition and classification result, and then output.

[0041] Further,

[0042] The process of processing the real-time grayscale image of the Metaverse using a convolutional neural network specifically includes the following steps:

[0043] Perform image preprocessing on the real-time grayscale image at the input layer, and use each pixel data of the processed image as the input of the convolutional neural network;

[0044] In the first convolutional layer, the first feature map is obtained by extracting the data of the input layer and the convolution kernel, and the output value is pooled;

[0045] In the first pooling layer, the feature map is received and feature selection and information filtering are performed, and the second feature map is output;

[0046] In the second convolution layer, the second feature map is received and the third feature map is output through the convolution kernel;

[0047] In the second pooling layer, the third feature map is received, feature selection and information filtering are performed, and a fourth feature map is output;

[0048] In the fully connected layer, the data processed by the second pooling layer is received, and the data is connected to the output layer through the fully connected layer neural network;

[0049] The output layer outputs a multidimensional vector.

[0050] Further,

[0051] Setting generation conditions of different control commands based on different recognition and classification results;

[0052] The main control module pre-sets control commands corresponding to the recognition and classification results, and the main control module receives the recognition and classification results and sends control commands to the metaverse image manipulation module.

[0053] Further,

[0054] The multimodal signal system includes electroencephalogram signal data, magnetoencephalogram signal data, electromyography signal data, electrocardiogram signal data, eye movement signal data, and posture signal data;

[0055] The EEG signal data is acquired by an electroencephalogram, the MEG signal data is acquired by a magnetoencephalogram, the EMG signal data is acquired by an electromyogram, the ECG signal data is acquired by an electrocardiogram, and the eye movement signal data and the posture signal data are acquired by a brain ring device.

[0056] In order to solve the above technical problems, the present invention further provides the following technical solutions: a system using a multimodal interaction method based on a metaverse and a brain-computer interface, comprising:

[0057] Brain-computer interface to acquire multimodal signal data;

[0058] A data processing module is connected to the brain-computer interface for communication. The data processing module extracts and classifies the signal data of different modes in batches, selects different layers for feature fusion based on different batches of data processing, and obtains recognition and classification results.

[0059] The main control module receives the recognition and classification results and converts them into control commands based on the generation conditions of the control commands set by the algorithm;

[0060] The metaverse image manipulation module receives control commands issued by the main control module and changes the state of the virtual character and the real-time grayscale image of the metaverse according to the control commands.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] In the present invention, signal data of different modes are processed in batches. In one batch, feature extraction, feature classification and decision are performed on the signal data of each mode in turn to obtain a preliminary decision result. Signal data of several individuals are obtained and compared horizontally to identify individualized signal features different from other individuals. The weights of the individualized signal features are adjusted, and then the preliminary decision results are substituted according to the weighted summation fusion rule to calculate the identification and classification results. The influence of the individualized signal features on the signal processing process is reduced or directly eliminated by reducing their weights, which takes individual differences into account and improves the accuracy of individual motivation analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0064] Figure 1 A flowchart of a multimodal interaction method based on metaverse and brain-computer interface provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] like Figure 1 As shown, the present invention provides a multimodal interaction method based on metaverse and brain-computer interface, comprising the following steps:

[0067] Step 100, construct a convolutional neural network model, obtain a real-time grayscale image of the metaverse, input the real-time grayscale image into the convolutional neural network model, and output the image recognition result;

[0068] Step 200, establishing a multimodal signal system, and acquiring multimodal signal data based on a brain-computer interface under the display of image recognition results;

[0069] Step 300, all signal data of different modes are divided into batches for feature extraction and feature classification, different layers are selected for feature fusion based on different batch data processing processes, and recognition and classification results are obtained;

[0070] Step 400, converting the recognition and classification results into control commands according to the generation conditions of the control commands set by the algorithm;

[0071] Step 500, the Metaverse image manipulation module receives the control command and changes the state of the virtual character and the real-time grayscale image of the Metaverse according to the control command;

[0072] In one of the batches, the signal data of each mode are subjected to feature extraction, feature classification, and decision-making in turn to obtain preliminary decision results. The signal data of several individuals are obtained and compared horizontally to identify individualized signal features that are different from other individuals. The weights of the individualized signal features are adjusted, and then the preliminary decision results are substituted according to the weighted summation fusion rule to calculate the recognition and classification results.

[0073] In the present invention, signal data of different modes are processed in batches. In one batch, feature extraction, feature classification and decision are performed on the signal data of each mode in turn to obtain a preliminary decision result. Signal data of several individuals are obtained and compared horizontally to identify individualized signal features different from other individuals. The weights of the individualized signal features are adjusted, and then the preliminary decision results are substituted according to the weighted summation fusion rule to calculate the identification and classification results. The influence of the individualized signal features on the signal processing process is reduced or directly eliminated by reducing their weights, which takes individual differences into account and improves the accuracy of individual motivation analysis.

[0074] In the present invention, the process of obtaining preliminary decision results from signal data of each modality through different batches includes the following methods:

[0075] The signal data of each mode are integrated at the data layer, and then feature extraction and feature classification are performed to output the preliminary decision results;

[0076] Extract features from the signal data of each mode, perform data fusion at the feature layer, and then perform feature classification to output preliminary decision results;

[0077] The signal data of each mode is subjected to feature extraction and feature classification decision in turn to obtain a preliminary decision result. At the decision layer, the preliminary decision result is substituted into the weighted sum fusion rule to calculate the recognition and classification result, and then output.

[0078] In general, the process of obtaining preliminary decision results from signal data of each modality through different batches includes data layer fusion, feature layer fusion, and decision layer fusion. The data layer fusion is to fuse the signal data of each modality at the data layer, and then perform feature extraction and feature classification to output preliminary decision results; the feature layer fusion is to extract features from the signal data of each modality, perform data fusion at the feature layer, and then perform feature classification to output preliminary decision results; the decision layer fusion is to perform feature extraction and feature classification decisions on the signal data of each modality in turn to obtain preliminary decision results, and substitute the preliminary decision results into the decision layer according to the weighted sum fusion rule to calculate the recognition and classification results, and output them.

[0079] In the data layer fusion, feature layer fusion, and decision layer fusion, the weighted sum fusion rule is used to substitute the data and calculate the results. Therefore, the present invention can adopt an implementation method in which each batch performs individualized signal feature screening and one batch performs individualized signal feature screening.

[0080] That is, one embodiment is: screening the individualized signal features only in the data processing batch of the decision layer fusion, and adjusting the weights of the individualized signal features in the process of feature fusion;

[0081] Another embodiment is: in the data processing batches of data layer fusion, feature layer fusion, and decision layer fusion, the individualized signal features are screened before feature fusion, and the weights of the individualized signal features are adjusted during the feature fusion process.

[0082] In the present invention, the determination process of individualized signal features (screening process of individualized signal features) specifically includes the following methods:

[0083] Acquire signal data of several individuals and compare them horizontally to identify individualized signal characteristics that are different from other individuals and the expression mechanism of individualized signal characteristics;

[0084] Conduct staged sampling studies on individuals to obtain the manifestations of individualized signal characteristics at different stages, and analyze the uncorrelated manifestation mechanism of individualized signal characteristics through longitudinal comparison;

[0085] Among them, all possible mechanisms of individualized signal characteristics are composed of the manifestation mechanism of individualized signal characteristics and the unrelated manifestation mechanism.

[0086] In the above embodiment, firstly, through horizontal comparison between different individuals, individualized signal features different from other individuals are identified through horizontal comparison. For example, if an individual's head habitually turns to the right by 15°, the head rotation action is defined as the individualized signal feature of the individual, and the direction and angle of the individual's habitual head rotation are defined as the expression mechanism of the individual's individualized signal feature.

[0087] Secondly, through the staged sampling study of individuals, we can obtain the manifestation forms of individualized signal characteristics at different stages, and analyze the unrelated manifestation mechanism of individualized signal characteristics through longitudinal comparison. For example, the individual's head is accustomed to turning to the right by 15°, 13°, etc. without consciousness. The overall analysis shows that the unrelated manifestation mechanism of the individualized signal characteristics is that the individual's head turns to the right by more than 15° or to the left. In other words, when the individual's head turns to the right by more than 15° or to the left, it is a conscious control action of the individual.

[0088] In actual applications, individualized signal features are generally reflected in posture signals and eye movement signals.

[0089] After the individualized signal feature is screened out, the weight of the feature needs to be adjusted. The weight adjustment method includes the following two implementation methods:

[0090] The first implementation method is:

[0091] Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism.

[0092] Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted to 0;

[0093] Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights.

[0094] Taking the individual turning his head to the left as an example, this feature belongs to the unrelated expression mechanism of the individualized signal feature, so the weight of the individualized signal feature remains unchanged. Taking the individual turning 10° to the right as an example, this feature belongs to the expression mechanism of the individualized signal feature. The weight of the individualized signal feature is adjusted to 0, and the influence of the individualized signal feature under this expression mechanism on the final analysis result is directly eliminated.

[0095] The second implementation method is:

[0096] Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism.

[0097] Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted from a% to na%;

[0098] Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights;

[0099] Here, n is greater than 0 and less than 1.

[0100] Taking the individual turning his head to the left as an example, this feature belongs to the uncorrelated expression mechanism of the individualized signal feature, so the weight of the individualized signal feature remains unchanged. Taking the individual turning 10° to the right as an example, this feature belongs to the expression mechanism of the individualized signal feature, and the weight of the individualized signal feature is adjusted from a% to na%, thereby reducing the proportion of the individualized signal feature in signal fusion, thereby reducing the impact on the data processing results.

[0101] In order to calculate the best n value, the present invention also discloses the following method for calculating the optimized n value:

[0102] Get several individual samples as test sets;

[0103] Initially set an n value, perform feature extraction and feature classification on the signal data of multiple individuals in the test set in batches, and select different layers for feature fusion based on the processing of different batches of data to obtain simulated classification results;

[0104] Compare the simulated classification results with the actual identification and classification results corresponding to the individual, and calculate the coincidence rate;

[0105] Adjust the n value and repeat the above steps to calculate the coincidence rate between the simulated classification results and the actual recognition classification results obtained under different n values;

[0106] The n value corresponding to the simulation classification result with the highest coincidence rate is selected and used as the proportion standard for weight adjustment of individualized signal features;

[0107] The data in the sample contains the signal data of each mode and the actual recognition and classification results adapted thereto.

[0108] In the above process, a difference, such as b, can be removed between adjacent n values. Then, after an n value is initially set (assuming the initial n value is 0.99), the simulated classification result is obtained by feature fusion according to the weights adjusted under the n value, which is compared with the actual recognition and classification result, and the overlap rate is calculated. Then, the n value is adjusted to nb, and the simulated classification result is obtained by feature fusion according to the weights adjusted under the nb value, which is compared with the actual recognition and classification result, and the overlap rate is calculated. This process is repeated until the n value is adjusted to 0. The overlap rate between the simulated classification results and the actual recognition and classification results obtained under different n values ​​is then compared, and the n value corresponding to the simulated classification result with the highest overlap rate is screened out, which is used as the proportional standard for adjusting the weights of individualized signal features, and the weights of individualized signal features are adjusted according to the n value.

[0109] The above-mentioned n value calculation process is based on the process of adjusting the weights for each batch. That is to say, after adjusting the n value, in the data processing batches of data layer fusion, feature layer fusion, and decision layer fusion, the weights of individualized signal features are adjusted according to the n value during the feature fusion process, and finally the overall simulated classification result is obtained.

[0110] The present invention also uses a convolutional neural network to process the real-time grayscale image of the metaverse, which specifically includes the following steps:

[0111] Perform image preprocessing on the real-time grayscale image at the input layer, and use each pixel data of the processed image as the input of the convolutional neural network;

[0112] In the first convolutional layer, the first feature map is obtained by extracting the data of the input layer and the convolution kernel, and the output value is pooled;

[0113] In the first pooling layer, the feature map is received and feature selection and information filtering are performed, and the second feature map is output;

[0114] In the second convolution layer, the second feature map is received and the third feature map is output through the convolution kernel;

[0115] In the second pooling layer, the third feature map is received, feature selection and information filtering are performed, and a fourth feature map is output;

[0116] In the fully connected layer, the data processed by the second pooling layer is received, and the data is connected to the output layer through the fully connected layer neural network;

[0117] The output layer outputs a multidimensional vector.

[0118] In the above process, the first convolutional layer and the first pooling layer, as well as the second convolutional layer and the second pooling layer, undergo a downsampling process. In the downsampling layer, each neuron is connected to the neuron in the previous convolutional layer through a local connection.

[0119] The second pooling layer and the convolutional layer are not fully connected.

[0120] The output layer is composed of Euclidean radial basis functions. The output layer outputs a multidimensional vector, and the corresponding output bits can correspond to parameters such as environment and position.

[0121] In the present invention, different conditions for generating different control commands are set based on different recognition and classification results, and the control commands corresponding to the recognition and classification results are pre-set by the main control module. The main control module receives the recognition and classification results and sends control commands to the metaverse image manipulation module.

[0122] Assuming that the recognition and classification result is a 30° rotation to the right, the main control module pre-sets a control command corresponding to the recognition and classification result. The main control module receives the recognition and classification result and sends a control command to the metaverse image manipulation module. The metaverse image manipulation module adjusts the real-time grayscale image according to the control command.

[0123] The adjusted real-time grayscale image is outputted through the convolutional neural network model to obtain an image recognition result and is displayed to the individual. The individual receives the displayed image recognition result and takes the next action, and repeats steps 300, 400, and 500 according to the action.

[0124] In the present invention, the multimodal signal system includes EEG signal data, MEG signal data, EMG signal data, ECG signal data, eye movement signal data, and posture signal data;

[0125] The EEG signal data is obtained by an electroencephalogram, the MEG signal data is obtained by a magnetoencephalogram, the EMG signal data is obtained by an electromyogram, the ECG signal data is obtained by an electrocardiogram, the eye movement signal data and the posture signal data are obtained by a brain loop device, wherein the posture signal data can be collected and obtained by a gyroscope arranged in the brain loop device.

[0126] The present invention also provides a multimodal interaction system based on the metaverse and brain-computer interface, which has:

[0127] Brain-computer interface to acquire multimodal signal data;

[0128] The data processing module is connected to the brain-computer interface communication. The data processing module extracts and classifies the signal data of different modes in batches, selects different layers for feature fusion based on the data processing process of different batches, and obtains the recognition and classification results;

[0129] The main control module receives the recognition and classification results and converts them into control commands based on the generation conditions of the control commands set by the algorithm;

[0130] The metaverse image manipulation module receives control commands issued by the main control module and changes the state of the virtual character and the real-time grayscale image of the metaverse according to the control commands.

[0131] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present application. The protection scope of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present application.

Claims

1. A multimodal interaction method based on metaverse and brain-computer interface, characterized in that: The following steps are involved: Step 100, construct a convolutional neural network model, obtain a real-time grayscale image of the metaverse, input the real-time grayscale image into the convolutional neural network model, and output the image recognition result; Step 200, establishing a multimodal signal system, and acquiring multimodal signal data based on a brain-computer interface under the display of image recognition results; Step 300, all signal data of different modes are divided into batches for feature extraction and feature classification, different layers are selected for feature fusion based on different batch data processing processes, and recognition and classification results are obtained; Step 400, converting the recognition and classification results into control commands according to the generation conditions of the control commands set by the algorithm; Step 500, the Metaverse image manipulation module receives the control command and changes the state of the virtual character and the real-time grayscale image of the Metaverse according to the control command; In one of the batches, the signal data of each mode is subjected to feature extraction, feature classification, and decision-making in turn to obtain a preliminary decision result. The signal data of several individuals are obtained and compared horizontally to identify the individualized signal features that are different from other individuals. The weights of the individualized signal features are adjusted, and then the preliminary decision results are substituted according to the weighted summation fusion rule to calculate the identification and classification results. The process of determining the individualized signal characteristics specifically includes the following methods: Acquire signal data of several individuals and compare them horizontally to identify individualized signal characteristics that are different from other individuals and the expression mechanism of individualized signal characteristics; Conduct staged sampling studies on individuals to obtain the manifestations of individualized signal characteristics at different stages, and analyze the uncorrelated manifestation mechanism of individualized signal characteristics through longitudinal comparison; Wherein, all possible mechanisms of the individualized signal characteristics consist of the manifestation mechanism of the individualized signal characteristics and the unrelated manifestation mechanism; The weight adjustment process based on at least one individualized signal feature specifically includes the following methods: Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism; Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted to 0; Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights; The weight adjustment process based on at least one individualized signal feature specifically includes the following methods: Determine whether the characteristic value of the individualized signal feature belongs to the expression mechanism of the individualized signal feature or the unrelated expression mechanism; Based on the performance mechanism of individualized signal features, the weight of individualized signal features is adjusted from a% to na%; Based on the uncorrelated representation mechanism of individualized signal features, the weights of individualized signal features maintain the values ​​of the initial weights; Here, n is greater than 0 and less than 1.

2. The multimodal interaction method based on metaverse and brain-computer interface according to claim 1, characterized in that: Get several individual samples as test sets; Initially set an n value, perform feature extraction and feature classification on the signal data of multiple individuals in the test set in batches, and select different layers for feature fusion based on the processing of different batches of data to obtain simulated classification results; Compare the simulated classification results with the actual identification and classification results corresponding to the individual, and calculate the coincidence rate; Adjust the n value and repeat the above steps to calculate the coincidence rate between the simulated classification results and the actual recognition classification results obtained under different n values; The n value corresponding to the simulation classification result with the highest coincidence rate is selected and used as the proportion standard for weight adjustment of individualized signal features; The data in the sample contains signal data of each mode and the actual recognition and classification results adapted thereto.

3. The multimodal interaction method based on metaverse and brain-computer interface according to claim 1, characterized in that: The process of obtaining preliminary decision results from signal data of each mode through different batches includes the following methods: The signal data of each mode are integrated at the data layer, and then feature extraction and feature classification are performed to output the preliminary decision results; Extract features from the signal data of each mode, perform data fusion at the feature layer, and then perform feature classification to output preliminary decision results; The signal data of each mode is subjected to feature extraction and feature classification decision in turn to obtain a preliminary decision result. At the decision layer, the preliminary decision result is substituted into the weighted sum fusion rule to calculate the recognition and classification result, and then output.

4. The multimodal interaction method based on metaverse and brain-computer interface according to claim 1, characterized in that: The process of processing the real-time grayscale image of the Metaverse using a convolutional neural network specifically includes the following steps: Perform image preprocessing on the real-time grayscale image at the input layer, and use each pixel data of the processed image as the input of the convolutional neural network; In the first convolutional layer, the first feature map is obtained by extracting the data of the input layer and the convolution kernel, and the output value is pooled; In the first pooling layer, the feature map is received and feature selection and information filtering are performed, and the second feature map is output; In the second convolution layer, the second feature map is received and the third feature map is output through the convolution kernel; In the second pooling layer, the third feature map is received, feature selection and information filtering are performed, and a fourth feature map is output; In the fully connected layer, the data processed by the second pooling layer is received, and the data is connected to the output layer through the fully connected layer neural network; The output layer outputs a multidimensional vector.

5. The multimodal interaction method based on metaverse and brain-computer interface according to claim 1, characterized in that: Setting generation conditions of different control commands based on different recognition and classification results; The main control module pre-sets control commands corresponding to the recognition and classification results, and the main control module receives the recognition and classification results and sends control commands to the metaverse image manipulation module.

6. The multimodal interaction method based on metaverse and brain-computer interface according to claim 1, characterized in that: The multimodal signal system includes electroencephalogram signal data, magnetoencephalogram signal data, electromyography signal data, electrocardiogram signal data, eye movement signal data, and posture signal data; The EEG signal data is acquired by an electroencephalogram, the MEG signal data is acquired by a magnetoencephalogram, the EMG signal data is acquired by an electromyogram, the ECG signal data is acquired by an electrocardiogram, and the eye movement signal data and the posture signal data are acquired by a brain ring device.

7. A system using the multimodal interaction method based on metaverse and brain-computer interface according to any one of claims 1 to 6, characterized in that: have: Brain-computer interface to acquire multimodal signal data; A data processing module is connected to the brain-computer interface for communication. The data processing module extracts and classifies the signal data of different modes in batches, selects different layers for feature fusion based on different batches of data processing, and obtains recognition and classification results. The main control module receives the recognition and classification results and converts them into control commands based on the generation conditions of the control commands set by the algorithm; The metaverse image manipulation module receives control commands issued by the main control module and changes the state of the virtual character and the real-time grayscale image of the metaverse according to the control commands.

Images