Visual scene reconstruction method based on brain wave data pre-trained model and storage medium

Abstract

The application discloses a visual scene reconstruction method based on a brain wave data pre-training model and a storage medium, and comprises the following steps: encoding brain wave data collected by using a pre-training encoding module; and generating an image according to the brain wave data after encoding by using a pre-training decoding module. The pre-training encoding module is used for simulating the training method of a BERT model in natural language processing, a pre-training model is trained based on brain wave data, and the pre-training model is built into the pre-training encoding module to encode the brain wave data. The pre-training decoding module is built with a pre-trained neural network model, the brain wave data after encoding is decoded, and an image causing the brain wave data is restored. The application can analyze brain wave data into user observation content, and can reflect the activity characteristics of brain waves to a certain extent. The application pre-trains a model similar to the BERT structure, can fully utilize existing brain wave data, solves the problem of small brain wave data to a certain extent, and is helpful to the accuracy of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of brain wave data processing, and particularly relates to a visual scene reconstruction method based on a brain wave data pre-training model and a storage medium. BACKGROUND

[0002] Brain waves (English: brain waves) are also called "brain waves". Brain waves are bioelectric signals generated when neurons transmit information; they are ion exchange-generated electrical wave signals produced when pyramidal cells in the cortex are active. Brain cells will generate corresponding bioelectricity as long as they are active, and these electrical signals are transmitted through dendrites to connect with other brain cells to form a network of thinking. In scientific applications, brain waves that can be detected by instruments must be generated by a certain number of neuron activities, and are classified into the following types according to frequency: delta waves, theta waves, alpha waves, beta waves, gamma waves, etc. According to different brain state management, beta waves are subdivided into Smr waves, beta1 waves (low beta waves), and beta2 waves (high beta waves).

[0003] Current research on brain waves includes brain wave state analysis and brain wave induction. With the development of technology, people have gradually explored the conversion of brain wave changes during brain activity into pictures. Such pictures can be meaningful electroencephalogram feature maps or visual stimulation pictures, that is, the content of pictures seen by the human eye. Most of the related research is focused on magnetic resonance imaging technology and brain-computer interfaces. This type of research has a high threshold, requires professional equipment, and is still in the preliminary exploration stage.

[0004] In summary, the existing technology is also in the preliminary exploration stage, and the equipment used has a high cost. Brain wave data acquisition is not convenient, and it is difficult to form a large-scale data set, which will reduce the effect of the deep learning model. SUMMARY

[0005] The visual scene reconstruction method based on a brain wave data pre-training model can solve at least one of the above technical problems.

[0006] To achieve the above purpose, the present application adopts the following technical scheme:

[0007] A visual scene reconstruction method based on a brain wave data pre-training model, comprising the following steps,

[0008] S1, encoding the collected brain wave data using a pre-training encoding module;

[0009] S2, generating an image according to the encoded brain wave data using a pre-training decoding module;

[0010] Among them,

[0011] The pre-training coding module is used to simulate the training method of the BERT model in natural language processing, and a pre-training model is trained based on brain wave data and built into the pre-training coding module to code the brain wave data.

[0012] The pre-training decoding module has a pre-trained neural network model built in to decode the coded brain wave data and restore the image causing the brain wave data.

[0013] Further, the brain wave data collected in S1 is coded by using the pre-training coding module, and the specific steps are as follows:

[0014] The training method of the BERT model in natural language processing is simulated, and the training task is improved, as follows:

[0015] S11, brain wave data symbolization:

[0016] First, according to the data collected by the collection device, represented as X(x1, x2, …x n ), the maximum value V max and the minimum value V min of each time step of the data are obtained, the interval [V min , V max ] is uniformly divided into N boxes, and 1-N is used respectively; for the value x i in the data, if V min <=x i <=V min +(V max -V min ) / N, then x i is recorded as 1; if V min +(V max -V min ) / N<=x i <=V min +2*(V max -V min ) / N, then x i is recorded as 2; and so on, X is coded into a sequence represented by 1-N;

[0017] S12, Embedding process of symbolized data:

[0018] After the brain wave data is processed into a symbol sequence, the Embedding method of Bert is simulated; wherein:

[0019] Token Embeddings is a word vector, the first word is a CLS flag representing the data boundary, used for subsequent classification tasks;

[0020] Segment Embeddings are used to distinguish input and output in sequence prediction, which refers to predicting the data of the next L2 time steps using the data of the previous L1 time steps; it is analogous to sentence prediction in BERT;

[0021] Position Embeddings represent position encodings that can be learned; their related parameters can be learned in subsequent training tasks;

[0022] S13, using the network structure of BERT to train the improved task.

[0023] Further, the S13, using the network structure of BERT to train the improved task, directly uses the network structure of BERT without modification, and performs the following two training tasks:

[0024] S131, T1: Masked language model

[0025] This step is consistent with the original BERT training; during the training process, 15% of the tokens are randomly masked, and the final loss function only calculates the masked token;

[0026] S132, T2: Sentence prediction

[0027] In this step, sequence prediction is performed, using the data of the first L1 time steps in the brain wave data to predict the data of the last L2 time steps.

[0028] Further, the S2, using the pre-trained decoding module to generate images according to the encoded brain wave data, specifically includes:

[0029] The user observes the image, wears the brain wave acquisition device, observes the image for a certain period of time, assuming L seconds; records the brain wave data and the corresponding image.

[0030] Further, the pre-trained neural network model built-in the pre-training decoding module includes:

[0031] Transformation matrix: a set of trainable parameter matrices, which can convert the encoded brain wave data into an N*N matrix, and the N*N matrix represents the state transition graph that may exist in the brain wave data; the N*N matrix can be used as a matrix representation of the graph structure, and N can be determined through debugging experiments, and the best one can be selected;

[0032] Graph convolution layer: a neural network structure for convolution of graph structure data;

[0033] Deconvolution block: the deconvolution block is used to convert the output of the GCN layer into image content consistent with the channel number of the image observed by the user;

[0034] Convolution Block: The two Convolution Blocks in the structure are the same, used for extracting image features;

[0035] Flatten layer: The two Flatten layers in the structure are also the same, which are used for converting image features into one-dimensional vectors for facilitating the calculation of the loss function;

[0036] Loss function: The mean absolute error is used as the loss function, and the mean absolute error of the two vectors output by the Flatten layer is directly calculated.

[0037] Further, the pre-training decoding module has a pre-trained neural network model built-in, and the training process is as follows:

[0038] The brain wave pre-encoding data is transformed into a graph by a transformation matrix, and a graph convolution operation is performed on the graph matrix, and the output result is output through a deconvolution Block to output an image consistent with the channel number of the user's observed image; the generated image and the user's observed image are respectively subjected to feature extraction by the Convolution Block, and after being converted into one-dimensional vectors by the Flatten layer, the loss function is calculated, the loss function is back propagated, and the model parameters are updated;

[0039] After the model training is completed, the parameters of the transformation matrix, the graph convolution layer and the deconvolution Block are built into the pre-training decoding module; when the pre-training decoding module receives the brain wave data processed by the pre-training encoding module, the corresponding image can be generated.

[0040] On the other hand, the computer readable storage medium of the present application stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the above method.

[0041] From the above technical solution, the visual scene reconstruction method based on the brain wave data pre-training model of the present application has the following beneficial effects:

[0042] 1. The present application provides a method for analyzing brain wave data into user observation content, which can reflect the activity characteristics of brain waves to a certain extent;

[0043] 2. The present application pre-trains a model similar to BERT structure, which can fully utilize the existing brain wave data and to a certain extent solve the problem of small amount of brain wave data, and has a certain help for the accuracy of the model;

[0044] 3. The present application constructs a decoding model based on a graph convolutional neural network, which can capture the characteristics of state changes in brain wave data and has a certain help for the accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 is a schematic diagram of the method of the present application;

[0046] Figure 2 is a schematic diagram of the Embedding process of the symbolic data of the embodiment of the present application;

[0047] Figure 3 is a schematic diagram of the pre-trained neural network model structure of the present application. DETAILED DESCRIPTION

[0048] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments of the present application.

[0049] As shown in Figure 1 the visual scene reconstruction method based on the brain wave data pre-trained model described in the present embodiment includes a pre-training encoding module and a pre-training decoding module, and the main functions and implementation methods are as follows:

[0050] Pre-training encoding module: simulate the training method of the BERT (Bidirectional Encoder Representation from Transformers) model in natural language processing, train a pre-trained model similar to BERT based on brain wave data, and embed the pre-training encoding module in the present application to encode the brain wave data.

[0051] Pre-training decoding module: this module embeds a pre-trained neural network model to decode the brain wave data after encoding and restore the image causing the brain wave data.

[0052] The workflow and specific implementation are as follows:

[0053] S1, the pre-training encoding module encodes the collected brain wave data

[0054] During operation, brain wave collection can be collected by wearing existing conventional brain wave collection equipment on the market, which is generally collected for L seconds, and the specific duration is determined according to practical verification. When encoding, only the collected brain wave data needs to be input into the pre-training encoding model embedded in the module, and the output result of the model is the data after encoding.

[0055] The specific training process of the pre-training encoding model used is mainly to simulate the training method of the BERT model in natural language processing, and to improve the training task, which is as follows:

[0056] (1) Brain wave data symbolization:

[0057] First, according to the data collected by the collection device, represented as X(x1, x2, … x n ), the maximum value V max and the minimum value V min of each time step of the data are obtained, and the interval [V min , V max ] is uniformly divided into N boxes, represented by 1-N respectively. For the value x i in the data, if V min <= x i <= V min +(V max -V min ) / N, then x i is recorded as 1; if V min +(V max -V min ) / N <= x i <= V min +2*(V max -V min ) / N, then x i is recorded as 2; and so on, X is encoded into a sequence represented by 1-N.

[0058] (2) The Embedding process of the symbolic data is shown in the following figure: Figure 2

[0059] After processing the brain wave data into a symbolic sequence, the Embedding method of BERT is simulated. Among them:

[0060] Token Embeddings is a word vector, the first word is a CLS mark, indicating the boundary of the data, which can be used for subsequent classification tasks.

[0061] Segment Embeddings is used to distinguish input and output in sequence prediction, which refers to using the data of the previous L1 time steps to predict the data of the subsequent L2 time steps. It is analogous to sentence prediction in BERT.

[0062] Position Embeddings represents the position encoding that can be learned. Its related parameters can be learned in subsequent training tasks.

[0063] (1) Use the network structure of BERT to improve the training of the task.

[0064] Here, the network structure of BERT is directly used without modification. The following two training tasks are performed:

[0065] 1) T1: Masked language model ​

[0066] This step is consistent with the original BERT training. During training, 15% of the tokens are randomly masked, and the final loss function only calculates the loss of the masked token.

[0067] 2) T2: Sentence prediction

[0068] This step uses a common sequence prediction task instead because there is no so-called sentence in the brain wave data. That is, sequence prediction is performed in this step, using the first L1 time steps of data in the brain wave data to predict the data of the last L2 time steps.

[0069] S2, the pre-trained decoding module generates an image according to the encoded brain wave data.

[0070] The key point of this module is the construction of the decoding model. Here, the graph neural network and the deconvolution operation are combined to generate the image. At the same time, a common image feature extraction structure is maintained to simplify the calculation of the loss function.

[0071] Here, the method for obtaining training data is introduced: the user observes the image, wears the brain wave acquisition device, observes the image for a certain time, and assumes L seconds. Record the brain wave data and the corresponding image.

[0072] The specific network structure is shown in the dashed box in Figure 3 :

[0073] The functions of the above modules are as follows:

[0074] (1) Transformation matrix: a set of trainable parameter matrices that can convert the encoded brain wave data into an N*N matrix. This N*N matrix represents the state transition graph that may exist in the brain wave data. The N*N matrix can be used as a matrix representation of the graph structure. N can be determined through debugging experiments, and the best one can be selected.

[0075] (2) Graph convolution (GCN, Graph Convolutional Networks) layer: a neural network structure for convolution on graph structure data. The basic idea of GCN is to obtain the feature information of each node from all its neighbor nodes, including its own feature. Suppose an aggregation function is used. The same operation is performed on all nodes. Finally, the average values calculated are input into the neural network.

[0076] (3) Deconvolution block: the deconvolution block is used to convert the output of the GCN layer into an image content consistent with the channel number of the user observation image. Theoretically, its output is the required image.

[0077] (4) Convolutional Block: The two convolutional blocks in the structure are the same, and the main function is to extract image features. The implementation can be to embed the existing pre-trained convolutional structure into the network, or to retrain in the entire network structure.

[0078] (5) Flatten layer: The two Flatten layers in the structure are also the same, which mainly converts image features into one-dimensional vectors to facilitate the calculation of the loss function.

[0079] (6) Loss function: The mean absolute error is used as the loss function here, and the mean absolute error of the two vectors output by the Flatten layer is directly calculated.

[0080] The specific training process is: the brain wave pre-encoding data is transformed into a graph by a transformation matrix, and a graph convolution operation is performed on the graph matrix, and the output result is output through the deconvolutional block to output an image consistent with the channel number of the user's observed image. The convolutional block extracts features from the generated image and the user's observed image, and after being converted into a one-dimensional vector through the Flatten layer, the loss function is calculated, the loss function is back propagated, and the model parameters are updated.

[0081] After the model training is completed, only the parameters of the transformation matrix, the graph convolution layer and the deconvolutional block need to be embedded into the pre-trained decoding module. When the pre-trained decoding module receives the brain wave data processed by the pre-trained encoding module, the corresponding image can be generated.

[0082] The implementation of the convolutional block of the embodiment of the application: the existing pre-trained model can be fine-tuned, or the entire decoding model can be trained.

[0083] In summary, the application provides a method for parsing brain wave data into user observation content, which can reflect the activity characteristics of brain waves to a certain extent. The application pre-trains a model similar to the BERT structure, which can fully utilize the existing brain wave data, to a certain extent, solve the problem of small amount of brain wave data, and help the accuracy of the model. The decoding model based on the graph convolutional neural network can capture the characteristics of the state change in the brain wave data, which helps to improve the accuracy of the model.

[0084] In another aspect, the application also discloses a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to make the processor execute the steps of any of the above methods.

[0085] In still another aspect, the present application also discloses a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to perform the steps of any of the above methods.

[0086] In still another aspect, the present application also discloses a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to perform the steps of any of the above methods.

[0087] It is to be understood that the system provided by the embodiments of the present application corresponds to the method provided by the embodiments of the present application, and the explanation, examples and advantages of related contents can be referred to the corresponding part of the above method.

[0088] It is to be understood that the system provided by the embodiments of the present application corresponds to the method provided by the embodiments of the present application, and the explanation, examples and advantages of related contents can be referred to the corresponding part of the above method.

[0089] The technical features of the above embodiments can be combined in any manner. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictions, it should be considered that the combinations are within the scope of the present application.

[0090] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for reconstructing a visual scene based on a pre-trained model of brain wave data, characterized in that, The method comprises the following steps of: S1, encoding the collected brain wave data by using a pre-trained encoding module; S2, generating an image according to the encoded brain wave data by using a pre-trained decoding module; Wherein, The pre-trained encoding module is used to simulate the training method of the BERT model in natural language processing, train a pre-trained model based on brain wave data, and is built into the pre-trained encoding module to encode brain wave data; The pre-trained decoding module is built-in pre-trained neural network model, decodes the encoded brain wave data, and restores the image causing the brain wave data; The encoding of the collected brain wave data by using the pre-trained encoding module in S1 comprises the following steps: The training method of the BERT model in natural language processing is simulated, and the training task is improved, as follows: S11, brain wave data symbolization: First, according to the data collected by the collection device, represented as X (x1, x2, … x n ), the maximum value V max and the minimum value V min of each time step of the data are obtained, and the interval [V min , V max ] is uniformly divided into N boxes, represented by 1-N respectively; for the value x i in the data, if V min <=x i <=V min +(V max -V min ) / N, then x i is recorded as 1; if V min +(V max -V min ) / N<=x i <=V min +2*(V max -V min ) / N, then x i is recorded as 2; and so on, X is encoded as a sequence represented by 1-N; S12, Embedding process of symbolized data: After the brain wave data is processed into a symbol sequence, the Embedding mode of Bert is simulated; wherein: Token Embeddings is a word vector, the first word is a CLS mark, indicating the data boundary, and is used for subsequent classification tasks; Segment Embeddings is used to distinguish input and output in sequence prediction, which refers to predicting data in L2 time steps using data in L1 time steps; It is analogous to sentence prediction in BERT; Position Embeddings represents position encoding that can be learned; Its related parameters can be learned in subsequent training tasks; S13, training of improved task using BERT network structure. 2.The visual scene reconstruction method based on a brainwave data pre-trained model according to claim 1, characterized in that: The S13, training of improved task using BERT network structure directly uses the network structure of BERT without changing, and performs the following two training tasks: S131, T1: Masked language model This step is consistent with the original BERT training; during the training process, 15% of the tokens are randomly masked, and the final loss function only calculates the masked token; S132, T2: sentence prediction In this step, sequence prediction is performed, and data in L2 time steps is predicted using data in L1 time steps in brain wave data. 3.The visual scene reconstruction method based on a brainwave data pre-trained model according to claim 1, characterized in that: The S2, generating an image according to the encoded brain wave data by using a pre-trained decoding module, specifically comprises: The user observes the image, wears the brain wave collection device, observes the image for a certain time, assuming L seconds; records the brain wave data and the corresponding image. 4.The visual scene reconstruction method based on a brainwave data pre-trained model according to claim 1, characterized in that: The pre-trained decoding module built-in pre-trained neural network model comprises: Transformation matrix: a set of trainable parameter matrices, which can convert the encoded brain wave data into an N*N matrix, and the N*N matrix represents the state transition graph that may exist in the brain wave data; The N*N matrix can be regarded as using a matrix to represent a graph structure, and N can be finally determined through debugging experiments, and the best one can be selected; Graph convolution layer: a neural network structure for convolution of graph structure data; Deconvolution Block: The deconvolution block is used to convert the output of the GCN layer into image content consistent with the number of channels of the user observation image; Convolution Block: The two convolution blocks in the structure are the same, which are used to extract image features; Flatten layer: The two Flatten layers in the structure are also the same, which are used to convert image features into one-dimensional vectors for easy calculation of loss functions; Loss function: The mean absolute error is used as the loss function, which directly calculates the mean absolute error of the two vectors output by the Flatten layer.

5. The method of claim 4, wherein the method further comprises: The pre-training decoding module has a pre-trained neural network model, and the training process is as follows: The brain wave pre-encoding data is transformed into a graph by a transformation matrix, and a graph convolution operation is performed on the graph matrix. The output result is output as an image consistent with the number of channels of the user observation image through a deconvolution block. The convolution block extracts features from the generated image and the user observation image respectively, and then converts the features into one-dimensional vectors through a Flatten layer. Then, the loss function is calculated, the loss function is back propagated, and the model parameters are updated. After the model training is completed, the parameters of the transformation matrix, the graph convolution layer, and the deconvolution block are built into the pre-training decoding module. When the pre-training decoding module receives the brain wave data processed by the pre-training encoding module, the corresponding image can be generated.

6. A computer-readable storage medium storing a computer program, wherein the computer program is executed by a processor to make the processor execute the steps of the method according to any one of claims 1 to 5.

Images