Visual Signal Processing Method, System and Device Based on Hybrid Coding

Abstract

The present invention discloses a visual signal processing method, system and device based on hybrid coding. The visual signal processing method includes performing signal preprocessing on the collected electroencephalogram (EEG) signals and separating the visual cortex EEG signals; performing hybrid coding on the preprocessed signals and the user's identity information according to the set hybrid coding rules to generate signal feature codes corresponding to image stimuli; at the same time, converting the corresponding stimulus images into image feature codes; establishing a mapping relationship between the EEG feature distribution and the image feature distribution, and calculating the Euclidean distance between the clustering center vectors of the mapped EEG feature distribution and the image feature distribution to obtain a mapping error; training the feature mapping network through a backpropagation algorithm and other steps. The present invention preferably amplifies the collected signals first and then filters them, and at the same time adds component analysis and filtering of interference signals; takes into account the differences of actual users, and improves the accuracy of the output results.

Description

Technical Field

[0001] The present invention relates to the technical field of brain-computer interfaces, and more specifically, to a visual signal processing method, system, and device based on hybrid coding. Background Art

[0002] An implantable brain-machine interface (IBMI) is a cutting-edge scientific technology that allows humans to directly control external devices through the brain, thereby enabling direct communication between the brain and the machine. The visual brain-computer interface can detect the electrical signals of the brain's visual cortex by implanting electrodes in the brain. After a series of signal processing processes, it finally maps them to actual images, establishing a mapping relationship between the electrical signals of the brain's visual cortex and the visual images of the human eye. Through this mapping relationship, it is possible to reversely convert actual images into corresponding electroencephalogram (EEG) signals and "write" them into the cerebral cortex through implantable motors, enabling blind patients to see again.

[0003] The current visual signal processing system only considers the interference of electrooculogram (EOG) signals in the preprocessing of EEG signals and does not consider the interference of electromyogram (EMG) signals. Therefore, it is necessary to perform component analysis and filtering of the corresponding interference during the preprocessing process. Secondly, the prior art does not consider the differences in the EEG signal characteristics of the subjects due to differences in identity and cognition. Therefore, the classification results often need to be corrected to meet the accuracy requirements. Summary of the Invention

[0004] In view of the above-mentioned defects of the prior art, the present invention provides a visual signal processing method, system, and device based on hybrid coding, aiming to solve the problem that the EEG signals of the brain's visual cortex collected through implantable electrodes cannot be directly converted into recognizable visual images.

[0005] To achieve the above object, in a first aspect, the present invention provides a visual signal processing method based on hybrid coding, which is characterized by including the following steps:

[0006] Step 401: Implant and wear potential sampling electrodes for the subject, and collect the visual cortex EEG signals, EOG signals, and head EMG signals of the subject under the stimulation of test images in real time;

[0007] Step 402: Preprocess the collected potential signals and separate the visual cortex EEG signals;

[0008] Step 403: Use a neural network to perform hybrid encoding on the visual cortex EEG signals and the identity or status information of the subject to obtain signal feature encoding, and at the same time perform image feature encoding on the test images; establish a feature mapping network to complete the mapping between the signal feature encoding and the image feature encoding, and train the network parameters according to the loss function and the backpropagation algorithm;

[0009] Step 404: Decode the image feature encoding obtained by mapping from the signal feature encoding, and use the reconstructed image generation network to generate the corresponding image.

[0010] Further, in step 402, first amplify the small-amplitude signals collected, remove the noise caused by the environment from the amplified signals through band-pass filtering, perform principal component analysis on the filtered signals, and remove the interference of electrooculogram signals and electromyogram signals on the EEG signals to complete signal separation.

[0011] Further, in step 403, the identity or status information of the subject includes, but is not limited to, the age and gender information of the subject.

[0012] Further, in step 403, the signal feature encoding is to perform hybrid encoding on the preprocessed signals and the identity information of the subject to generate signal feature codes corresponding to the image stimuli; the loss function of the neural network for signal feature encoding is where n represents the sample size in one training, σ i,tar and σ i,out respectively represent the one-hot code (One-Hot Encoding) of the signal feature classification label of the i-th sample and the probability code of the network output result.

[0013] Further, in step 403, the loss function of the neural network for the image feature encoding is where n represents the sample size in one training, γ i,tar and respectively represent the one-hot code of the image label of the i-th sample and the network output result.

[0014] Further, in step 403, the mapping between the signal feature encoding and the image feature encoding is to input the clustering centers of the corresponding EEG feature distributions and image feature distributions into the feature mapping network to obtain the clustering center vectors of the mapped EEG feature distributions and image feature distributions, that is, the mapping relationship between the EEG feature distributions and the image feature distributions, and calculate the Euclidean distance between the clustering center vectors of the mapped EEG feature distributions and the clustering center vectors of the image feature distributions to obtain the mapping error; the mapping error can be used to train the feature mapping network through the backpropagation algorithm; the loss function of the feature mapping network is Among them, n represents the sample size in one training, ε i,img and ε i,map respectively represent the cluster center codes of the actual image features and the signal features of the i-th mapping after passing through the mapping network, and ω i is the weight of different classification labels.

[0015] In a second aspect, the present invention provides a visual signal processing system based on hybrid coding, which is characterized in that it is used to implement the visual signal processing method based on hybrid coding as described above, including:

[0016] A signal acquisition module that real-time acquires the visual cortex electroencephalogram signal, electrooculogram signal, and head electromyogram signal of the subject under the stimulation of the test image;

[0017] A signal preprocessing module that performs signal preprocessing on the acquired electroencephalogram signal and separates the visual cortex electroencephalogram signal;

[0018] A signal feature encoding network that mixes and encodes the preprocessed signal and the user's identity information according to the set hybrid coding rules to generate a signal feature code corresponding to the image stimulus;

[0019] An image feature encoding network that converts the corresponding stimulus image into an image feature code;

[0020] A feature mapping network that inputs the cluster centers of the electroencephalogram feature distribution and the image feature distribution of the corresponding category into the feature mapping network to obtain the cluster center vectors of the mapped electroencephalogram feature distribution and the image feature distribution, that is, the mapping relationship between the electroencephalogram feature distribution and the image feature distribution, and calculates the Euclidean distance between the cluster center vector of the mapped electroencephalogram feature distribution and the cluster center vector of the image feature distribution to obtain a mapping error; the mapping error can be used to train the feature mapping network through the backpropagation algorithm;

[0021] A reconstructed image generation system that decodes the feature encoding of the image to obtain a reconstructed image result.

[0022] Further, the signal acquisition system includes but is not limited to implantable brain electrodes EEG, electrooculogram signal measuring instruments, and microarray surface electromyogram acquisition electrodes.

[0023] In a third aspect, the present invention provides a visual signal processing device based on hybrid coding, which is characterized in that it includes the visual signal processing system as described above and an image display device.

[0024] In a last aspect, the present invention provides an electronic device, which is characterized by comprising: a memory storing executable program codes; a processor coupled to the memory; and the processor invoking the executable program codes stored in the memory for executing the above-mentioned visual signal processing method based on hybrid coding.

[0025] Compared with the prior art, the present invention has the following technical effects:

[0026] The present invention preprocesses visual electroencephalogram signals according to the collected electrooculogram signals and head electromyogram signals, reduces the interference of electrooculogram signals and electromyogram signals, and improves the result accuracy of the system; by taking into account information such as the age and gender of the subject, the result accuracy of the system is improved. More specifically:

[0027] (1) The present invention amplifies the collected signals first and then filters them, and at the same time adds component analysis and filtering of interference signals such as electrooculogram signals and electromyogram signals.

[0028] (2) The information processing method of the present invention takes into account the differences of actual users, adds user identity status features, and improves the accuracy of the output results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flowchart of the visual signal processing method in an embodiment of the present invention;

[0030] Figure 2 is an architecture diagram of the visual signal processing system in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, but it is not limited to the present invention.

[0032] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it will be apparent to those skilled in the art that well-known algorithms or models (such as signal amplifiers, band-pass filters, principal component analysis, neural network models) are not shown in detail to avoid obscuring the gist of the present invention.

[0033] Embodiment 1

[0034] Referring to Figure 1 , this embodiment provides a visual signal processing method based on hybrid coding, including the following steps:

[0035] Step 401: Implant and wear potential sampling electrodes on the subject, and collect visual cortex electroencephalogram signals, electrooculogram signals, and head electromyogram signals of the subject under the stimulation of test images in real time.

[0036] Step 402: Preprocess the collected potential signals and isolate the electroencephalogram (EEG) signals of the visual cortex. As a specific example, first amplify the small-amplitude signals collected, remove the noise caused by the environment from the amplified signals through band-pass filtering, perform principal component analysis on the filtered signals, and remove the interference of electrooculogram (EOG) signals and electromyogram (EMG) signals on the EEG signals to complete signal separation. The signals collected are first amplified and then filtered, and at the same time, component analysis and filtering of interference signals such as EOG signals and EMG signals are added.

[0037] Step 403: Considering that the P300-related potential has a high correlation with cognition and may be affected by factors such as the gender and age of the subject, use a neural network to perform hybrid encoding on the EEG signals of the visual cortex including P300 and the identity or status information of the subject to obtain signal feature encoding, and at the same time perform image feature encoding on the test images; establish a feature mapping network to complete the mapping between the signal feature encoding and the image feature encoding, and train the network parameters according to the loss function and the backpropagation algorithm. The identity or status information of the subject includes but is not limited to the age and gender information of the subject.

[0038] As a specific example, the signal feature encoding is to perform hybrid encoding on the preprocessed signals and the identity information of the subject to generate signal feature codes corresponding to the image stimuli; the rule of hybrid encoding is to extract time-domain features (such as mean, variance, etc.) and frequency-domain features (such as spectral energy, etc.) of the preprocessed signals to form corresponding feature vectors, and then splice and fuse them with the body status information vector of the subject. The loss function of the neural network for signal feature encoding is where n represents the sample size in one training, σ i,tar and σ i,out respectively represent the one-hot code of the signal feature classification label of the i-th sample and the probability code of the network output result. Using this loss function can make the encoding result of the model closer to the true label, enable the encoding result to more accurately reflect the actual signal features, and improve the accuracy of the model.

[0039] The loss function of the neural network for the image feature encoding is where n represents the sample size in one training, γ i,tar and respectively represent the one-hot code of the image label of the i-th sample and the network output result.

[0040] The mapping between the signal feature encoding and the image feature encoding is to input the clustering centers of the EEG feature distribution and the image feature distribution of the corresponding category into the feature mapping network, obtain the clustering center vectors of the mapped EEG feature distribution and the image feature distribution, that is, the mapping relationship between the EEG feature distribution and the image feature distribution, and calculate the Euclidean distance between the clustering center vector of the mapped EEG feature distribution and the clustering center vector of the image feature distribution to obtain the mapping error; the mapping error can be used to train the feature mapping network through the backpropagation algorithm; the loss function of the feature mapping network is where n represents the sample size in one training, ε i,img and ε i,map respectively represent the clustering center encodings after the mapped network of the actual image feature clustering center encoding and the signal feature clustering center encoding of the i-th mapping, ω i is the weight of different classification labels, which is used for the imbalanced data in the dataset, so that the model can pay more attention to some categories and improve the overall performance of the model. Initially, all label weights are 1, and this weight is adjusted according to the image categories in the actual dataset, which can improve the recognition accuracy of the model for each different category. As an example, the image weight of animal objects can be set to 1.1, and the image weight of vehicles can be set to 0.9. Such an assignment can make the model have a higher recognition accuracy for animal categories.

[0041] Step 404, by decoding the image feature encoding obtained by mapping the signal feature encoding, use the reconstructed image generation network to generate the corresponding image. As a specific example, the reconstructed image generation network includes a trained Instance-Conditioned GAN (IC-GAN, IC-generative adversarial network) model, which can decode the mapped image and generate the corresponding image. Using this model can visualize the EEG signal in the form of an image, so that it can be more vividly compared with the original image.

[0042] As can be seen from the above description, in this embodiment, the visual EEG signal is preprocessed according to the collected electrooculogram signal and the head electromyogram signal to reduce the interference of the electrooculogram signal and the electromyogram signal and improve the result accuracy of the system; by taking into account the information such as the age and gender of the subject, the accuracy of the processing result is improved.

[0043] The visual signal processing method described in this embodiment can be used as a computer program product. When the computer program is executed by a processor, it implements the visual signal processing method described in Embodiment 1. If the above functions are implemented in the form of software function units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0044] Similarly, the visual signal processing method described in this embodiment can be run on an electronic device including a memory storing executable program codes and a processor coupled to the memory. The processor calls the executable program codes stored in the memory to execute the above-described visual signal processing method based on hybrid coding.

[0045] Embodiment 2

[0046] See Figure 2 , this embodiment provides a visual signal processing system based on hybrid coding for implementing the visual signal processing method based on hybrid coding described in Embodiment 1, including:

[0047] A signal acquisition module 100 that real-time acquires electrocorticogram signals, electrooculogram signals, and head electromyogram signals of a subject under the stimulation of a test image;

[0048] A signal preprocessing module 200 that performs signal preprocessing on the acquired electrocorticogram signals and separates the electrocorticogram signals of the visual cortex;

[0049] A signal feature encoding network 300 that mixes and encodes the preprocessed signals and the user's identity information according to a set hybrid coding rule to generate a signal feature code corresponding to the image stimulation;

[0050] An image feature encoding network 400 that converts the corresponding stimulation image into an image feature code;

[0051] The feature mapping network 500 inputs the cluster centers of the EEG feature distribution and the image feature distribution corresponding to the categories into the feature mapping network, obtains the cluster center vectors of the mapped EEG feature distribution and the image feature distribution, that is, the mapping relationship between the EEG feature distribution and the image feature distribution, and calculates the Euclidean distance between the cluster center vector of the mapped EEG feature distribution and the cluster center vector of the image feature distribution to obtain the mapping error; the mapping error can be used to train the feature mapping network through the backpropagation algorithm;

[0052] The reconstructed image generation system 600 decodes the feature encoding of the image to obtain the reconstructed image result.

[0053] Preferably, the signal acquisition system includes but is not limited to implantable brain electrodes EEG, electrooculogram signal measuring instruments, and microarray surface electromyogram acquisition electrodes.

[0054] Finally, the present invention provides an electronic device, which is characterized by including: a memory storing executable program code; a processor coupled to the memory; the processor calls the executable program code stored in the memory to execute the above-mentioned visual signal processing method based on hybrid coding.

[0055] Embodiment 3

[0056] This embodiment provides a visual signal processing device based on hybrid coding, including the visual signal processing system and an image display device as described in Embodiment 2. The image display device is a device that can display an image with certain characteristics prepared in advance to the subject, and options include a screen, head-mounted glasses, etc.

[0057] Those skilled in the art should understand that those skilled in the art can implement variations in combination with the prior art and the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention and will not be elaborated here.

[0058] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and the devices and structures not described in detail should be understood to be implemented in a common manner in the art; any person skilled in the art can make many possible changes and modifications to the technical solution of the present invention, or modify it into an equivalent embodiment with equivalent changes, without departing from the scope of the technical solution of the present invention, which does not affect the essence of the present invention. Therefore, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of protection of the technical solution of the present invention.

[0059] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with this embodiment can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such an implementation should not be considered to exceed the scope of this application.

Claims

1. A visual signal processing method based on hybrid coding, characterized in that: The following steps are involved: Step 401: implanting and wearing potential sampling electrodes for the subject to collect the subject's visual cortex electroencephalogram signals, eye contact signals and head electromyography signals under the stimulation of the test image in real time; Step 402: pre-process the collected potential signal and separate the visual cortex EEG signal; Step 403: Use a neural network to perform mixed encoding on the visual cortex EEG signal and the identity or state information of the subject to obtain signal feature encoding, and perform image feature encoding on the test image at the same time; Establishing a feature mapping network to complete the mapping between the signal feature coding and the image feature coding, and training the network parameters according to the loss function and the back propagation algorithm; Step 404: Decoding the image feature code obtained by the signal feature code mapping, and generating a corresponding image using a reconstructed image generation network; In step 403, the identity or status information of the subject includes the age and gender information of the subject; Signal feature coding is to mix and encode the preprocessed signal and the identity or state information of the subject to generate a signal feature code corresponding to the image stimulus; the rule of mixed coding is to extract the time domain features and frequency domain features of the preprocessed signal to form a corresponding feature vector, and then splice and fuse it with the identity or state information vector of the subject; The loss function of the neural network for signal feature encoding is ,in, represents the sample size in one training. and Respectively represent The unique hot code of the signal feature classification label of each sample and the probability code of the network output result.

2. The method for processing visual signals based on hybrid coding according to claim 1, characterized in that: In step 402, the collected small-amplitude signal is first amplified, the amplified signal is subjected to bandpass filtering to remove noise caused by the environment, and the filtered signal is subjected to principal component analysis to remove interference of electrooculographic signals and electromyographic signals on visual cortical electroencephalogram signals, thereby completing signal separation.

3. The method for processing visual signals based on hybrid coding according to claim 1, characterized in that: In step 403, the loss function of the neural network for image feature encoding is: ,in, represents the sample size in one training. and Respectively represent The image labels of the samples are one-hot codes and the network output results.

4. The method for processing visual signals based on hybrid coding according to claim 3, characterized in that: In step 403, the mapping between the signal feature coding and the image feature coding is to input the cluster centers of the EEG feature distribution and the image feature distribution of the corresponding categories into the feature mapping network, obtain the cluster center vectors of the mapped EEG feature distribution and the image feature distribution, that is, the mapping relationship between the EEG feature distribution and the image feature distribution, and calculate the Euclidean distance between the cluster center vectors of the mapped EEG feature distribution and the cluster center vectors of the image feature distribution to obtain the mapping error; the mapping error is used to train the feature mapping network through a back propagation algorithm; The loss function of the feature mapping network is ,in, represents the sample size in one training. and Respectively represent The actual image feature cluster center encoding and signal feature cluster center encoding mapped by the network are the cluster center encoding after the mapping. are the weights of different classification labels.

5. A visual signal processing system based on hybrid coding, characterized in that: Used to implement the visual signal processing method based on hybrid coding as described in any one of claims 1 to 4, comprising: The signal acquisition module collects the visual cortex electroencephalogram signals, eye electrogram signals and head electromyography signals of the subjects under the stimulation of the test images in real time; The signal preprocessing module performs signal preprocessing on the collected EEG signals and separates the visual cortex EEG signals; The signal feature coding network performs mixed coding of the pre-processed signal and the user's identity or status information through a set mixed coding rule to generate a signal feature code corresponding to the image stimulus; Image feature encoding network, which converts the corresponding stimulus image into an image feature code; A feature mapping network, inputting the cluster centers of the EEG feature distribution and the image feature distribution of the corresponding category into the feature mapping network, obtaining the cluster center vectors of the mapped EEG feature distribution and the image feature distribution, that is, the mapping relationship between the EEG feature distribution and the image feature distribution, and calculating the Euclidean distance between the cluster center vectors of the mapped EEG feature distribution and the cluster center vectors of the image feature distribution to obtain a mapping error; the mapping error is used to train the feature mapping network through a back propagation algorithm; The reconstructed image generation system decodes the feature coding of the image to obtain a reconstructed image result.

6. A visual signal processing system based on hybrid coding according to claim 5, characterized in that: The signal acquisition module includes an implantable EEG electrode, an electrooculogram signal measuring instrument and a microarray surface electromyography acquisition electrode.

7. A visual signal processing device based on hybrid coding, characterized in that: The method comprises a visual signal processing system and an image display device as described in any one of claims 5 or 6.

8. An electronic device, characterized in that: include: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the visual signal processing method based on hybrid coding according to any one of claims 1 to 4.

Images