Neural signal decoding method and system based on deviation prediction and feature compensation

Abstract

The invention discloses a neural signal decoding method and system based on deviation prediction and feature compensation, and the method comprises the steps: obtaining a visual image and a neural signal, extracting a reference visual feature of the visual image based on a visual encoder, and extracting an initial neural signal feature of the neural signal based on a to-be-trained neural signal encoder; based on a deviation prediction module, deviation information is generated through the initial neural signal feature, and the deviation information is used for representing the difference between the initial neural signal feature and an ideal feature capable of being aligned with the reference visual feature; compensating the initial neural signal feature through the deviation information to generate a compensated neural signal feature; optimizing a neural signal encoder and a deviation prediction module; and obtaining a target neural signal, and decoding the target neural signal based on the optimized neural signal encoder and the deviation prediction module. According to the invention, the effect of highly aligning the decoded neural signal and visual signal can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of neural signal processing, and particularly relates to a neural signal decoding method based on bias prediction and feature compensation. BACKGROUND

[0002] As a key direction in the intersection of brain-computer interface and artificial intelligence, the goal of visual neural decoding technology is to reconstruct or recognize the visual content perceived by individuals from brain activity. Existing technologies generally adopt a multi-modal learning framework, which maps neural signals to a representation space shared with image features through training a neural signal encoder, and performs content matching or reconstruction in this space. However, there is a huge information gap between neural signals and original visual images. On the one hand, this is due to the physiological limitations of the human visual system and the subjectivity of cognitive activities. On the other hand, it is due to the inevitable technical noise in the signal acquisition process.

[0003] In order to bridge this gap, in related technologies, a "downward compatible" strategy is usually adopted, that is, active degradation processing (such as blurring) is performed on the original high-fidelity visual image to simulate the information loss of the visual system, thereby reducing the information amount of the image to match the incomplete neural signal. The degradation model (such as fixed Gaussian blur) used cannot adapt to the dynamic changes of perception characteristics and attention allocation in different individuals and different task scenarios, and lacks personalization and adaptability. The pre-processing of the image permanently loses high-frequency details and artificially reduces the theoretical upper limit of the decoding accuracy of the model. The essence of the method is to passively reduce the reference standard to adapt to the defective signal, rather than actively repairing the problems existing in the signal itself.

[0004] Therefore, there is an urgent need in the art for a new neural signal decoding method that can break through the above limitations and directly align with high-fidelity visual information. SUMMARY

[0005] The purpose of the present application is to provide a neural signal decoding method and system based on bias prediction and feature compensation, which can realize the effect of high alignment between the decoded neural signal and the visual signal.

[0006] To achieve the above purpose, the present application provides the following solutions: In a first aspect, the application provides a neural signal decoding method based on bias prediction and feature compensation, characterized in that the neural signal decoding method based on bias prediction and feature compensation comprises: acquiring a visual image and a neural signal, extracting a reference visual feature of the visual image based on a visual encoder, and extracting an initial neural signal feature of the neural signal based on a neural signal encoder to be trained; based on a bias prediction module, bias information is obtained from the initial neural signal feature, wherein the bias information is used to represent the difference between the initial neural signal feature and an ideal feature that can be aligned with the reference visual feature; the initial neural signal feature is compensated by the bias information to generate a compensated neural signal feature; the neural signal encoder and the bias prediction module are optimized with the goal of minimizing the distance between the compensated neural signal feature and the reference visual feature; and a target neural signal is acquired, and the target neural signal is decoded based on the optimized neural signal encoder and the bias prediction module.

[0007] Exemplarily, the bias prediction module comprises a bias prediction neural network configured to receive the initial neural signal feature and output a bias vector with the same dimension as the initial neural signal feature.

[0008] Exemplarily, the bias prediction module is a multi-layer perception comprising a non-linear activation function and at least two fully connected layers.

[0009] Exemplarily, the calculation process of the bias vector is as follows: wherein, is the initial neural signal feature, f D represents the bias prediction neural network, θ D is a parameter to be optimized of the bias prediction neural network; d represents the bias vector; the initial neural signal feature is compensated by the bias information to generate a compensated neural signal feature, comprising: applying the bias vector to the initial neural signal feature in the form of vector addition, vector subtraction or gate fusion, specifically as follows: wherein, represents the compensated neural signal feature after compensation by the bias vector.

[0010] Exemplarily, the bias prediction module and the neural signal encoder are jointly trained end-to-end by a joint loss function to achieve optimization; the joint loss function comprises an alignment loss and a bias regularization loss, wherein the alignment loss is calculated based on a contrast learning principle, and the bias regularization loss constrains the numerical value of the bias information.

[0011] Exemplarily, the calculation process of the alignment loss is as follows: wherein, represents the alignment loss, sim(·) is a cosine similarity function, τ is a temperature hyperparameter for adjusting the sharpness of the similarity distribution, is the batch size, represents the compensated neural signal feature, hv positive represents the same as hb compensated The paired reference visual features constitute a positive sample pair, represents the first reference visual feature in the same batch as the current sample, which constitutes a negative sample pair.

[0012] Exemplarily, the bias regularization loss is regularized by L2 norm, and the calculation method of the bias regularization loss is as follows: wherein, is the bias information, is a hyperparameter for balancing the alignment loss and the regularization loss, represents the square of the L2 norm of the bias information; the expression of the joint loss function is as follows: L total =L align +L reg wherein, represents the alignment loss, L reg represents the bias regularization loss, L total represents the total loss.

[0013] Exemplarily, the joint training is performed end-to-end through a joint loss function, including: calculating partial derivatives of a total loss with respect to model parameters as gradients, wherein the model parameters include to-be-optimized parameters of the bias prediction module and the neural signal encoder; and synchronously updating the to-be-optimized parameters of the neural signal encoder and the bias prediction module according to the gradients based on an optimization algorithm.

[0014] In a second aspect, the present application provides a neural signal decoding system based on bias prediction and feature compensation, comprising: a visual encoder configured to extract reference visual features of an original visual image; a neural signal encoder configured to extract initial brain signal features of a neural signal; a bias prediction module configured to receive the initial brain signal features and output bias information; a feature compensation module configured to generate compensated neural signal features according to the bias information; and a model training module configured to update the neural signal encoder and the bias prediction module by aligning the compensated neural signal features and the reference visual features, so as to decode a target neural signal according to the optimized neural signal encoder and the bias prediction module.

[0015] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the neural signal decoding method based on bias prediction and feature compensation according to any one of the above aspects.

[0016] According to the specific embodiments provided by the present application, the following technical effects are disclosed: The present application provides a neural signal decoding method and system based on bias prediction and feature compensation, extracts reference visual features of an original visual image, and through subsequent compensation and optimization, finally enables the neural signal features to be directly aligned with high-fidelity information in the feature space, overcoming the information loss problem caused by image degradation in the prior art, and improving the performance upper limit of the decoding model; by introducing a bias prediction module to generate bias information, the decoding strategy is changed to actively predict and repair defects in the signal, so that the model can dynamically compensate for the unique noise and bias pattern of each neural signal sample, and through minimizing the distance between the compensated features and the reference visual features as the target, the neural signal encoder and the bias prediction module are jointly optimized, so that the finally trained model has stronger and more stable overall performance, and the decoding accuracy and robustness are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only illustrate some of the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0018] Figure 1 The flow chart of the neural signal decoding method based on bias prediction and feature compensation in the embodiments of the present application.

[0019] Figure 2 The flow chart of the optimization training of the brain signal encoder and the bias prediction model in the embodiments of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] The above purposes, features and advantages of the present application can be more obvious and easy to understand. The present application will be further described in detail below with reference to the drawings and specific embodiments.

[0022] As shown in Figure 1 The embodiments of the present application provide a neural signal decoding method based on bias prediction and feature compensation. The method comprises the following steps: S110, acquiring a visual image and a neural signal, extracting a reference visual feature of the visual image based on a visual encoder, and extracting an initial neural signal feature of the neural signal based on a neural signal encoder to be trained.

[0023] S120, generating bias information based on a bias prediction module through the initial neural signal feature, wherein the bias information is used to represent the difference between the initial neural signal feature and an ideal feature that can be aligned with the reference visual feature.

[0024] S130, compensating the initial neural signal feature through the bias information to generate a compensated neural signal feature.

[0025] S140, optimizing the neural signal encoder and the bias prediction module with the objective of minimizing the distance between the compensated neural signal feature and the reference visual feature.

[0026] S150, acquiring a target neural signal, and decoding the target neural signal based on the optimized neural signal encoder and the bias prediction module.

[0027] The neural signal decoding method based on deviation prediction and feature compensation in the embodiment of the application extracts the reference visual features of the original visual image, and through compensation and optimization in subsequent steps, finally enables the neural signal features to be directly aligned with high-fidelity information in the feature space, overcoming the information loss problem caused by image degradation in the prior art, and improving the performance upper limit of the decoding model; by introducing a deviation prediction module to generate deviation information, the decoding strategy is changed to actively predict and repair defects in the signal, so that the model can dynamically compensate for the unique noise and deviation mode of each neural signal sample, and by minimizing the distance between the compensated features and the reference visual features as the target, the neural signal encoder and the deviation prediction module are jointly optimized, so that the overall performance of the finally trained model is stronger and more stable, and the accuracy and robustness of decoding are enhanced.

[0028] Exemplarily, the deviation prediction module includes a deviation prediction neural network configured to receive initial neural signal features and output a deviation vector with the same dimension as the initial neural signal features. The deviation prediction module is a multi-layer perception (MLP) containing a nonlinear activation function and at least two fully connected layers. The nonlinear activation function adopts a ReLU function or a GELU function. In the embodiment of the application, the training data exists in pairs, and each pair of data includes an original high-fidelity visual image and a corresponding neural signal data. Based on a fixed visual encoder, the reference visual features of the original high-fidelity visual image are extracted, and based on the neural signal encoder to be trained, the initial neural signal features of the neural signal data are extracted, thereby obtaining the processed paired signal features. Exemplarily, the neural signal is a brain signal, and the neural signal features are brain signal features.

[0029] The target of the deviation prediction module is to learn and output a deviation vector as deviation information, and the output deviation vector is the correction amount applied to minimize the difference between the compensated brain signal features and the reference visual features. The calculation process of the deviation vector is as follows: (1) wherein, hb initial is the initial neural signal feature, f D represents the deviation prediction neural network, θ D is the optimization parameter of the deviation prediction neural network; d represents the deviation vector.

[0030] The initial neural signal feature is compensated by bias information (i.e., bias vector) to generate a compensated neural signal feature, including: applying the bias vector to the initial neural signal feature in a manner of vector addition, vector subtraction or gated fusion. Exemplarily, the compensation and calculation process is specifically as follows: (2) wherein, represents the compensated neural signal feature compensated by the bias vector, d represents the bias vector.

[0031] Exemplarily, in the training data, for any compensated neural signal feature (brain signal feature) , the corresponding reference visual feature h v constitutes a positive sample pair, and all other visual features in the batch constitute negative sample pairs. The goal of alignment is to narrow the distance between the positive sample pairs and to push away the distance between all negative sample pairs in the feature space. This process is completed by maximizing the similarity between the positive sample pairs and minimizing the similarity between the negative sample pairs. The calculation process of the similarity adopts cosine similarity, which is specifically calculated as follows: (3) wherein, represents the compensated neural signal feature hb compensated , represents the reference visual feature h v or other visual features (when constituting negative sample pairs). represents the Euclidean norm of , and is the same. represents the product of the lengths of two vectors, which is a normalization factor for eliminating the influence of vector length on similarity calculation. The calculation result of the cosine similarity takes the value range [-1, 1].

[0032] Exemplarily, the bias prediction module and the neural signal encoder are jointly trained end-to-end by a joint loss function to achieve optimization. The joint loss function includes alignment loss and bias regularization loss, wherein the alignment loss is calculated based on the principle of contrastive learning, and the bias regularization loss constrains the numerical value of the bias information.

[0033] wherein, for a positive sample pair and N-1 negative sample pairs, the calculation process of the alignment loss in the joint loss is as follows: (4) wherein, denotes the alignment loss, sim(·) is a cosine similarity function, τ is a temperature hyper-parameter, used to adjust the sharpness of the similarity distribution, is a batch size, denotes the compensated neural signal feature, hv positive denotes the same as hb compensated a pair of reference visual features, constituting a positive sample pair, denotes the i-th reference visual feature in the same batch as the current sample, constituting a negative sample pair.

[0034] An L2 regularization constraint is imposed on the size of the bias vector d output by the bias prediction module to ensure the stability of the compensation process. The calculation method of the bias regularization loss is as follows: (5) wherein, is the bias information, is a hyper-parameter for balancing the alignment loss and the regularization loss, denotes the square of the L2 norm of the bias information.

[0035] After obtaining the alignment loss and the bias regularization loss, the total loss, i.e., the joint loss, can be obtained by the sum of the two, and its expression is as follows: L total =L align +L reg (6) wherein, denotes the alignment loss, L reg denotes the bias regularization loss, L total denotes the total loss.

[0036] After confirming the total loss, end-to-end joint training is performed through the joint loss function, including the following processes: calculating the partial derivative of the total loss with respect to the model parameters as the gradient, wherein the model parameters include the to-be-optimized parameters of the bias prediction module and the neural signal encoder. Based on the optimization algorithm, the to-be-optimized parameters of the neural signal encoder and the bias prediction module are updated synchronously according to the gradient.

[0037] The optimization process in the embodiments of the present application will be described below in conjunction with Figure 2 The optimization process in the embodiments of the present application will be described below in conjunction with 1. Forward propagation. ​

[0038] Given a batch of training data pairs, where each pair consists of an original high-fidelity visual image xv and a corresponding neural signal data xb, perform the following computations in sequence: The baseline visual features hv are obtained by processing xv through a visual encoder (parameter freezing); the brain signal encoder f... B (Parameter is θ) B Processing xb yields the initial brain signal features hb. initial (As shown in equation (1) above); in the deviation prediction module (parameter θ) D In the process of generating a deviation vector d, the deviation vector d is then compared with the initial brain signal feature hb in the feature compensation module. initial The brain signal features are added together to obtain the compensated brain signal features; finally, the joint loss is calculated using the above formulas (4)-(6).

[0039] 2. Backpropagation.

[0040] The purpose of this step is to calculate the joint loss L. total Regarding the trainable parameters θ of the model B and θ D The gradient (partial derivative) of the loss. According to the chain rule, the gradient starts from the total loss and flows backward through the entire computation graph.

[0041] Gradient flow direction deviation prediction module, L reg The magnitude of the deviation vector d is directly constrained, therefore its gradient also directly affects θ. D This means that the deviation prediction module f D The parameters are adjusted based on the contribution of the output deviation vector to the final alignment effect and its own size.

[0042] Gradient flow to the brain signal encoder: makes the brain signal encoder f B The optimization is affected by the deviation prediction module f D The encoder learns not only how to extract effective semantic information, but also how to organize its output feature representation hb. initial This allows downstream deviation prediction modules to perform deviation compensation more accurately and easily.

[0043] 3. Parameter correction.

[0044] After calculating the gradient, the parameters of the two modules are updated synchronously using an optimizer (such as Adam), as shown below: (7) (8) in, η For learning rate, and denote the updated brain signal encoder parameters and the bias prediction module parameters, and denote the gradient, this procedure ensures that the brain signal encoder and the bias prediction network are co-optimized.

[0045] After updating the parameters, a target neural signal is obtained, and the target neural signal is decoded based on the optimized neural signal encoder and the bias prediction module.

[0046] In an exemplary embodiment, a neural signal decoding system based on bias prediction and feature compensation is also provided, which comprises a visual encoder, a neural signal encoder, a bias prediction module and a model training module.

[0047] The visual encoder is configured to extract reference visual features of an original visual image, the neural signal encoder is configured to extract initial brain signal features of a neural signal, the bias prediction module is configured to receive the initial brain signal features and output bias information, and the feature compensation module is configured to generate compensated neural signal features according to the bias information. The model training module is configured to update the neural signal encoder and the bias prediction module by aligning the compensated neural signal features and the reference visual features, so as to decode a target neural signal according to the optimized neural signal encoder and the bias prediction module.

[0048] The neural signal decoding method and system based on bias prediction and feature compensation provided in the embodiments of the present application extract reference visual features of an original visual image, and through compensation and optimization in subsequent steps, finally make the neural signal features directly alignable with high-fidelity information in a feature space, overcome the information loss problem caused by image degradation in the prior art, and improve the performance upper limit of the decoding model. By introducing a bias prediction module to generate bias information, the decoding strategy is changed to actively predict and repair defects in the signal, so that the model can dynamically compensate for the unique noise and bias mode of each neural signal sample, and through minimizing the distance between the compensated features and the reference visual features as the target, the neural signal encoder and the bias prediction module are jointly optimized, so that the finally trained model has stronger and more stable overall performance, and the decoding accuracy and robustness are enhanced.

[0049] In an exemplary embodiment, a computer device is also provided, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0050] In an exemplary embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0051] In an example embodiment, a computer program product is provided, including a computer program which, when executed by a processor, implements the steps of the above-mentioned method embodiments.

[0052] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0053] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, database or other medium used in the embodiments provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0054] The database involved in the embodiments provided by the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on blockchain, etc., without being limited thereto. The processor involved in the embodiments provided by the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0055] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, it should be understood that the application encompasses all possible combinations of the technical features unless such a combination is not technically possible.

[0056] The principles and implementation manners of the present application are described herein by using specific examples, and the above embodiments are only used to help understand the method of the present application and its core idea; meanwhile, according to the idea of the present application, the specific implementation manners and application scopes will be changed by those skilled in the art. In conclusion, the content of the present specification should not be understood as a limitation of the present application.

Claims

1. A neural signal decoding method based on bias prediction and feature compensation, characterized in that, The neural signal decoding method based on bias prediction and feature compensation includes: Acquire visual images and neural signals, extract baseline visual features of the visual images based on a visual encoder, and extract initial neural signal features of the neural signals based on a neural signal encoder to be trained; Based on the deviation prediction module, deviation information is generated from the initial neural signal features, wherein the deviation information is used to characterize the difference between the initial neural signal features and an ideal feature that can be aligned with the benchmark visual features; The initial neural signal features are compensated using the deviation information to generate compensated neural signal features; The neural signal encoder and the bias prediction module are optimized with the goal of minimizing the distance between the compensated neural signal features and the baseline visual features. The target neural signal is acquired, and the target neural signal is decoded based on the optimized neural signal encoder and the deviation prediction module.

2. The neural signal decoding method based on bias prediction and feature compensation according to claim 1, characterized in that, The deviation prediction module includes a deviation prediction neural network, which is configured to receive the initial neural signal features and output a deviation vector with the same dimension as the initial neural signal features.

3. The neural signal decoding method based on bias prediction and feature compensation according to claim 2, characterized in that, The deviation prediction module is a multilayer perceptron, which includes a nonlinear activation function and at least two fully connected layers.

4. The neural signal decoding method based on bias prediction and feature compensation according to claim 2, characterized in that, The calculation process of the deviation vector is as follows: in, hb initial Initial neural signal characteristics, f D Representative bias prediction neural network, θ D The parameters to be optimized for the deviation prediction neural network; d Represents the deviation vector; The initial neural signal features are compensated using the deviation information to generate compensated neural signal features, including: applying the deviation vector to the initial neural signal features using vector addition, vector subtraction, or gated fusion, as detailed below: in, This represents the characteristics of the compensated neural signal after compensation by the aforementioned deviation vector.

5. The neural signal decoding method based on bias prediction and feature compensation according to claim 1, characterized in that, The deviation prediction module and the neural signal encoder are jointly trained end-to-end using a joint loss function to achieve optimization. The joint loss function includes alignment loss and deviation regularization loss, wherein the alignment loss is calculated based on the principle of contrastive learning, and the deviation regularization loss constrains the numerical value of the deviation information.

6. The neural signal decoding method based on bias prediction and feature compensation according to claim 5, characterized in that, The calculation process for the alignment loss is as follows: in, This represents the alignment loss, where sim(⋅) is the cosine similarity function. τ This is a temperature hyperparameter used to adjust the sharpness of the similarity distribution. For batch size, This indicates the characteristics of the compensated neural signals after compensation. hv positive Indicates and hb compensated Paired baseline visual features constitute positive sample pairs. This indicates the first sample in the same batch that is not matched with the current sample. Each baseline visual feature constitutes a negative sample pair.

7. The neural signal decoding method based on bias prediction and feature compensation according to claim 6, characterized in that, The deviation regularization loss is regularized using the L2 norm, and the calculation method for the deviation regularization loss is as follows: in, For deviation information, To balance the hyperparameters of alignment loss and regularization loss, The square of the L2 norm of the deviation information; The expression for the joint loss function is as follows: L total = L align + L reg in, Indicates alignment loss. L reg This represents the deviation regularization loss. L total This indicates the total loss.

8. The neural signal decoding method based on bias prediction and feature compensation according to any one of claims 5-7, characterized in that, End-to-end joint training using a joint loss function includes: The partial derivative of the total loss with respect to the model parameters is calculated as the gradient, where the model parameters include the parameters to be optimized for the bias prediction module and the neural signal encoder; Based on the optimization algorithm, the parameters to be optimized for the neural signal encoder and the deviation prediction module are updated synchronously according to the gradient.

9. A neural signal decoding system based on bias prediction and feature compensation, characterized in that, The neural signal decoding system based on bias prediction and feature compensation includes: A visual encoder is used to extract baseline visual features from raw visual images; A neural signal encoder is used to extract the initial brain signal features of neural signals. The deviation prediction module is configured to receive the initial brain signal features and output deviation information; The feature compensation module is configured to generate compensated neural signal features based on the deviation information; The model training module is configured to update the neural signal encoder and the bias prediction module by aligning the compensated neural signal features and the baseline visual features, so as to decode the target neural signal based on the optimized neural signal encoder and the bias prediction module.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the neural signal decoding method based on bias prediction and feature compensation as described in any one of claims 1-8.

Images