A neural signal decoding method and system based on bias prediction and feature compensation

Abstract

This application discloses a neural signal decoding method and system based on bias prediction and feature compensation. The method includes: acquiring a visual image and a neural signal; extracting reference visual features of the visual image based on a visual encoder, and extracting initial neural signal features of the neural signal based on a neural signal encoder to be trained; generating bias information based on the initial neural signal features using a bias prediction module, wherein the bias information is used to characterize the difference between the initial neural signal features and an ideal feature that can be aligned with the reference visual features; compensating the initial neural signal features using the bias information to generate compensated neural signal features; optimizing the neural signal encoder and the bias prediction module; acquiring a target neural signal, and decoding the target neural signal based on the optimized neural signal encoder and the bias prediction module. This application can achieve a high degree of alignment between the decoded neural signal and the visual signal.

Description

Technical Field

[0001] This application relates to the technical field of neural signal processing, and in particular to a neural signal decoding method based on bias prediction and feature compensation. Background Technology

[0002] Visual neural decoding technology, a key direction at the intersection of brain-computer interfaces and artificial intelligence, aims to reconstruct or identify visual content perceived by an individual from brain activity. Current technologies generally employ a multimodal learning framework, training a neural signal encoder to map neural signals into a representation space shared with image features, and then performing content matching or reconstruction within this space. However, a significant information gap exists between neural signals and the original visual image. This stems partly from the physiological limitations of the human visual system and the subjectivity of cognitive activity, and partly from unavoidable technical noise during signal acquisition.

[0003] To bridge this gap, related technologies typically employ a "backward-compatible" strategy, which involves actively degrading the original high-fidelity visual image (e.g., blurring) to simulate information loss in the visual system, thereby reducing the image's information content to match incomplete neural signals. However, the degradation models used (such as fixed Gaussian blur) cannot adapt to the dynamically changing perceptual characteristics and attention allocation of different individuals and task scenarios, lacking personalization and adaptability. This image preprocessing permanently loses high-frequency details, artificially lowering the theoretical upper limit of the model's decoding accuracy. Essentially, the method passively lowers the reference standard to accommodate defective signals, rather than actively repairing the inherent problems in the signal itself.

[0004] Therefore, there is an urgent need in this field for a new method of neural signal decoding that can overcome the above limitations and directly align with high-fidelity visual information. Summary of the Invention

[0005] The purpose of this application is to provide a neural signal decoding method and system based on bias prediction and feature compensation, which can achieve a high degree of alignment between the decoded neural signal and the visual signal.

[0006] To achieve the above objectives, this application provides the following solution:

[0007] In a first aspect, this application provides a neural signal decoding method based on bias prediction and feature compensation, characterized in that the neural signal decoding method based on bias prediction and feature compensation includes: acquiring a visual image and a neural signal; extracting a reference visual feature of the visual image based on a visual encoder, and extracting initial neural signal features of the neural signal based on a neural signal encoder to be trained; generating bias information through the initial neural signal using a bias prediction module, wherein the bias information is used to characterize the difference between the initial neural signal features and an ideal feature that can be aligned with the reference visual feature; compensating the initial neural signal features using the bias information to generate compensated neural signal features; optimizing the neural signal encoder and the bias prediction module with the goal of minimizing the distance between the compensated neural signal features and the reference visual feature; acquiring a target neural signal, and decoding the target neural signal based on the optimized neural signal encoder and the bias prediction module.

[0008] For example, the deviation prediction module includes a deviation prediction neural network configured to receive the initial neural signal features and output a deviation vector with the same dimension as the initial neural signal features.

[0009] For example, the deviation prediction module is a multilayer perceptron, which includes a nonlinear activation function and at least two fully connected layers.

[0010] For example, the calculation process of the deviation vector is as follows:

[0011]

[0012] in, Initial neural signal characteristics, f D Representative bias prediction neural network, θ D The parameters to be optimized for the deviation prediction neural network; d The deviation vector represents the initial neural signal features. The initial neural signal features are compensated using the deviation information to generate compensated neural signal features, including applying the deviation vector to the initial neural signal features via vector addition, vector subtraction, or gated fusion, as detailed below:

[0013]

[0014] in, This represents the characteristics of the compensated neural signal after compensation by the aforementioned deviation vector.

[0015] For example, the deviation prediction module and the neural signal encoder are jointly trained end-to-end using a joint loss function to achieve optimization; the joint loss function includes alignment loss and deviation regularization loss, wherein the alignment loss is calculated based on the principle of contrastive learning, and the deviation regularization loss constrains the value of the deviation information.

[0016] For example, the calculation process of the alignment loss is as follows:

[0017]

[0018] in, Represents alignment loss, sim(⋅) The cosine similarity function is used. τ This is a temperature hyperparameter used to adjust the sharpness of the similarity distribution. For batch size, This indicates the characteristics of the compensated neural signals after compensation. hv positive Indicates and hb compensated Paired baseline visual features constitute positive sample pairs. This indicates the first sample in the same batch that is not matched with the current sample. Each baseline visual feature constitutes a negative sample pair.

[0019] For example, the deviation regularization loss is regularized using the L2 norm, and the deviation regularization loss is calculated as follows:

[0020]

[0021] in, For deviation information, To balance the hyperparameters of alignment loss and regularization loss, The square of the L2 norm of the bias information is represented; the expression for the joint loss function is as follows:

[0022] L total =L align +L reg

[0023] in, Indicates alignment loss. L reg This represents the deviation regularization loss. L total This indicates the total loss.

[0024] For example, end-to-end joint training via a joint loss function includes: calculating the partial derivative of the total loss with respect to the model parameters as a gradient, wherein the model parameters include the parameters to be optimized for the bias prediction module and the neural signal encoder; and synchronously updating the parameters to be optimized for the neural signal encoder and the bias prediction module according to the gradient based on an optimization algorithm.

[0025] Secondly, this application provides a neural signal decoding system based on bias prediction and feature compensation, comprising: a visual encoder for extracting baseline visual features of an original visual image; a neural signal encoder for extracting initial brain signal features of a neural signal; a bias prediction module configured to receive the initial brain signal features and output bias information; a feature compensation module configured to generate compensated neural signal features based on the bias information; and a model training module configured to update the neural signal encoder and the bias prediction module by aligning the compensated neural signal features and the baseline visual features, for decoding a target neural signal based on the optimized neural signal encoder and the bias prediction module.

[0026] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the neural signal decoding method based on bias prediction and feature compensation as described above.

[0027] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0028] This application provides a neural signal decoding method and system based on bias prediction and feature compensation. It extracts the baseline visual features of the original visual image, and through subsequent compensation and optimization steps, enables the neural signal features to be directly aligned with high-fidelity information in the feature space. This overcomes the information loss problem caused by image degradation in existing technologies and improves the performance ceiling of the decoding model. By introducing a bias prediction module to generate bias information, the decoding strategy is transformed into actively predicting and repairing defects in the signal. This allows the model to dynamically and individually compensate for the unique noise and bias patterns of each neural signal sample. By minimizing the distance between the compensated features and the baseline visual features, the neural signal encoder and bias prediction module are jointly optimized, resulting in a stronger and more stable overall model after training, enhancing the accuracy and robustness of decoding. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the neural signal decoding method based on bias prediction and feature compensation in the embodiments of this application.

[0031] Figure 2 This is a flowchart illustrating the optimized training process of the brain signal encoder and the bias prediction model in the embodiments of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] like Figure 1 As shown in the figure, this application provides a neural signal decoding method based on bias prediction and feature compensation. The method includes the following steps:

[0035] S110. Acquire visual images and neural signals, extract baseline visual features of the visual images based on the visual encoder, and extract initial neural signal features of the neural signals based on the neural signal encoder to be trained.

[0036] S120. Based on the deviation prediction module, deviation information is generated from the initial neural signal features, wherein the deviation information is used to characterize the difference between the initial neural signal features and an ideal feature that can be aligned with the benchmark visual features.

[0037] S130. The initial neural signal features are compensated by the deviation information to generate compensated neural signal features.

[0038] S140. Optimize the neural signal encoder and bias prediction module with the goal of minimizing the distance between the compensated neural signal features and the baseline visual features.

[0039] S150. Acquire the target neural signal and decode the target neural signal based on the optimized neural signal encoder and bias prediction module.

[0040] The neural signal decoding method based on bias prediction and feature compensation in this application extracts the baseline visual features of the original visual image. Through subsequent compensation and optimization steps, the neural signal features can be directly aligned with high-fidelity information in the feature space, overcoming the information loss problem caused by image degradation in existing technologies and improving the performance ceiling of the decoding model. By introducing a bias prediction module to generate bias information, the decoding strategy is transformed into actively predicting and repairing defects in the signal, enabling the model to dynamically and individually compensate for the unique noise and bias patterns of each neural signal sample. By minimizing the distance between the compensated features and the baseline visual features, the neural signal encoder and the bias prediction module are jointly optimized, resulting in a stronger and more stable overall model after training, and enhancing the accuracy and robustness of decoding.

[0041] Exemplarily, the bias prediction module includes a bias prediction neural network configured to receive initial neural signal features and output a bias vector with the same dimension as the initial neural signal features. The bias prediction module is a multilayer perceptron (MLP), which includes a nonlinear activation function and at least two fully connected layers. The nonlinear activation function is either the ReLU function or the GELU function. In this embodiment, the training data exists in pairs, each pair including an original high-fidelity visual image and a corresponding neural signal data. Based on a fixed visual encoder, baseline visual features of the original high-fidelity visual image are extracted; based on the neural signal encoder to be trained, initial neural signal features of the neural signal data are extracted, thereby obtaining processed paired signal features. Exemplarily, the neural signal is a brain signal, and the neural signal features are brain signal features.

[0042] The goal of the bias prediction module is to learn and output a bias vector as bias information. This output bias vector is a correction measure applied to minimize the difference between the compensating hindbrain signal features and the baseline visual features. The calculation process of the bias vector is as follows:

[0043] (1)

[0044] in, hb initial Initial neural signal characteristics, f D Representative bias prediction neural network, θ D The parameters to be optimized for the deviation prediction neural network; d This represents the deviation vector.

[0045] Compensating the initial neural signal features with deviation information (i.e., deviation vector) to generate compensated neural signal features includes applying the deviation vector to the initial neural signal features using vector addition, vector subtraction, or gated fusion. For example, the compensation and calculation process is detailed below:

[0046] (2)

[0047] in, This represents the characteristics of the compensated neural signal after compensation using the bias vector. d This represents the deviation vector.

[0048] For example, in the training data, for any compensated neural signal feature (brain signal feature) after compensation. Its corresponding baseline visual features h v A positive sample pair is formed, and all other visual features within the batch form a negative sample pair. The goal of alignment is to bring positive sample pairs closer together in the feature space while simultaneously distancing all negative sample pairs. This process is achieved by maximizing the similarity between positive sample pairs and minimizing the similarity between negative sample pairs. The similarity is calculated using cosine similarity, as shown in the following formula:

[0049] (3)

[0050] in, Representative neural signal characteristics after compensation ( hb compensated ), Represents the baseline visual features ( h v ) or other visual features (when forming negative sample pairs). Indicates to The Euclidean norm, for Similarly. This represents the product of the lengths of two vectors. It serves as a normalization factor to eliminate the influence of vector length on similarity calculation. The cosine similarity calculation result ranges from [-1, 1].

[0051] For example, the bias prediction module and the neural signal encoder are jointly trained end-to-end using a joint loss function to achieve optimization. The joint loss function includes an alignment loss and a bias regularization loss, where the alignment loss is calculated based on the principle of contrastive learning, and the bias regularization loss constrains the numerical value of the bias information.

[0052] For a positive sample pair and N-1 negative sample pairs, the calculation process of the alignment loss in the joint loss is as follows:

[0053] (4)

[0054] in, Represents alignment loss, sim(⋅) The cosine similarity function is used. τ This is a temperature hyperparameter used to adjust the sharpness of the similarity distribution. For batch size, This indicates the characteristics of the compensated neural signals after compensation. hv positive Indicates and hb compensated Paired baseline visual features constitute positive sample pairs. This indicates the first sample in the same batch that is not matched with the current sample. Each baseline visual feature constitutes a negative sample pair.

[0055] An L2 regularization constraint is applied to the magnitude of the deviation vector d output by the deviation prediction module to ensure the stability of the compensation process. The deviation regularization loss is calculated as follows:

[0056] (5)

[0057] in, For deviation information, To balance the hyperparameters of alignment loss and regularization loss, The square of the L2 norm of the deviation information is represented.

[0058] After obtaining the alignment loss and the bias regularization loss, the total loss, also known as the joint loss, can be obtained by summing the two, as shown in the following expression:

[0059] L total =L align +L reg (6)

[0060] in, Indicates alignment loss. L reg This represents the deviation regularization loss. L total This indicates the total loss.

[0061] After confirming the total loss, end-to-end joint training is performed using a joint loss function, including the following steps: The partial derivatives of the total loss with respect to the model parameters are calculated as the gradient, where the model parameters include the parameters to be optimized for the bias prediction module and the neural signal encoder. Based on the optimization algorithm, the parameters to be optimized for the neural signal encoder and the bias prediction module are updated synchronously according to the gradient.

[0062] The following is combined Figure 2 The optimization process in the embodiments of this application is described, and it specifically includes the following steps:

[0063] 1. Forward propagation.

[0064] Given a batch of training data pairs, where each pair consists of an original high-fidelity visual image xv and a corresponding neural signal data xb, perform the following computations in sequence:

[0065] The baseline visual features hv are obtained by processing xv through a visual encoder (parameter freezing); the brain signal encoder f... B (Parameter is θ) B Processing xb yields the initial brain signal features hb. initial (As shown in equation (1) above); in the deviation prediction module (parameter θ) D In the process of generating a deviation vector d, the deviation vector d is then compared with the initial brain signal feature hb in the feature compensation module. initial The brain signal features are added together to obtain the compensated brain signal features; finally, the joint loss is calculated using the above formulas (4)-(6).

[0066] 2. Backpropagation.

[0067] The purpose of this step is to calculate the joint loss L. total Regarding the trainable parameters θ of the model B and θ D The gradient (partial derivative) of the loss. According to the chain rule, the gradient starts from the total loss and flows backward through the entire computation graph.

[0068] Gradient flow direction deviation prediction module, L reg The magnitude of the deviation vector d is directly constrained, therefore its gradient also directly affects θ. D This means that the deviation prediction module f D The parameters are adjusted based on the contribution of the output deviation vector to the final alignment effect and its own size.

[0069] Gradient flow to the brain signal encoder: makes the brain signal encoder f B The optimization is affected by the deviation prediction module f DThe encoder learns not only how to extract effective semantic information, but also how to organize its output feature representation hb. initial This allows downstream deviation prediction modules to perform deviation compensation more accurately and easily.

[0070] 3. Parameter correction.

[0071] After calculating the gradient, the parameters of the two modules are updated synchronously using an optimizer (such as Adam), as shown below:

[0072] (7)

[0073] (8)

[0074] in, η For learning rate, and This indicates the updated parameters of the brain signal encoder and the parameters of the bias prediction module. and The gradient is represented by this process, which ensures that the brain signal encoder and the bias prediction network are co-optimized.

[0075] After updating and modifying the parameters, the target neural signal is acquired, and the target neural signal is decoded based on the optimized neural signal encoder and bias prediction module.

[0076] In one exemplary embodiment, a neural signal decoding system based on bias prediction and feature compensation is also provided. The system includes a visual encoder, a neural signal encoder, a bias prediction module, and a model training module.

[0077] A visual encoder is used to extract baseline visual features from the original visual image, a neural signal encoder is used to extract initial brain signal features from the neural signal, a bias prediction module is configured to receive the initial brain signal features and output bias information, and a feature compensation module is configured to generate compensated neural signal features based on the bias information. A model training module is configured to update the neural signal encoder and bias prediction module by aligning the compensated neural signal features with the baseline visual features, so as to decode the target neural signal based on the optimized neural signal encoder and bias prediction module.

[0078] This application provides a neural signal decoding method and system based on bias prediction and feature compensation. It extracts the baseline visual features of the original visual image and, through subsequent compensation and optimization steps, enables the neural signal features to be directly aligned with high-fidelity information in the feature space. This overcomes the information loss problem caused by image degradation in existing technologies and improves the performance ceiling of the decoding model. By introducing a bias prediction module to generate bias information, the decoding strategy is transformed into actively predicting and repairing defects in the signal. This allows the model to dynamically and individually compensate for the unique noise and bias patterns of each neural signal sample. By minimizing the distance between the compensated features and the baseline visual features, the neural signal encoder and bias prediction module are jointly optimized, resulting in a stronger and more stable overall model after training, enhancing the accuracy and robustness of decoding.

[0079] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0080] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0081] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0082] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0083] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0084] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A neural signal decoding method based on bias prediction and feature compensation, characterized in that, The neural signal decoding method based on bias prediction and feature compensation includes: Acquire visual images and neural signals, extract baseline visual features of the visual images based on a visual encoder, and extract initial neural signal features of the neural signals based on a neural signal encoder to be trained; Based on the deviation prediction module, deviation information is generated from the initial neural signal features, wherein the deviation information is used to characterize the difference between the initial neural signal features and an ideal feature that can be aligned with the benchmark visual features; The initial neural signal features are compensated using the deviation information to generate compensated neural signal features; The neural signal encoder and the bias prediction module are optimized with the goal of minimizing the distance between the compensated neural signal features and the baseline visual features. The target neural signal is acquired, and the target neural signal is decoded based on the optimized neural signal encoder and the deviation prediction module.

2. The neural signal decoding method based on bias prediction and feature compensation according to claim 1, characterized in that, The deviation prediction module includes a deviation prediction neural network, which is configured to receive the initial neural signal features and output a deviation vector with the same dimension as the initial neural signal features.

3. The neural signal decoding method based on bias prediction and feature compensation according to claim 2, characterized in that, The deviation prediction module is a multilayer perceptron, which includes a nonlinear activation function and at least two fully connected layers.

4. The neural signal decoding method based on bias prediction and feature compensation according to claim 2, characterized in that, The calculation process of the deviation vector is as follows: in, hb initial Initial neural signal characteristics, f D Representative bias prediction neural network, θ D The parameters to be optimized for the deviation prediction neural network; d Represents the deviation vector; The initial neural signal features are compensated using the deviation information to generate compensated neural signal features, including: applying the deviation vector to the initial neural signal features using vector addition, vector subtraction, or gated fusion, as detailed below: in, This represents the characteristics of the compensated neural signal after compensation by the aforementioned deviation vector.

5. The neural signal decoding method based on bias prediction and feature compensation according to claim 1, characterized in that, The deviation prediction module and the neural signal encoder are jointly trained end-to-end using a joint loss function to achieve optimization. The joint loss function includes alignment loss and deviation regularization loss, wherein the alignment loss is calculated based on the principle of contrastive learning, and the deviation regularization loss constrains the value of the deviation information.

6. The neural signal decoding method based on bias prediction and feature compensation according to claim 5, characterized in that, The calculation process for the alignment loss is as follows: in, This represents the alignment loss, where sim(⋅) is the cosine similarity function. τ This is a temperature hyperparameter used to adjust the sharpness of the similarity distribution. For batch size, This indicates the characteristics of the compensated neural signals after compensation. hv positive Indicates and hb compensated Paired baseline visual features constitute positive sample pairs. This indicates the first sample in the same batch that is not matched with the current sample. Each baseline visual feature constitutes a negative sample pair.

7. The neural signal decoding method based on bias prediction and feature compensation according to claim 6, characterized in that, The deviation regularization loss is regularized using the L2 norm, and the calculation method for the deviation regularization loss is as follows: in, For deviation information, To balance the hyperparameters of alignment loss and regularization loss, The square of the L2 norm of the deviation information; The expression for the joint loss function is as follows: L total = L align + L reg in, Indicates alignment loss. L reg This represents the deviation regularization loss. L total This indicates the total loss.

8. The neural signal decoding method based on bias prediction and feature compensation according to any one of claims 5-7, characterized in that, End-to-end joint training using a joint loss function includes: The partial derivative of the total loss with respect to the model parameters is calculated as the gradient, where the model parameters include the parameters to be optimized for the bias prediction module and the neural signal encoder; Based on the optimization algorithm, the parameters to be optimized for the neural signal encoder and the deviation prediction module are updated synchronously according to the gradient.

9. A neural signal decoding system based on bias prediction and feature compensation, characterized in that, The neural signal decoding system based on bias prediction and feature compensation includes: A visual encoder is used to extract baseline visual features from raw visual images; A neural signal encoder is used to extract the initial brain signal features of neural signals. The deviation prediction module is configured to receive the initial brain signal features and output deviation information; The feature compensation module is configured to generate compensated neural signal features based on the deviation information; The model training module is configured to update the neural signal encoder and the bias prediction module by aligning the compensated neural signal features and the baseline visual features, so as to decode the target neural signal based on the optimized neural signal encoder and the bias prediction module.

10. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the neural signal decoding method based on bias prediction and feature compensation as described in any one of claims 1-8.

Images