Signal processing method and related apparatus

Abstract

The present application provides a signal processing method, which is applied to the technical field of brain-computer interfaces (BCI), and comprises: acquiring a first signal, the first signal being a signal collected from brain tissue; acquiring a downstream feedback signal; and then, according to the feedback signal, processing the first signal. On the basis of the feedback signal, the method processes the first signal collected from the brain tissue, rather than directly inputting into a decoding apparatus the first signal to be decoded, thereby ensuring the long-term stability of signal coding and decoding so as to avoid the decrease or even failure of the long-term stability of signal coding and decoding caused by the non-stationarity of the signal, and to satisfy service requirements.

Description

A signal processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 3, 2023, with application number 202311133501.2 and invention name “A signal processing method and related device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a signal processing method, device, chip, system, computer-readable storage medium, and computer program product. Background Art

[0003] In recent years, brain-computer interfaces (BCIs) have become a key enabling technology for the future intelligent world. For example, in the medical field, a neurosurgery robot can implant a proprietary chip and information strip into brain tissue. Brain signals can then be read directly via a Universal Serial Bus (USB) interface. This method is minimally invasive and has broad application prospects.

[0004] BCI is a new communication and control technology that establishes connections between brain tissue (such as the human or animal brain) and computers or other electronic devices, independent of conventional brain information output pathways (peripheral nerves and muscle tissue). BCI is categorized into two main types: invasive and non-invasive, depending on the degree of electrode implantation.

[0005] Both invasively and non-invasively acquired signals are non-stationary, significantly affected by factors such as electrode inactivation, random noise, motion artifacts, and physical and mental state. This can lead to a decrease in the long-term stability of signal encoding and decoding, or even failure, making it difficult to meet business needs.

[0006] Summary of the Invention

[0007] This application provides a signal processing method that processes a first signal collected from brain tissue in conjunction with a downstream feedback signal, rather than directly inputting the first signal into a decoding device for decoding. This method ensures the long-term stability of signal encoding and decoding, avoids signal non-stationarity leading to a decrease in the long-term stability of signal encoding and decoding, or even failure, and can meet business needs. This application also provides a signal processing device, chip, brain-computer interface system, computing device cluster, computer-readable storage medium, and computer program product corresponding to the above-mentioned signal processing method.

[0008] In a first aspect, the present application provides a signal processing method. The method can be performed by a signal processing device. The signal processing device can be a device in a brain-computer interface system for processing signals collected from brain tissue. The signal processing device can be a software device, such as locally deployed software or cloud services. The software device can be deployed in a computing device cluster (including at least one computing device), and the computing device cluster executes the program code of the software device to execute the signal processing method of the present application. The signal processing device can also be a hardware device, such as a computing device cluster with signal processing capabilities. The signal processing method of the present application is executed when the hardware device is running.

[0009] Specifically, the signal processing device can obtain a first signal, which is a signal collected from brain tissue, and a downstream feedback signal. The signal processing device can then process the first signal based on the feedback signal. This allows the first signal to be processed in conjunction with the feedback signal, rather than directly inputting the first signal into a decoding device for decoding. This ensures the long-term stability of the signal encoding and decoding, avoids signal non-stationarity causing a decrease in the long-term stability of the signal encoding and decoding, or even failure, and can meet the needs of services (especially long-term services such as medical rehabilitation services).

[0010] In some possible implementations, the downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector. The feedback signal from the decoding device itself or the feedback signal from an effector controlled by the decoding device can be used to determine decoding accuracy. Signal processing based on this decoding accuracy, such as data integration, can ensure decoding stability.

[0011] In some possible implementations, the first signal may be a signal collected from brain tissue at a first time, and the feedback signal is used to represent feedback on a second signal, where the second signal is a signal collected from brain tissue at a second time, the second time being earlier than the first time. The first time and the second time may be moments or time periods (time series). This method can process the current signal through feedback from historical signals to ensure decoding stability.

[0012] In some possible implementations, the feedback signal includes a brain command or brain state at a second time obtained by decoding the second signal by a decoding device, or the feedback signal includes an action command or action state output by the effector in response to the brain command (brain command) or brain state (brain status) at the second time. A brain command refers to an instruction issued by brain tissue, which can usually be a control instruction, such as a brain instruction for controlling the movement of a prosthesis, and a brain state refers to an active state or an excited state of at least one area in the brain tissue. The action command or action state can be an instruction or state output by the effector in response to the brain command or brain state. Wherein, when the effector is an external application (external application), the above-mentioned action command / action state can also be an application command (application command) / application status (application status). For example, the application status can include posture.

[0013] The decoding accuracy can be determined by the above-mentioned brain instructions / brain states or action instructions / action states, and signal processing based on the decoding accuracy, such as data integration, can ensure decoding stability.

[0014] In some possible implementations, the signal processing device may determine, based on the feedback signal, that a trigger condition has been met and then process the first signal using a processing strategy corresponding to the trigger condition. The trigger condition may be a condition related to a monitoring indicator, which can be set as needed, such as decoding accuracy. By processing the first signal according to the processing strategy corresponding to the trigger condition when the trigger condition is met, the signal processing device can ensure decoding stability while avoiding the additional overhead of signal processing during stable decoding.

[0015] In some possible implementations, the processing strategy includes at least one of the following: calibration, coordinated control, data integration, stable channel selection, or paradigm update. When a trigger condition is met, the signal is processed using the corresponding processing strategy, enabling targeted processing and improving decoding accuracy and stability.

[0016] In some possible implementations, the feedback signal includes brain commands or brain states. The signal processing device can determine the decoding accuracy based on the brain commands or brain states in the feedback signal and the target brain commands or target brain states, and then update the decoding parameters for decoding the first signal based on the decoding accuracy. This can achieve signal calibration, and decoding stability can be improved by calibrating the signal. It should be noted that when calibrating the signal, either timed calibration or adaptive calibration can be used. Among them, adaptive calibration is triggered when the decoding accuracy is less than a set value, which can reduce unnecessary overhead.

[0017] In some possible implementations, the feedback signal includes at least one of a brain command or a brain state and at least one of a motion command or a motion state. Accordingly, the signal processing device can determine the user's first motion intention based on the brain command or the brain state, predict the user's second motion intention based on the motion command or the motion state, determine the ratio of the first motion intention to the second motion intention, and send the ratio to the decoding device.

[0018] This allows for coordinated control of motion intent derived from decoding and motion intent predicted by the algorithm, ensuring decoding stability. Coordinated control can be categorized as time-sharing coordinated control or adaptive coordinated control. Adaptive coordinated control supports manual intervention during algorithmic control and offers higher reliability.

[0019] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the first setting condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirements is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirements.

[0020] The first set condition may be that the decrease in decoding accuracy within a set time period is greater than a set amplitude, indicating a sudden and substantial decrease in decoding accuracy (also called a sag). Considering that a sag in decoding accuracy may be caused by electrode inactivation or changes in emission characteristics, the signal processing device determines the emission stability of multiple acquisition channels and sends information of acquisition channels whose emission stability meets the requirements to the decoding device, thereby improving decoding stability.

[0021] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the second set condition, the task parameters are updated, and the first signal modulated based on the updated task parameters is sent to the decoding device.

[0022] The second set condition may be that the decoding accuracy remains below an accuracy threshold for a set period of time, indicating that the decoding accuracy has been low for a period of time. The reason for the continued low level of decoding accuracy may be that the existing task is too challenging or difficult for the user. Based on this, the signal processing device can update the task parameters and send a first signal modulated based on the updated task parameters to the decoding device. This can reduce the task difficulty and thus ensure decoding stability.

[0023] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the third setting condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is pre-trained by training the decoding model in the decoding device based on historical data.

[0024] The third set condition can be that the decoding accuracy remains below the accuracy threshold for a set period of time, indicating that the decoding accuracy has been low for a period of time. The signal processing device may consider loading a pre-trained model and fine-tuning it based on a small amount of data from that day. Specifically, the signal processing device may use the first signal to fine-tune the pre-trained model, obtain updated decoding parameters, and send the updated decoding parameters to the decoding device. This allows for data integration and improves decoding stability.

[0025] In some possible implementations, the first signal is a digital signal, and the signal processing device can obtain the neural signal collected from the brain tissue, convert the neural signal into an analog signal, and then convert the analog signal into the digital signal through analog-to-digital conversion. This can lay the foundation for brain-computer interaction.

[0026] In some possible implementations, the signal processing device may preprocess the analog signal or the digital signal to achieve signal compression. Preprocessing the analog signal or the digital signal can improve signal quality and reduce signal data volume.

[0027] In some possible implementations, the signal processing device can pre-process the analog signal or the digital signal on the hardware signal acquisition device side. This can achieve that as brain-computer interface acquisition technology develops towards high throughput, high precision, high sampling rate and wireless transmission, analog and / or digital signal pre-processing can also be performed on the perception end, extracting functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reducing communication bandwidth and power consumption pressure, and realizing integrated sensing and computing.

[0028] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0029] This method also supports the independent generation of stimulation signals to stimulate brain tissue to generate neural signals, thereby realizing brain-computer interaction.

[0030] In some possible implementations, the prosthesis is a wearable device, such as smart glasses or a smart head display. The wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0031] By generating stimulation signals from the encoding results of the above-mentioned external input signals and stimulating the corresponding functional areas in the brain tissue, it can help the prosthesis user achieve perception. For example, it can help the blind perceive the position and size of obstacles or the status of traffic lights, thereby improving convenience.

[0032] In some possible implementations, the neural signal includes a signal generated by the brain tissue being stimulated by a stimulation signal output by a stimulation device. The stimulation signal is generated based on the encoding result of at least one modal internal input signal generated when a decoding device decodes a set brain state or brain instruction. The encoding result includes parameters of a stimulation pattern that matches the modality. This allows for brain-computer interaction that combines acquisition and stimulation, meeting the needs of corresponding business scenarios.

[0033] In a second aspect, the present application provides a signal processing device. The device includes:

[0034] an acquisition unit, configured to acquire a first signal, the first signal being a signal collected from brain tissue, and a downstream feedback signal;

[0035] A processing unit is configured to process the first signal according to the feedback signal.

[0036] In some possible implementations, the downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

[0037] In some possible implementations, the first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback on a second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

[0038] In some possible implementations, the feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

[0039] In some possible implementations, the processing unit is specifically configured to:

[0040] When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

[0041] In some possible implementations, the processing strategy includes at least one of the following:

[0042] Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

[0043] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit is specifically configured to:

[0044] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0045] A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

[0046] In some possible implementations, the feedback signal includes at least one of a brain command or a brain state and at least one of an action command or an action state, and the processing unit is specifically configured to:

[0047] determining a first movement intention of the user according to the brain instruction or brain state, and predicting a second movement intention of the user according to the action instruction or action state;

[0048] A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

[0049] In some possible implementations, the feedback signal includes a brain instruction or a brain state at the second time, and the processing unit is specifically configured to:

[0050] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0051] When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

[0052] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit is specifically configured to:

[0053] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0054] When the decoding accuracy meets the second set condition, the task parameters are updated, and the first signal modulated based on the updated task parameters is sent to the decoding device.

[0055] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit is specifically configured to:

[0056] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0057] When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

[0058] In some possible implementations, the first signal is a digital signal, and the acquiring unit is specifically configured to:

[0059] acquiring neural signals collected from the brain tissue;

[0060] converting the neural signal into an analog signal;

[0061] The analog signal is converted into the digital signal through analog-to-digital conversion.

[0062] In some possible implementations, the processing unit is further configured to:

[0063] The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

[0064] In some possible implementations, the signal preprocessing module is integrated or deployed in a hardware signal acquisition device, and the processing unit is specifically configured to:

[0065] The analog signal or the digital signal is preprocessed on the hardware signal acquisition device side through the signal preprocessing module.

[0066] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0067] In some possible implementations, the prosthesis is a wearable device, the wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0068] In some possible implementations, the neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0069] In a third aspect, the present application provides a chip. The chip includes a processor and a communication interface, the communication interface being configured to acquire a first signal, where the first signal is a signal collected from brain tissue, and the processor being configured to execute computer-readable instructions to perform the signal processing method described in the first aspect or any implementation of the first aspect.

[0070] In a fourth aspect, the present application provides a brain-computer interface system. The system includes a signal processing device and a hardware signal acquisition device, wherein the signal processing device is configured to execute the signal processing method described in the first aspect or any implementation of the first aspect to process the signal output by the hardware signal acquisition device to achieve brain-computer interaction.

[0071] In some possible implementations, the signal processing device includes a signal preprocessing module, which is integrated and deployed in the hardware signal acquisition device, and the signal preprocessing module is used to:

[0072] The signal output by the hardware signal acquisition device is preprocessed on the hardware signal acquisition device side.

[0073] In some possible implementations, the system further includes:

[0074] A stimulation device is used to generate a stimulation signal based on the encoding result of an external input signal of at least one modality collected by a prosthesis, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

[0075] In some possible implementations, the system further includes:

[0076] A stimulation device is used to generate a stimulation signal based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

[0077] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, wherein the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or computing device cluster performs the signal processing method described in the first aspect or any implementation of the first aspect.

[0078] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the signal processing method described in the first aspect or any one of the implementations of the first aspect.

[0079] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the signal processing method described in the first aspect or any one of the implementations of the first aspect.

[0080] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0082] FIG1 is a schematic diagram of the architecture of a brain-computer interface system provided in an embodiment of the present application;

[0083] FIG2 is a schematic diagram of the architecture of another brain-computer interface system provided in an embodiment of the present application;

[0084] FIG3 is a schematic diagram of the architecture of another brain-computer interface system provided in an embodiment of the present application;

[0085] FIG4 is a schematic diagram of a signal preprocessing process provided by an embodiment of the present application;

[0086] FIG5 is a hierarchical logic diagram of a storage system provided in an embodiment of the present application;

[0087] FIG6 is a schematic diagram of different brain-computer interaction modes provided in an embodiment of the present application;

[0088] FIG7 is a schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0089] FIG8 is a schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0090] FIG9 is a schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0091] FIG10 is a schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0092] FIG11 is a schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0093] FIG12 is a schematic diagram of another scenario of a signal processing method provided in an embodiment of the present application;

[0094] FIG13 is a schematic diagram of another scenario of a signal processing method provided in an embodiment of the present application;

[0095] FIG14 is a schematic structural diagram of a signal processing device provided in an embodiment of the present application;

[0096] FIG15 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0097] FIG16 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0099] First, some technical terms involved in the embodiments of this application are introduced.

[0100] The brain-computer interface (BCI) is an in-depth development of the human-computer interface (HCI) technology. Traditional human-computer interaction technology requires the use of media layer hardware and software, such as computer display screens, visual interfaces, mice, etc. The brain-computer interface is a new type of human-computer interaction based on neuroscience and engineering technology. It can acquire and analyze brain signals in real time, establish a direct communication and control channel between brain tissue (such as the human brain, animal brain) and computers or other electronic devices, and directly convert central nervous system activities into information output, thereby achieving an understanding of brain consciousness. It can also intervene and even participate in brain activities through bioengineering methods such as optogenetics and brain electrodes. Users no longer need to express their ideas or manipulate external devices through language or body movements.

[0101] Brain-computer interface technology is divided into two major directions: invasive and non-invasive according to the degree of implantation of the hardware signal acquisition device (also called hardware signal acquisition module).

[0102] A non-invasive brain-computer interface refers to a hardware signal acquisition device installed on the scalp and other areas, which can be worn externally. The non-invasive brain-computer interface mainly collects scalp electroencephalogram (EEG) signals. EEG is an electrical signal of multiple local field potentials that pass through the cortex. Non-invasive brain-computer interfaces can be used in scenarios such as physiological state assessment and regulation (attention, sleep, stress detection), auxiliary human-computer interaction (disability assistance, augmented reality / virtual reality), and sports and health.

[0103] An invasive brain-computer interface refers to a hardware signal acquisition device installed in the scalp, which can usually be implanted through surgery. The invasive brain-computer interface mainly collects pulse (spike) signals, local field potential (LFP), and electrocorticography (ECoG). Among them, the local field potential represents the sum of the electrical signals of neurons in the local area of ​​the electrode. The electrocorticography is similar to the local field potential, but can only record electrical signals on the surface of the cortex. The invasive brain-computer interface collects high-quality signals and can be used in medical rehabilitation scenarios.

[0104] The following is an example of a brain-computer interface system, combining the above explanations of brain-computer interface concepts. A brain-computer interface system processes signals collected from brain tissue and can therefore also be referred to as a signal processing system or an EEG signal processing system.

[0105] As shown in Figure 1, the brain-computer interface system 10 includes a hardware signal acquisition device 100 (also known as a hardware signal acquisition module), a signal processing device 200 (or a signal processing module), a decoding device 300 (or a decoding module), an effector 400, and a feedback device 500 (or a feedback module). After performing signal processing, the above-mentioned signal processing device 200 can also store the processed signal for subsequent use. Based on this, the signal processing device 200 can be a signal processing and storage device. Similarly, the decoding device 300 is used to perform feature extraction on the processed signal and then decode the extracted features. Therefore, the decoding device 300 can be a feature extraction and decoding device.

[0106] The hardware signal acquisition device 100 is connected to the signal processing device 200, which is connected to the decoding device 300, which is connected to the effector 400. The effector 400 can act directly on the brain tissue or be connected to the feedback device 500 to act on the brain tissue through the feedback device 500.

[0107] The functions, inputs, outputs, etc. of the hardware signal acquisition device 100 , signal processing device 200 , decoding device 300 , effector 400 , and feedback device 500 are described in detail below.

[0108] The hardware signal acquisition device 100 is used to collect signals from brain tissue. The hardware signal acquisition device 100 can be set on the scalp and areas outside the scalp to collect signals in a non-invasive manner. The hardware signal acquisition device 100 can also be implanted in the scalp to collect signals in an invasive manner. As shown in Table 1, the input of the hardware signal acquisition device 100 can be a neural signal from brain tissue, and the output of the hardware signal acquisition device 100 can be a digital signal. The hardware signal acquisition device 100 converts the neural signal into an electrical signal, such as an analog electrical signal (also called an analog signal), through electrodes and circuits, and converts the electrical signal into a processable digital electrical signal (also called a digital signal). It should be noted that the above-mentioned analog signals and digital signals can be collectively referred to as electroencephalogram signals.

[0109] Table 1

[0110] The signal processing device 200 is used to process the digital signal output by the hardware signal acquisition device 100, thereby improving the signal quality. The signal processing device 200 also has a storage function for storing the processed digital signal for subsequent use. The input of the signal processing device 200 can be the digital signal output by the signal acquisition device 100, and the output of the signal processing device 200 can be the processed digital signal, which is typically a high-quality digital signal.

[0111] The decoding device 300 is used to decode the signal processed by the signal processing device 200, thereby obtaining brain instructions or brain states, thereby realizing the perception of brain consciousness. The input of the decoding device 300 can be the processed digital signal output by the signal processing device 200, and the output of the decoding device can be brain instructions or brain states. Among them, brain instructions refer to instructions issued by brain tissue, which can generally be control instructions, such as brain instructions to control the movement of a prosthesis. Brain state refers to the active state or excited state of at least one area of ​​brain tissue. Different areas of brain tissue can correspond to different functions. For example, the cerebellum is responsible for regulating and controlling fine movements, posture and balance, while the cerebral cortex is generally associated with higher brain functions, such as conscious thinking, action selection and control. In specific implementation, the decoding device 300 can extract features from the processed signal and then decode based on the features to obtain brain instructions or brain states. It should be noted that the features extracted by the decoding device 300 can be different depending on the signal type.

[0112] The effector 400 is used to respond to brain instructions or brain states and perform actions corresponding to the brain instructions or brain states. The effector 400, as an object controlled by the decoding device 300, can generally be at least one of an external device or a stimulation device. Among them, external devices are sometimes also called peripherals or external applications, including but not limited to computer cursors, manipulators (mechanical arms), mechanical skeletons, wheelchairs, and monitoring and security equipment. The stimulation device can also be called a stimulation device, and can include the following types: muscle stimulation devices and neuromodulation devices. The muscle stimulation device can be functional electrical stimulation (FES). FES can apply low-frequency pulse current or send it into the human body after amplification through signal-current conversion to produce immediate effects, artificially causing a person paralyzed due to damage to the central nervous system to produce movements. Neuromodulation devices can include but are not limited to deep brain stimulation (DBS), spinal cord stimulation (SCS), and peripheral nerve stimulation (PNS). The input of the effector 400 may be a brain command or a brain state, and the output of the effector 400 may be an action command or an action state.

[0113] Feedback device 500 is used to provide system feedback to the brain, establishing a closed loop between the user's brain actions / states and system responses. The input to feedback device 500 can be the action instructions or action states output by effector 400, and the output of feedback device 500 can be feedback on the execution results (i.e., system feedback).

[0114] It should be noted that the feedback device 500 is an optional device. For example, the effector 400 can also be fed back directly to the brain tissue in the form of visual feedback without going through the feedback device 500. Taking the effector 400 as an example of a computer cursor, the movement of the computer cursor can be fed back to the brain tissue in the form of visual feedback. In some cases, the feedback device 500 can serve as a supplement to visual feedback. For example, a robotic arm controlled by a brain-computer interface can usually enable patients with quadriplegia to complete functional movements, but the visual feedback provided to patients is limited. Grasping movements are often faster, more accurate and more realistic, as signals are transmitted through the feedback device 500 in the form of tactile feedback.

[0115] However, the signals collected by these invasive or non-invasive BCIs are significantly affected by factors such as electrode inactivation, random noise, motion artifacts (interference caused by slight movement of the subject during BCI, such as head shaking), and their physical and mental state, resulting in non-stationarity. This can lead to a decrease in the long-term stability of signal encoding and decoding, or even failure, making it difficult to meet business needs.

[0116] In view of this, the present application provides a signal processing method. The method can be applied to the brain-computer interface system 10 shown in Figure 2. Different from the brain-computer interface system 10 shown in Figure 1, in the brain-computer interface system 10 shown in Figure 2, the downstream of the signal processing device 200 is provided with a feedback path to the signal processing device 200, wherein the feedback path can be a feedback path between the downstream effector 400 and the signal processing device 200, or a feedback path between the downstream decoding device 300 and the signal processing device 200, and the above feedback path is used to transmit the feedback signal of the effector 400 or the decoding device 300 to the signal processing device 200. Accordingly, the signal processing device 200 can process the signal (also referred to as the first signal) collected from the brain tissue based on the feedback signal, instead of directly inputting the signal into the decoding device 300 for decoding, thereby ensuring the long-term stability of the signal encoding and decoding, avoiding the non-stationarity of the signal causing the long-term stability of the signal encoding and decoding to decline or even fail, and can meet business needs.

[0117] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the brain-computer interface system 10 provided in the present application is introduced below with reference to the accompanying drawings.

[0118] Referring to the schematic diagram of the architecture of a brain-computer interface system 10 shown in FIG2 , the brain-computer interface system 10 includes a hardware signal acquisition device 100, a signal processing device 200, a decoding device 300, an effector 400, and a feedback device 500. The hardware signal acquisition device 100 is connected to the signal processing device 200, the signal processing device 200 is connected to the decoding device 300, and the decoding device 300 is connected to the effector 400. The effector 400 can act directly on the brain tissue, or it can be connected to the feedback device 500 and act on the brain tissue through the feedback device 500. In addition, a feedback path to the signal processing device 200 is also provided downstream of the signal processing device 200. FIG2 illustrates an example in which a feedback path is provided between the decoding device 300 and the signal processing device 200. The feedback path is used to provide a feedback signal to the signal processing device 200 to assist the signal processing device 200 in signal processing.

[0119] The following is an introduction to the functions, inputs, and outputs of each device / module in the brain-computer interface system 10 shown in Figure 2 in conjunction with Table 2.

[0120] Table 2

[0121] The functions, inputs, and outputs of the hardware signal acquisition device 100, effector 400, and feedback device 500 can be described with reference to the relevant contents of Table 1 or Figure 1. The input of the signal processing device 200, in addition to Table 1, also includes a feedback signal from the decoding device 300. This feedback signal can be a brain command or brain state. Accordingly, the signal processing device 200 is configured to process the first signal based on the feedback signal, thereby improving decoding stability. The first signal can be a signal collected from brain tissue at a first time (e.g., the current time or current moment), and the second signal can be a signal collected from brain tissue at a second time (e.g., a historical time or historical moment). The feedback signal is used to represent the feedback of the decoding device 300 on the second signal. In other words, the signal processing device 200 can process the signal at the current time in combination with the brain command or brain state at a historical time (e.g., the previous moment), including selecting / adjusting the signal postprocessing process, to improve decoding stability. The output of the decoding device 300 is added to the output of the signal processing device 200 on the basis of Table 1, specifically outputting brain instructions / brain states to the signal processing device 200, so that the signal processing device 200 performs signal processing in combination with feedback signals such as brain instructions / brain states.

[0122] When the feedback path is provided between the effector 400 and the signal processing device 200, the feedback signal may be an action command or an action state. The signal processing device 200 may process the signal in combination with the action command or action state of the previous moment, including selecting / adjusting the signal post-processing process to improve decoding stability. The output of the effector 400 is added to the output of the signal processing device 200 based on Table 1, specifically outputting the action command / action state to the signal processing device 200, so that the signal processing device 200 performs signal processing in combination with the feedback signal such as the action command / action state.

[0123] The signal processing device 200 is further configured to store the trigger condition and the processing strategy corresponding to the trigger condition. Accordingly, when the signal processing device 200 determines that the trigger condition is satisfied according to the feedback signal, it processes the first signal using the processing strategy corresponding to the trigger condition.

[0124] The trigger condition may be a condition related to a monitoring indicator. The monitoring indicator may be set according to business needs, for example, the monitoring indicator may include decoding accuracy. The decoding accuracy may be determined based on the brain instructions or brain state obtained by decoding at the second time, as well as the target brain instructions and target brain state.

[0125] The processing strategy may include at least one of calibration, collaborative control, data integration, stable delivery channel screening, or paradigm update. Calibration may include timed calibration or adaptive calibration. Similar to calibration, collaborative control may also include time-sharing collaborative control or adaptive collaborative control. The specific implementation of calibration, collaborative control, data integration, stable delivery channel screening, or paradigm update is described in detail below.

[0126] In the system shown in FIG2 , the first signal processed by the signal processing device 200 may be a digital signal. This digital signal may be acquired and processed by the hardware signal acquisition device 100. Specifically, the hardware signal acquisition device 100 is used to acquire neural signals from brain tissue, convert the neural signals into model signals, and then convert the analog signals into digital signals through analog-to-digital conversion, such as an analog-to-digital converter (ADC). The signal processing device 200 obtains the digital signal from the hardware signal acquisition device 100 and performs signal processing.

[0127] In some possible implementations, referring to FIG3 , a signal processing device 200 may include a signal preprocessing module 202 and a signal postprocessing module 204. The signal preprocessing module 202 and the signal postprocessing module 204 may transmit signals via a transmission module 206. The transmission module 206 may be a module that implements signal transmission based on wired or wireless technologies. Wireless technologies may include, but are not limited to, Bluetooth or wireless networks (e.g., Wi-Fi).

[0128] The signal preprocessing module 202 is used to preprocess the signals collected by the hardware signal acquisition device 100. This preprocessing can include at least one of analog signal preprocessing or digital signal preprocessing. The signal preprocessing module 202 can be integrated or deployed in the hardware signal acquisition device 100, thereby preprocessing the signal at the sensing end (e.g., the hardware signal acquisition device 100) to achieve signal compression.

[0129] As shown in FIG4 , the neural signals collected from brain tissue can be converted into electrical signals that can be processed by a computing device by an analog front-end (AFE) chip in the hardware signal acquisition device 100. The AFE can integrate an ADC, so the processable electrical signals converted by the AFE can be digital signals. The digital signals can be preprocessed by at least one of the following modules in the signal preprocessing module 202: filtering, signal-to-noise ratio enhancement (SNR Boost), adaptive threshold calculation, spike detection, false positive detection (FP Reduction), and signal compression (such as spike compression). False positive detection refers to detecting noise that is identified as a peak, and false positive detection can be achieved through waveform detection. The signal preprocessing module 202 can be integrated or deployed in the hardware signal acquisition device 100 through an on-chip implementation. The input of the signal preprocessing module 202 can be a voltage signal converted by the ADC, and the output can include but is not limited to pulse waveforms, pulse emission time points, pulse emission frequencies in a specific time window, specific spectral signals, and other modalities.

[0130] The preprocessed digital signal can be sent to the transmission module 206 according to the corresponding transmission protocol. The transmission module 206 transmits the preprocessed digital signal to the signal post-processing module 204 based on the corresponding transmission protocol. It should be noted that Figure 4 is only an example of the preprocessing process of the digital signal. Analog signals can also be preprocessed. For example, the analog signal can enter an amplifier or filter for preprocessing.

[0131] In this way, while brain-computer interface acquisition technology develops towards high throughput, high precision, high sampling rate and wireless transmission, it can also perform analog signal and / or digital signal preprocessing at the perception end, extract functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reduce communication bandwidth and power consumption pressure, and realize integrated sensing and computing.

[0132] In some possible implementations, the signal processing device 200 may further include a storage module 208, also referred to as a storage system. The pre-processed signal may be transmitted to the storage module 208 via the transmission module 206 for storage for subsequent use. It should be noted that the digital signal converted by the ADC may also be sent directly to the transmission module 206 without pre-processing. The transmission module 206 may transmit the digital signal to the signal post-processing module 204, which may also transmit the digital signal to the storage module 208 for subsequent processing. For example, the signal post-processing module 204 may read the stored digital signal from the storage module 208 for signal processing (e.g., post-processing).

[0133] The signal post-processing module 204 is used to process the first signal according to the feedback signal (for example, the brain instruction or brain state fed back by the decoding device 300, or the action instruction or action state fed back by the effector 400, FIG3 is illustrated by the feedback signal of the decoding device 300). As shown in FIG3, a feedback path is provided between the decoding device 300 and the signal post-processing module 204 of the signal processing device 200, and the feedback path is used to transmit the feedback signal of the decoding device 300 to the signal post-processing module 204 of the signal processing device 200. Accordingly, the signal post-processing module 204 can process the first signal based on the brain instruction or brain state (such as visual information, posture information) fed back by the decoding device 300, for example, performing timing or adaptive calibration, time-sharing or adaptive collaborative control, data integration, stable release channel screening, paradigm optimization, etc., to improve the long-term stability of decoding.

[0134] The above describes an example of collecting signals from brain tissue, processing and decoding the signals, and controlling the effector 400. In some possible implementations, the effector 400 may include a stimulation device that generates stimulation signals to stimulate the brain tissue to generate neural signals.

[0135] In some possible implementations, the stimulation device can independently generate stimulation signals. Specifically, the stimulation device can obtain an external input signal of at least one modality collected by a prosthesis (such as a visual prosthesis or an auditory prosthesis), and encode the external input signal of at least one modality to obtain an encoding result, where the encoding result includes parameters of a stimulation pattern that matches the modality. The stimulation device can generate stimulation signals based on the encoding result.

[0136] Among them, the prosthesis can be a wearable device, including but not limited to smart glasses and smart head-mounted display devices. The wearable device can deploy a sensor of at least one modality, and the wearable device is used to collect an external input signal of at least one modality through the sensor of the at least one modality. Taking the prosthesis as smart glasses as an example, the smart glasses can deploy an image sensor (such as a camera), which can collect environmental images and identify the environmental images through an image recognition model to obtain obstacle information or traffic light status information. The stimulation device can obtain the above-mentioned obstacle information or traffic light status information, and encode the obstacle information or traffic light status information to obtain a coding result. Then, based on the coding result, a stimulation signal is generated to stimulate the area of ​​​​the brain tissue responsible for vision, so that the blind person wearing the smart glasses can perceive the position and size of the obstacle or the status of the traffic light.

[0137] In some other possible implementations, the stimulation device can generate a stimulation signal in combination with the collected information. Specifically, when the decoding device decodes the set brain state or brain instruction, it can generate an internal input signal of at least one modality. For example, the brain state may be a state in which the user is about to have an epilepsy, and the decoding device can generate an internal input signal that suppresses epilepsy. Accordingly, the stimulation device can obtain the internal input signal of at least one modality output by the decoding device, encode the internal input signal of at least one modality to obtain an encoding result, and the encoding result includes the parameters of the stimulation mode that matches the modality. The stimulation device can generate a stimulation signal based on the encoding result. The stimulation signal can stimulate the area in the brain tissue responsible for movement, thereby suppressing the user from having epilepsy.

[0138] It should be noted that the aforementioned brain-computer interface system 10 can implement brain-computer interaction through different modes. As shown in Figure 5, the brain-computer interface system 10 can implement brain-computer interaction through active mode, passive mode, or interactive mode. In active mode, the brain-computer interface system 10 can control peripherals (such as a robotic arm) through intentional thoughts. In passive mode, the brain-computer interface system 10 can receive specific sensory stimulation from the user and collect corresponding EEG signals (such as the first signal) to control the peripherals. In interactive mode, the brain-computer interface system 10 can trigger relevant EEG signals to control peripherals based on the occurrence of specific sensory, cognitive, or motor events in the external environment. Whether in active mode, passive mode, or interactive mode, the signal processing device 200 in the brain-computer interface system 10 can receive feedback signals, such as feedback signals (brain commands or brain states) from the decoding device 300, to determine decoding accuracy, thereby determining whether the decoding is accurate or stable. When the decoding is inaccurate or unstable, for example, the degree of decrease in decoding accuracy or stability exceeds a set threshold, the signal post-processing module 204 of the signal processing device 200 can be triggered to post-process the first signal to improve decoding accuracy or decoding stability.

[0139] Based on the aforementioned brain-computer interface system 10, the present application provides a signal processing method. The signal processing method of the present application is described below in conjunction with embodiments.

[0140] 6 shows a flow chart of a signal processing method. The method may be performed by the signal processing device 200 in the brain-computer interface system 10. The method includes the following steps:

[0141] S602: The signal processing device 200 obtains a first signal.

[0142] The first signal is a signal collected from brain tissue. The first signal can be a digital signal. For example, the first signal can be a neural signal collected from brain tissue by the hardware signal acquisition device 100. The neural signal can be converted into an analog signal and then converted into a processable digital signal by an ADC.

[0143] In some possible implementations, the first signal may also be a preprocessed signal, and the signal quality may be improved by preprocessing. The preprocessing may include analog signal preprocessing and / or digital signal preprocessing. The analog signal preprocessing may include at least one of amplification and filtering (analog filtering). The digital signal preprocessing may include at least one of filtering (digital filtering), signal-to-noise ratio enhancement, adaptive threshold calculation, pulse detection waveform detection (such as FP Reduction), and signal compression (such as Spike Compression).

[0144] The specific implementation of digital signal preprocessing can be seen in Figure 4. The digital signal output by the AFE through the AFC can be filtered and enhanced to improve the signal-to-noise ratio. When adaptive threshold calculation is required, the filtered signal (if not filtered, the raw data after ADC conversion) can be divided into two paths: one for the pulse detection module and the other for the adaptive threshold calculation module. The threshold is determined based on the two outputs. After the threshold is determined, pulse detection can be performed based on this threshold. For pulse signals, waveform detection or pulse compression can be performed. This achieves the highest possible compression rate while maintaining high precision and low power consumption.

[0145] It should be noted that the above-mentioned preprocessing process can be performed on the hardware signal acquisition device 100 side. As shown in Figure 3, the signal processing device 200 may include a signal preprocessing module 202 and a signal post-processing module 204. The signal preprocessing module 202 can be integrated or deployed in the hardware signal acquisition device 100, and the signal processing device 200 can preprocess the signal through the signal preprocessing module 202 at the perception end (such as the hardware signal acquisition device 100) to achieve signal compression. Even if the hardware signal acquisition device 100 performs signal acquisition based on high throughput, high precision, and high sampling rate, or adopts wireless transmission, it can also perform analog signal and / or digital signal preprocessing at the perception end, extract functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reduce communication bandwidth and power consumption pressure, and realize sensing and computing integration.

[0146] S604: The signal processing device 200 obtains a downstream feedback signal.

[0147] The downstream feedback signal refers to a signal fed back by a downstream device (downstream equipment) of the signal processing apparatus 200. The downstream feedback signal may include a feedback signal from the decoding apparatus 300 or a feedback signal from the effector 400. The feedback signal is used to represent feedback on a historical signal preceding the first signal.

[0148] In some possible implementations, the first signal may be a signal collected from brain tissue at a first time, and the second signal may be a signal collected from brain tissue at a second time, and the second time is earlier than the first time. Accordingly, the feedback signal is used to characterize the feedback of the second signal. Based on this, the feedback signal may include the brain instruction or brain state at the second time obtained by the decoding device 300 decoding the second signal, or the feedback signal may include the action instruction or action state output by the effector 400 in response to the brain instruction or brain state at the second time. Among them, the action state may be visual information or posture information. Depending on the application scenario (such as different control peripherals), the action instruction and action state may be different.

[0149] It should be noted that the first time and the second time may be moments, or time sequences (time periods, control cycles), which are not limited in the present embodiment.

[0150] S606: The signal processing device 200 processes the first signal according to the feedback signal.

[0151] Specifically, the signal processing device 200 can process the first signal at a regular interval. Alternatively, the signal processing device 200 can also process the first signal using a processing strategy corresponding to the trigger condition when determining that a trigger condition is satisfied based on a feedback signal. The trigger condition can be a condition related to a monitoring indicator. The monitoring indicator can be set according to business needs. For example, the monitoring indicator can include decoding accuracy. The decoding accuracy can be determined based on the brain instructions or brain state at the second time obtained by decoding, as well as the target brain instructions and target brain state.

[0152] For example, when the decoding accuracy drops sharply, it may be due to electrode inactivation or changes in the release characteristics. The signal processing device 200 can reselect an acquisition channel whose release stability meets the requirements. For another example, when the decoding accuracy remains at a low level for a period of time, it may be considered that the existing task is too challenging / difficult for the user. The signal processing device 200 can redefine the task parameters and adjust the difficulty of the task. Based on this, the trigger condition may include that the decoding accuracy meets the set conditions, and the set conditions may include multiple conditions. Different trigger conditions may correspond to different processing strategies. Processing strategies may include calibration, collaborative control, data integration, stable release channel screening, or paradigm update (specifically updating task parameters).

[0153] The signal processing device 200 may store the correspondence between the trigger conditions and the processing strategies. When the trigger condition is satisfied, the signal processing device 200 may determine the processing strategy corresponding to the currently satisfied trigger condition based on the correspondence between the trigger and the processing strategy.

[0154] Based on the above description, it can be seen that the signal processing method of the present application can process the signal (such as the first signal) collected from the brain tissue based on the downstream feedback signal (such as the feedback signal of the decoding device 300), instead of directly inputting the signal into the decoding device 300 for decoding, thereby ensuring the long-term stability of the signal encoding and decoding, avoiding the long-term stability of the signal encoding and decoding caused by the non-stationarity of the signal, and being able to meet business needs.

[0155] Next, the signal processing method of the present application is illustrated with examples in combination with different application scenarios.

[0156] First, refer to a scenario diagram of a signal processing method shown in Figure 7. As shown in Figure 7, the post-processing module 204 in the signal processing device 200 may include a calibration module, such as a regular calibration module or an adaptive calibration module.

[0157] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain commands or brain states decoded by the decoding device 300 can be fed back to the calibration module. The following describes the process by which the calibration module performs timed calibration or adaptive calibration.

[0158] When the calibration module adopts timed calibration, it can obtain the brain instructions or brain state fed back by the decoding device 300 at a set frequency (or time interval), and determine the decoding accuracy (or error) based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions (such as the target brain state or brain instructions stored in the storage module 208). The calibration module can then update the decoding parameters (such as the parameters of the decoding model) for decoding the first signal according to the decoding accuracy. In specific implementation, the calibration module can estimate the decoding parameters according to the decoding accuracy, and send the estimated decoding parameters (estimated decoding parameters) as updated decoding parameters to the decoding device 300, thereby updating the decoding parameters.

[0159] Furthermore, the calibration module can adjust the frequency or interval of regular calibration based on decoding accuracy. For example, if decoding accuracy remains above a set value for a set period of time, the calibration module can reduce the frequency of regular calibration or increase the interval between them. For example, if decoding accuracy remains above 0.95 for a sustained period, the calibration module can increase the regular calibration interval from once a day to once a week.

[0160] When the calibration module adopts adaptive calibration, it can obtain the brain instructions or brain state fed back by the decoding device 300, and determine the decoding accuracy (or error) based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions. When the decoding accuracy is less than a set value (or the error is greater than another set value), the calibration module can update the decoding parameters for decoding the first signal according to the decoding accuracy (or error). The calibration module can determine the updated decoding parameters in an iterative manner until the decoding accuracy is greater than the set value (or the error is reduced to another set value).

[0161] It should be noted that the first signal may be a signal collected from brain tissue at a first time, and the second signal may be a signal collected from brain tissue at a second time, the second time being earlier than the first time. Accordingly, the brain command or brain state fed back by the decoding device 300 may be the brain command or brain state at the second time.

[0162] Secondly, referring to a scenario diagram of a signal processing method shown in FIG8 , as shown in FIG8 , the post-processing module 204 in the signal processing device 200 may include a collaborative control module, also referred to as a shared control module, or a shared control parameter calculation module.

[0163] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 and the action instructions or action states output by the effector 400 (such as a peripheral) can be fed back to the collaborative control module. The following describes the process by which the collaborative control module performs collaborative control.

[0164] Specifically, the collaborative control module can obtain the brain instructions or brain states fed back by the decoding device 300 to determine the user's movement intention. Specifically, the collaborative control module can perform motion planning based on the brain instructions or brain states to determine the user's movement intention. For ease of distinction, this application will record the movement intention determined by motion planning based on the decoded brain instructions or brain states as the first movement intention. The collaborative control module can also obtain the action instruction action state fed back by the effector 400 (such as a peripheral device such as a robotic arm). In some examples, the action state may include a posture. The collaborative control module can predict the user's movement intention based on the action instruction or action state. Specifically, the collaborative control module can combine the movement instruction or movement state of the current control cycle with the movement instruction or movement state of the historical control cycle stored in the storage module, and predict the user's movement intention based on an algorithm (such as a Kalman filter, an AI model). Among them, the movement intention predicted based on the movement instruction or movement state is also called the second movement intention. The collaborative control module can determine the ratio of the first movement intention to the second movement intention and send the ratio to the decoding device 300.

[0165] The collaborative control module may compare the difference between the first motion intention and the second motion intention, determine the accuracy of the decoded user's motion intention, and adjust the ratio of the first motion intention to the second motion intention accordingly based on the accuracy.

[0166] When the collaborative control module adopts the time-sharing collaborative control mode, the collaborative control module can decide one motion intention as the final motion intention. In other words, the ratio of the first motion intention and the second motion intention is 100%:0% or 0%:100%. For example, if the difference between the first motion intention and the second motion intention is less than the preset threshold, it means that the accuracy is high, and the user's motion intention determined by the decoding method is reliable. The first motion intention determined by the decoding method can be selected, that is, the ratio of the first motion intention and the second motion intention is 100%:0%. If the difference between the first motion intention and the second motion intention is greater than the preset threshold, it means that the accuracy is low, and the user's motion intention determined by the decoding method is less reliable. The predicted second motion intention can be selected, that is, the ratio of the first motion intention and the second motion intention is 0%:100%. The coordinated control module sends the above ratio to the decoding device 300.

[0167] When the collaborative control module adopts the adaptive collaborative control mode, the collaborative control module can use multiple motion intentions to obtain the final motion intention. In other words, the ratio of the first motion intention to the second motion intention is x%: (100-x)%, where x is (0, 100). Specifically, the collaborative control module can obtain Nash equilibrium through a dynamic decision tree based on the paths corresponding to different motion intentions. For example, the collaborative control module can calculate the benefits brought by each decision (such as each ratio) based on the cost-benefit function, and calculate the Nash equilibrium point of the dynamic decision tree based on the benefits. The ratio corresponding to the Nash equilibrium point can be used as the ratio of the first motion intention to the second motion intention. The collaborative control module can send the ratio to the decoding device 300 so that the decoding device 300 can decode according to the ratio and control the peripherals based on the decoding results.

[0168] It should be noted that, when performing collaborative control, the collaborative control module may obtain the motion intention based on the feedback signal of the current control cycle, or may obtain the motion intention based on the feedback signal of the historical control cycle, and this embodiment does not impose any limitation on this.

[0169] Next, referring to a scenario diagram of a signal processing method shown in FIG9 , as shown in FIG9 , the post-processing module 204 in the signal processing device 200 may include a stable channel selection (also referred to as a stable channel selection, Stable Channel Selection) module.

[0170] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain commands or brain states decoded by the decoding device 300 can be fed back to the stable channel selection module. The following describes the process by which the stable channel selection module performs stable channel selection (or stable channel screening).

[0171] In specific implementation, the stable release channel selection module can obtain the brain instructions or brain state fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions (such as the target brain state or brain instructions stored in the storage module 208).

[0172] When the decoding accuracy meets the first set condition, the first set condition may be that the decrease in the decoding accuracy in the set time period is greater than the set amplitude, indicating that there is a sudden and substantial decrease in the decoding accuracy (also called a sudden drop). Considering that the sudden drop in decoding accuracy may be caused by electrode inactivation or changes in the emission characteristics, the stable emission channel selection module can determine the emission stability of multiple acquisition channels and send information of the acquisition channels whose emission stability meets the requirements (such as Filtered Stable Signals) to the decoding device 300, so that the decoding device 300 decodes the first signal according to the acquisition channels whose emission stability meets the requirements.

[0173] The stable delivery channel selection module can evaluate all acquisition channels based on a set delivery stability judgment standard to determine an acquisition channel that meets the required stability, i.e., a stable delivery channel. The stable delivery channel selection module selects a stable delivery channel and transmits information about the selected stable delivery channel (e.g., a channel identifier) ​​to the decoding device 300.

[0174] Then, referring to a scenario diagram of a signal processing method shown in FIG10 , as shown in FIG10 , the post-processing module 204 in the signal processing device 200 may include a task parameter update module, also called a paradigm update module.

[0175] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 can be fed back to the task parameter update module. The following describes the process by which the task parameter update module performs task parameter updates (or paradigm updates, or paradigm optimizations).

[0176] Specifically, the task parameter update module can obtain the brain instructions or brain state fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions (such as the target brain state or brain instructions stored in the storage module 208).

[0177] When the decoding accuracy meets the second set condition, the second set condition may be that the decoding accuracy is continuously lower than the accuracy threshold for a set time period, indicating that the decoding accuracy is continuously low for a period of time. The reason why the decoding accuracy continues to remain at a low level may be that the existing task is too challenging / difficult for the user. Based on this, the task parameter update module can update the task parameters and send a first signal modulated based on the updated task parameters (also called Task Parameter-Modulated Signals) to the decoding device 300. The purpose of adjusting the task parameters is to adjust the difficulty of the task. Taking the control of a robotic arm as an example, adjusting the task parameters can be to adjust the movement accuracy of the robotic arm, for example, from 1 millimeter (millimeter, mm) to 1 centimeter (centimeter, cm).

[0178] Next, referring to a scenario diagram of a signal processing method shown in FIG11 , as shown in FIG11 , the post-processing module 204 in the signal processing device 200 may include a data integration module, also referred to as a pretrained model loading module.

[0179] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain commands or brain states decoded by the decoding device 300 can be fed back to the data integration module. The following describes the data integration (pre-training model loading and fine-tuning) process performed by the data integration module.

[0180] Specifically, in an offline state, in order to improve the generalization of decoding, the data integration module may consider integrating data across days, across users (subjects), and multiple modalities to pre-train the decoding model, and retain the parameters of the pre-trained model. The parameters may include the weights of the pre-trained model. In some examples, the parameters may also include hyperparameters of the pre-trained model, such as optimizers and learning rates. During online decoding, the data integration module may obtain the brain instructions or brain states fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain states fed back by the decoding device 300 and the locally stored target brain states or target brain instructions (such as the target brain states or brain instructions stored in the storage module 208).

[0181] When the decoding accuracy meets the third set condition, the third set condition may be that the decoding accuracy is continuously lower than the accuracy threshold within a set time period, indicating that the decoding accuracy is continuously low for a period of time, and the data integration module may consider loading the pre-trained model and fine-tuning it based on a small amount of data on that day. Specifically, the data integration module may use the first signal to fine-tune the pre-trained model, obtain updated decoding parameters, and send the updated decoding parameters to the decoding device 300. The pre-trained model is obtained by pre-training the decoding model in the decoding device 300 based on historical data (such as cross-day, cross-user, and multi-modal data).

[0182] Based on the aforementioned brain-computer interface system 10, the present application further provides another signal processing method. The following describes another signal processing method of the present application in conjunction with an embodiment.

[0183] FIG12 shows a flow chart of a signal processing method. The method may be performed by the effector 400 in the brain-computer interface system 10. The effector 400 includes a stimulation device. The method includes the following steps:

[0184] S1202: The stimulation device obtains an external input signal of at least one modality collected by the prosthesis.

[0185] A prosthesis, also known as an artificial body part or prosthetic implant, is a man-made device used to replace a body part lost due to trauma, disease, or congenital impairment. Prostheses can be categorized into different types based on the body part they replace, including but not limited to visual prostheses, auditory prostheses, lower limb prostheses, and upper limb prostheses. Prostheses can also be wearable devices; for example, visual prostheses can include, but are not limited to, smart glasses.

[0186] The wearable device can be deployed with a sensor of at least one modality, and the modality may include but is not limited to a visual modality, an auditory modality, or a tactile modality. The wearable device is used to collect an external input signal of at least one modality through a sensor of at least one modality. Taking the prosthesis as an example of smart glasses, the smart glasses can be deployed with an image sensor (such as a camera), which can collect environmental images and identify the environmental images through an image recognition model to obtain obstacle information or traffic light status information. The stimulation device can obtain the above-mentioned obstacle information or traffic light status information. Based on this, the external input signal of at least one modality may include obstacle information or traffic light status information. Among them, the obstacle information includes obstacle size, obstacle location, and obstacle type (fixed or movable). When the obstacle type is movable, the obstacle information may also include moving speed and moving direction. Traffic light status information may include traffic light color and traffic light switching waiting time.

[0187] S1204: The stimulation device encodes an external input signal of at least one modality to obtain an encoding result.

[0188] The encoding result includes the parameters of the stimulation mode that matches the modality. The stimulation mode includes but is not limited to a light stimulation mode, an electrical stimulation mode, and an ultrasonic stimulation mode. The stimulation device can store the correspondence between the modality and the stimulation mode. When an external input signal of at least one modality is obtained, the stimulation mode that matches the modality can be determined based on the above correspondence. For example, when the input signal is a visual modality, the stimulation mode can be an electrical stimulation mode. The stimulation device can encode the external input signal according to the corresponding stimulation mode through an encoder to obtain an encoding result.

[0189] S1206. The stimulation device generates a stimulation signal based on the encoding result.

[0190] Specifically, the stimulation device can generate a stimulation signal of a corresponding stimulation pattern based on the stimulation pattern parameters in the encoding result. The stimulation signal can be used to stimulate brain tissue, and the brain tissue can generate a neural signal when receiving the stimulation signal output by the stimulation device, thereby achieving brain perception.

[0191] Still taking the example of the smart glasses scenario, the stimulation device generates a stimulation signal based on the encoding result. The stimulation signal can stimulate the area of ​​the brain tissue responsible for vision, so that the blind person wearing the smart glasses can perceive the position and size of obstacles or the status of traffic lights, and then avoid obstacles or pass when the traffic light is green.

[0192] The embodiment shown in FIG12 is described by taking the stimulation device as an example to generate the stimulation signal independently. In other possible implementations of the present application, the stimulation device can generate the stimulation signal in combination with the collected information.

[0193] 13 shows a flow chart of a signal processing method. The method may be performed by the effector 400 in the brain-computer interface system 10. The effector 400 includes a stimulation device. The method includes the following steps:

[0194] S1302 : The stimulation device receives an internal input signal of at least one modality generated by the decoding device 300 .

[0195] Specifically, when decoding a predetermined brain state or brain instruction, the decoding device 300 may generate an internal input signal of at least one modality. For example, if the brain state indicates that the user is about to have an epileptic seizure, the decoding device 300 may generate an internal input signal to suppress the seizure.

[0196] S1304: The stimulation device encodes an internal input signal of at least one modality to obtain an encoding result.

[0197] The encoded result includes parameters for a stimulation mode that matches the modality. Stimulation modes include, but are not limited to, light stimulation, electrical stimulation, and ultrasonic stimulation. The stimulation device can store a correspondence between modalities and stimulation modes. Upon acquiring an internal input signal for at least one modality, the device can determine a stimulation mode that matches the modality based on the correspondence.

[0198] S1306. The stimulation device generates a stimulation signal based on the encoding result.

[0199] The stimulation device can generate a stimulation signal corresponding to the stimulation pattern based on the parameters of the stimulation pattern in the encoded result. This stimulation signal can be used to stimulate brain tissue. The stimulation signal output by the stimulation device can generate a neural signal in the brain tissue, and brain control can be achieved based on this neural signal. For example, the stimulation signal can stimulate the area of ​​brain tissue responsible for movement, thereby suppressing the user's epilepsy.

[0200] Based on the aforementioned method embodiment, the present application further provides a signal processing device 200. As shown in FIG14 , the device 200 includes:

[0201] An acquisition unit 1402 is configured to acquire a first signal, where the first signal is a signal collected from brain tissue, and to acquire a downstream feedback signal;

[0202] The processing unit 1404 is configured to process the first signal according to the feedback signal.

[0203] Exemplarily, the acquisition unit 1402 and the processing unit 1404 can be implemented by software. When implemented by software, the acquisition unit 1402 and the processing unit 1404 can be applications running on a computing device. The application can be provided to users in the form of virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that uses virtualization technology to virtualize a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable. Its computing performance is no different from that of a traditional physical machine and has the characteristics of secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user spaces, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0204] The acquisition unit 1402 and processing unit 1404 may also be implemented in hardware. When implemented in hardware, the acquisition unit 1402 may be implemented via a communication interface, such as a transceiver, and the processing unit 1404 may be a processor, or a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0205] In some possible implementations, the downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

[0206] In some possible implementations, the first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback on a second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

[0207] In some possible implementations, the feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

[0208] In some possible implementations, the processing unit 1404 is specifically configured to:

[0209] When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

[0210] In some possible implementations, the processing strategy includes at least one of the following:

[0211] Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

[0212] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit 1404 is specifically configured to:

[0213] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0214] A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

[0215] In some possible implementations, the feedback signal includes at least one of a brain command or a brain state and at least one of an action command or an action state, and the processing unit 1404 is specifically configured to:

[0216] determining a first movement intention of the user according to the brain instruction or brain state, and predicting a second movement intention of the user according to the action instruction or action state;

[0217] A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

[0218] In some possible implementations, the feedback signal includes a brain instruction or a brain state at the second time, and the processing unit 1404 is specifically configured to:

[0219] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0220] When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

[0221] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit 1404 is specifically configured to:

[0222] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0223] When the decoding accuracy meets the second set condition, the task parameters are updated, and the first signal modulated based on the updated task parameters is sent to the decoding device.

[0224] In some possible implementations, the feedback signal includes a brain command or a brain state, and the processing unit 1404 is specifically configured to:

[0225] determining decoding accuracy according to the brain command or brain state and the target brain command or target brain state in the feedback signal;

[0226] When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

[0227] In some possible implementations, the first signal is a digital signal, and the acquiring unit 1402 is specifically configured to:

[0228] acquiring neural signals collected from the brain tissue;

[0229] converting the neural signal into an analog signal;

[0230] The analog signal is converted into the digital signal through analog-to-digital conversion.

[0231] In some possible implementations, the processing unit 1404 is further configured to:

[0232] The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

[0233] In some possible implementations, the signal preprocessing module 202 is integrated or deployed in the hardware signal acquisition device 100, and the processing unit 1404 is specifically configured to:

[0234] The analog signal or the digital signal is preprocessed on the hardware signal acquisition device 100 side by the signal preprocessing module 202 .

[0235] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0236] In some possible implementations, the prosthesis is a wearable device, the wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0237] In some possible implementations, the neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0238] This application also provides a computing device 1500. As shown in Figure 15, computing device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. Processor 1504, memory 1506, and communication interface 1508 communicate with each other via bus 1502. Computing device 1500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1500.

[0239] Bus 1502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG15 shows a single bus line, but this does not imply a single bus or type of bus. Bus 1502 may include a path for transmitting information between various components of computing device 1500 (e.g., memory 1506, processor 1504, and communication interface 1508).

[0240] The processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0241] The memory 1506 may include a volatile memory, such as a random access memory (RAM). The memory 1506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to implement the aforementioned signal processing method. Specifically, the memory 1506 stores instructions for the signal processing device 200 or the brain-computer interface system 10 to execute the signal processing method.

[0242] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or a communication network.

[0243] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0244] As shown in Figure 16, the computing device cluster includes at least one computing device 1500. The memory 1506 of one or more computing devices 1500 in the computing device cluster may store the same signal processing apparatus 200 or brain-computer interface system 10 for executing instructions of the signal processing method.

[0245] In some possible implementations, one or more computing devices 1500 in the computing device cluster can also be used to execute some of the instructions for executing the signal processing method by the signal processing apparatus 200 or the brain-computer interface system 10. In other words, the combination of one or more computing devices 1500 can jointly execute the instructions for executing the signal processing method by the signal processing apparatus 200 or the brain-computer interface system 10.

[0246] It should be noted that the memory 1506 in different computing devices 1500 in the computing device cluster can store different instructions for executing part of the functions of the signal processing device 200 or the brain-computer interface system 10.

[0247] The present application also provides a chip. The chip may include a processor and a communication interface, wherein the communication interface is configured to acquire a first signal, the first signal being a signal collected from brain tissue, and the processor is configured to execute computer-readable instructions to perform the aforementioned signal processing method.

[0248] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned signal processing method applied to the signal processing device 200 or the brain-computer interface system 10.

[0249] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned signal processing method.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A signal processing method, characterized in that: The method comprises: Acquiring a first signal, wherein the first signal is a signal collected from brain tissue, and acquiring a downstream feedback signal; The first signal is processed according to the feedback signal.

2. The method according to claim 1, characterized in that The downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

3. The method according to claim 1 or 2, characterized in that: The first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback to the second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

4. The method according to claim 3, characterized in that The feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

5. The method according to any one of claims 1 to 4, characterized in that: The processing of the first signal according to the feedback signal includes: When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

6. The method according to claim 5, characterized in that The processing strategy includes at least one of the following: Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

7. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing of the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

8. The method according to claim 5, characterized in that The feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing of the first signal by a processing strategy corresponding to the trigger condition includes: determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state; A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

9. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

10. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

11. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

12. The method according to any one of claims 1 to 11, characterized in that: The first signal is a digital signal, and obtaining the first signal includes: acquiring neural signals collected from the brain tissue; converting the neural signal into an analog signal; The analog signal is converted into the digital signal by analog-to-digital conversion.

13. The method according to claim 12, characterized in that The method further comprises: The analog signal or the digital signal is preprocessed to achieve signal compression.

14. The method according to claim 13, characterized in that The preprocessing of the analog signal or the digital signal comprises: The analog signal or the digital signal is preprocessed on the hardware signal acquisition device side.

15. The method according to any one of claims 12 to 14, characterized in that The neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

16. The method according to claim 15, characterized in that The prosthesis is a wearable device, and the wearable device is deployed with a sensor of at least one modality. The wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

17. The method according to any one of claims 12 to 14, characterized in that The neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

18. A signal processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first signal, wherein the first signal is a signal collected from brain tissue, and acquire a downstream feedback signal; A processing unit is used to process the first signal according to the feedback signal.

19. The device according to claim 18, characterized in that The downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

20. The device according to claim 18 or 19, characterized in that The first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback to the second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

21. The device according to claim 18, characterized in that The feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

22. The device according to any one of claims 18 to 20, characterized in that The processing unit is specifically used for: When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

23. The device according to claim 22, characterized in that The processing strategy includes at least one of the following: Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

24. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

25. The device according to claim 22, characterized in that The feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing unit is specifically used for: determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state; A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

26. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state at a second time, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

27. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

28. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

29. The device according to any one of claims 18 to 28, characterized in that The first signal is a digital signal, and the acquisition unit is specifically used for: acquiring neural signals collected from the brain tissue; converting the neural signal into an analog signal; The analog signal is converted into the digital signal by analog-to-digital conversion.

30. The device according to claim 29, characterized in that The processing unit is also used for: The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

31. The device according to claim 30, characterized in that The signal preprocessing module is integrated or deployed in a hardware signal acquisition device, and the processing unit is specifically used for: The analog signal or the digital signal is preprocessed by the signal preprocessing module on the hardware signal acquisition device side.

32. The device according to any one of claims 18 to 31, characterized in that The neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

33. The device according to claim 32, characterized in that The prosthesis is a wearable device, and the wearable device is deployed with a sensor of at least one modality. The wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

34. The device according to any one of claims 18 to 31, characterized in that The neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

35. A chip, characterized in that: The chip includes a processor and a communication interface, the communication interface is used to obtain a first signal, the first signal is a signal collected from brain tissue, and the processor is used to execute computer-readable instructions to perform the signal processing method according to any one of claims 1 to 17.

36. A brain-computer interface system, characterized in that: The system includes a signal processing device and a hardware signal acquisition device. The signal processing device is used to execute the signal processing method described in any one of items 1 to 17 to process the signal output by the hardware signal acquisition device to achieve brain-computer interaction.

37. The system according to claim 36, characterized in that The signal processing device includes a signal preprocessing module, which is integrated and deployed in the hardware signal acquisition device, and the signal preprocessing module is used to: The signal output by the hardware signal acquisition device is preprocessed on the hardware signal acquisition device side.

38. The system according to claim 36 or 37, characterized in that The system further comprises: A stimulation device is used to generate a stimulation signal based on the encoding result of an external input signal of at least one modality acquired by a prosthesis, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

39. The system according to claim 36 or 37, characterized in that The system further comprises: A stimulation device is used to generate a stimulation signal based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

Images