Signal processing method and related device

Abstract

The invention provides a signal processing method, which is applied to the technical field of brain-computer interfaces (BCI), and comprises the following steps: acquiring a first signal which is a signal acquired from brain tissue, acquiring a downstream feedback signal, and then processing the first signal according to the feedback signal. According to the method, the feedback signal is combined to process the first signal collected from the brain tissue instead of directly inputting the first signal into the decoding device for decoding, so that the long-term stability of signal coding and decoding is guaranteed, the situation that the long-term stability of signal coding and decoding is reduced or even fails due to the non-stability of the signal is avoided, and the service requirement can be met.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a signal processing method, device, chip, system, computer-readable storage medium, and computer program product. Background Art

[0002] In recent years, brain-computer interface (BCI) has gradually become an important enabling technology for the future intelligent world. For example, in the medical field, a neurosurgery robot is used to implant a proprietary chip and information strip into the brain tissue, and then the brain signal can be directly read through the Universal Serial Bus (USB) interface. This method causes less damage to the brain tissue and has broad application prospects.

[0003] BCI is a new communication and control technology established between brain tissue (such as human brain or animal brain) and computer or other electronic devices that does not rely on conventional brain information output pathways (peripheral nerves and muscle tissue). BCI is divided into two major directions: invasive and non-invasive according to the degree of electrode implantation.

[0004] Both invasively collected signals and non-invasively collected signals are non-stationary, which is specifically manifested in the fact that they are greatly affected by factors such as electrode inactivation, random noise, motion artifacts, physical and mental state, etc. This leads to a decrease in the long-term stability of signal encoding and decoding, or even failure, making it difficult to meet business needs. Summary of the invention

[0005] The present application provides a signal processing method, which processes a first signal collected from brain tissue in combination with a downstream feedback signal, rather than directly inputting the first signal into a decoding device for decoding, thereby ensuring the long-term stability of signal encoding and decoding, avoiding the long-term stability of signal encoding and decoding being reduced or even failing due to the non-stationarity of the signal, and meeting business needs. The present application also provides a signal processing device, a chip, a brain-computer interface system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above-mentioned signal processing method.

[0006] In the first aspect, the present application provides a signal processing method. The method can be performed by a signal processing device. The signal processing device can be a device in a brain-computer interface system for processing signals collected from brain tissue. The signal processing device can be a software device, such as locally deployed software or cloud services. The software device can be deployed in a computing device cluster (including at least one computing device), and the computing device cluster executes the degree code of the software device to execute the signal processing method of the present application. The signal processing device can also be a hardware device, such as a computing device cluster with signal processing functions. The signal processing method of the present application is executed when the hardware device is running.

[0007] Specifically, the signal processing device can obtain a first signal, which is a signal collected from brain tissue, and obtain a downstream feedback signal, and then the signal processing device can process the first signal according to the feedback signal. In this way, the first signal can be processed in combination with the feedback signal, rather than directly inputting the first signal into the decoding device for decoding, thereby ensuring the long-term stability of the signal encoding and decoding, avoiding the long-term stability of the signal encoding and decoding from being reduced or even failing due to the non-stationarity of the signal, and being able to meet the needs of the business (especially the business used for a long time, such as the medical rehabilitation business).

[0008] In some possible implementations, the downstream feedback signal includes a feedback signal of a decoding device or a feedback signal of an effector. The feedback signal of the decoding device itself or the feedback signal of an effector controlled by the decoding device can be used to determine the decoding accuracy, and the signal is processed based on the decoding accuracy, such as data integration, to ensure decoding stability.

[0009] In some possible implementations, the first signal may be a signal collected from brain tissue at a first time, and the feedback signal is used to characterize feedback on the second signal, the second signal is a signal collected from brain tissue at a second time, and the second time is earlier than the first time. The first time and the second time may be moments or time periods (time series). The method may process the signal at the current time through feedback of the signal at the historical time to ensure decoding stability.

[0010] In some possible implementations, the feedback signal includes a brain command or brain state at a second time obtained by decoding the second signal by a decoding device, or the feedback signal includes an action command or action state output by the effector in response to a brain command (brain command) or brain state (brain status) at the second time. Brain commands refer to commands issued by brain tissue, which can usually be control commands, such as brain commands for controlling the movement of a prosthesis, and brain states refer to the active state or excited state of at least one area in the brain tissue. Action commands or action states can be commands or states output by the effector in response to brain commands or brain states. Wherein, when the effector is an external application (external application), the above-mentioned action commands / action states can also be application commands (application command) / application status (application status). For example, the application status can include posture.

[0011] The decoding accuracy can be determined by the above-mentioned brain instructions / brain states or action instructions / action states, and signal processing based on the decoding accuracy, such as data integration, can ensure decoding stability.

[0012] In some possible implementations, the signal processing device may process the first signal according to a processing strategy corresponding to the trigger condition when determining that the trigger condition is satisfied according to the feedback signal. The trigger condition may be a condition related to a monitoring indicator, and the monitoring indicator may be set according to requirements, for example, the monitoring indicator may be decoding accuracy. By processing the first signal according to the processing strategy corresponding to the trigger condition when the trigger condition is satisfied, the signal processing device can ensure decoding stability on the one hand, and avoid additional overhead caused by signal processing when decoding is stable on the other hand.

[0013] In some possible implementations, the processing strategy includes at least one of the following: calibration, collaborative control, data integration, stable release channel screening, or paradigm update. When the trigger condition is met, the signal is processed using the processing strategy corresponding to the trigger condition, which can achieve targeted processing and thus improve decoding accuracy and decoding stability.

[0014] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states, and then update the decoding parameters for decoding the first signal based on the decoding accuracy. In this way, signal calibration can be achieved, and the decoding stability can be improved by calibrating the signal. It should be noted that when calibrating the signal, either timed calibration or adaptive calibration can be used. Among them, adaptive calibration is triggered when the decoding accuracy is less than a set value, which can reduce unnecessary overhead.

[0015] In some possible implementations, the feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state. Accordingly, the signal processing device can determine the user's first movement intention according to the brain instruction or the brain state, predict the user's second movement intention according to the action instruction or the action state, determine the ratio of the first movement intention to the second movement intention, and send the ratio to the decoding device.

[0016] In this way, the coordinated control of the motion intention obtained by decoding and the motion intention predicted by the algorithm can be achieved to ensure the stability of decoding. Among them, the coordinated control can be divided into time-sharing coordinated control or adaptive coordinated control. Adaptive coordinated control supports manual intervention in the algorithm control process and has high reliability.

[0017] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and the information of the acquisition channels whose emission stability meets the requirements is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirements.

[0018] The first set condition may be that the decrease in decoding accuracy within a set time period is greater than a set amplitude, indicating a sudden and substantial decrease in decoding accuracy (also referred to as a sag). Considering that a sag in decoding accuracy may be caused by electrode inactivation or changes in emission characteristics, the signal processing device determines the emission stability of multiple acquisition channels, and sends information on acquisition channels whose emission stability meets the requirements to the decoding device, thereby improving the decoding stability.

[0019] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

[0020] The second setting condition may be that the decoding accuracy is continuously lower than the accuracy threshold in a set time period, indicating that the decoding accuracy is continuously low for a period of time. The reason why the decoding accuracy is continuously maintained at a low level may be that the existing task is too challenging / difficult for the user. Based on this, the signal processing device may update the task parameters and send the first signal modulated based on the updated task parameters to the decoding device. In this way, the task difficulty can be reduced, thereby ensuring the decoding stability.

[0021] In some possible implementations, the feedback signal includes brain instructions or brain states, and the signal processing device can determine the decoding accuracy based on the brain instructions or brain states in the feedback signal and the target brain instructions or target brain states; when the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

[0022] The third set condition may be that the decoding accuracy is continuously lower than the accuracy threshold during a set period of time, indicating that the decoding accuracy is continuously low for a period of time, and the signal processing device may consider loading the pre-trained model and fine-tuning it based on a small amount of data on that day. Specifically, the signal processing device may use the first signal to fine-tune the pre-trained model, obtain updated decoding parameters, and send the updated decoding parameters to the decoding device. In this way, data integration can be performed to improve decoding stability.

[0023] In some possible implementations, the first signal is a digital signal, and the signal processing device can obtain the neural signal collected from the brain tissue, convert the neural signal into an analog signal, and convert the analog signal into the digital signal through analog-to-digital conversion. This can lay the foundation for realizing brain-computer interaction.

[0024] In some possible implementations, the signal processing device may preprocess the analog signal or the digital signal to achieve signal compression. Preprocessing the analog signal or the digital signal may improve signal quality and reduce the amount of signal data.

[0025] In some possible implementations, the signal processing device can pre-process the analog signal or the digital signal on the hardware signal acquisition device side. In this way, when the brain-computer interface acquisition technology develops towards high throughput, high precision, high sampling rate and wireless transmission, it can also perform analog signal and / or digital signal pre-processing at the perception end, extract functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reduce communication bandwidth and power consumption pressure, and realize sensing and computing integration.

[0026] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0027] This method also supports the independent generation of stimulation signals to stimulate brain tissue to generate neural signals, thereby achieving brain-computer interaction.

[0028] In some possible implementations, the prosthesis is a wearable device, such as smart glasses or a smart head display. The wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0029] By generating stimulation signals from the encoding results of the above external input signals and stimulating the corresponding functional areas in the brain tissue, the prosthesis user can be helped to achieve perception. For example, it can help the blind perceive the location and size of obstacles or the status of traffic lights, thereby improving convenience.

[0030] In some possible implementations, the neural signal includes a signal generated by the stimulation signal output by the stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on the encoding result of the internal input signal of at least one modality generated when the decoding device decodes the set brain state or brain instruction, and the encoding result includes the parameters of the stimulation mode matching the modality. In this way, brain-computer interaction combining acquisition and stimulation can be realized to meet the needs of corresponding business scenarios.

[0031] In a second aspect, the present application provides a signal processing device. The device comprises:

[0032] An acquisition unit, configured to acquire a first signal, wherein the first signal is a signal collected from brain tissue, and acquire a downstream feedback signal;

[0033] A processing unit is used to process the first signal according to the feedback signal.

[0034] In some possible implementations, the downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

[0035] In some possible implementations, the first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to characterize feedback to a second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

[0036] In some possible implementations, the feedback signal includes the brain instructions or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instructions or action state output by the effector in response to the brain instructions or brain state at the second time.

[0037] In some possible implementations, the processing unit is specifically configured to:

[0038] When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

[0039] In some possible implementations, the processing strategy includes at least one of the following:

[0040] Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

[0041] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit is specifically configured to:

[0042] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0043] A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

[0044] In some possible implementations, the feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing unit is specifically configured to:

[0045] determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state;

[0046] A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

[0047] In some possible implementations, the feedback signal includes a brain instruction or a brain state at a second time, and the processing unit is specifically configured to:

[0048] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0049] When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

[0050] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit is specifically configured to:

[0051] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0052] When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

[0053] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit is specifically configured to:

[0054] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0055] When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

[0056] In some possible implementations, the first signal is a digital signal, and the acquiring unit is specifically configured to:

[0057] acquiring neural signals collected from the brain tissue;

[0058] converting the neural signal into an analog signal;

[0059] The analog signal is converted into the digital signal by analog-to-digital conversion.

[0060] In some possible implementations, the processing unit is further configured to:

[0061] The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

[0062] In some possible implementations, the signal preprocessing module is integrated or deployed in a hardware signal acquisition device, and the processing unit is specifically used for:

[0063] The analog signal or the digital signal is preprocessed by the signal preprocessing module on the hardware signal acquisition device side.

[0064] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0065] In some possible implementations, the prosthesis is a wearable device, the wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0066] In some possible implementations, the neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0067] In a third aspect, the present application provides a chip. The chip includes a processor and a communication interface, the communication interface is used to obtain a first signal, the first signal is a signal collected from brain tissue, and the processor is used to execute computer-readable instructions to perform the signal processing method described in the first aspect or any implementation of the first aspect.

[0068] In a fourth aspect, the present application provides a brain-computer interface system. The system includes a signal processing device and a hardware signal acquisition device, wherein the signal processing device is used to execute the signal processing method described in the first aspect or any implementation of the first aspect to process the signal output by the hardware signal acquisition device to achieve brain-computer interaction.

[0069] In some possible implementations, the signal processing device includes a signal preprocessing module, which is integrated and deployed in the hardware signal acquisition device, and the signal preprocessing module is used to:

[0070] The signal output by the hardware signal acquisition device is preprocessed on the hardware signal acquisition device side.

[0071] In some possible implementations, the system further includes:

[0072] A stimulation device is used to generate a stimulation signal based on the encoding result of an external input signal of at least one modality acquired by a prosthesis, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

[0073] In some possible implementations, the system further includes:

[0074] A stimulation device is used to generate a stimulation signal based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

[0075] In a fifth aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory, so that the computing device or the computing device cluster performs the signal processing method as described in the first aspect or any implementation of the first aspect.

[0076] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the signal processing method described in the first aspect or any one of the implementations of the first aspect.

[0077] In a seventh aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or the computing device cluster to execute the signal processing method described in the first aspect or any one of the implementations of the first aspect.

[0078] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0080] Figure 1 A schematic diagram of the architecture of a brain-computer interface system provided in an embodiment of the present application;

[0081] Figure 2 A schematic diagram of the architecture of another brain-computer interface system provided in an embodiment of the present application;

[0082] Figure 3 A schematic diagram of the architecture of another brain-computer interface system provided in an embodiment of the present application;

[0083] Figure 4 A schematic diagram of a signal preprocessing process provided in an embodiment of the present application;

[0084] Figure 5 A hierarchical logic diagram of a storage system provided in an embodiment of the present application;

[0085] Figure 6 Schematic diagram of different brain-computer interaction modes provided in the embodiments of the present application;

[0086] Figure 7 A schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0087] Figure 8 A schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0088] Fig. 9 A schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0089] Fig.10 A schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0090] Fig.11 A schematic diagram of a scenario of a signal processing method provided in an embodiment of the present application;

[0091] Fig.12 A schematic diagram of another signal processing method provided in an embodiment of the present application;

[0092] Fig.13 A schematic diagram of another signal processing method provided in an embodiment of the present application;

[0093] Fig.14 A schematic diagram of the structure of a signal processing device provided in an embodiment of the present application;

[0094] Fig.15 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0095] Fig.16 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0096] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0097] First, some technical terms involved in the embodiments of the present application are introduced.

[0098] Brain-computer interface (BCI) is an in-depth development of human-computer interface technology (HCI). Traditional human-computer interaction technology requires the use of media layer software and hardware, such as computer display screens, visual interfaces, mice, etc. Brain-computer interface is a new type of human-computer interaction based on neuroscience and engineering technology. It can acquire and analyze brain signals in real time, establish direct communication and control channels between brain tissue (such as human brain, animal brain) and computers or other electronic devices, and directly convert central nervous system activities into information output, thereby realizing the understanding of brain consciousness. It can also intervene and even participate in brain activities through bioengineering methods, such as optogenetics, brain electrodes, etc. Users no longer need to express ideas or manipulate external devices through language or body movements.

[0099] Brain-computer interface technology is divided into two major categories: invasive and non-invasive according to the degree of implantation of the hardware signal acquisition device (also called hardware signal acquisition module).

[0100] Non-invasive brain-computer interface refers to a hardware signal acquisition device set up on the scalp and other areas, which can be worn outside the body. Non-invasive brain-computer interface mainly collects scalp electroencephalogram (EEG) signals. EEG is an electrical signal of multiple local field potentials passing through the cortex. Non-invasive brain-computer interface can be applied to physiological state assessment and regulation (attention, sleep, stress detection), auxiliary human-computer interaction (disability assistance, augmented reality / virtual reality), sports health and other scenarios.

[0101] Invasive brain-computer interface refers to a hardware signal acquisition device set up in the scalp, which can usually be implanted through surgery. Invasive brain-computer interface mainly collects pulse signals, local field potential (LFP), and electrocorticography (ECoG). Among them, the local field potential represents the sum of the electrical signals of neurons in the local area of ​​the electrode. The cortical ECoG is similar to the local field potential, but can only record electrical signals on the surface of the cortex. The invasive brain-computer interface collects high-quality signals and can be used in medical rehabilitation scenarios.

[0102] The following is an example of a brain-computer interface system, combined with the above explanation of the concepts related to brain-computer interface. Among them, the brain-computer interface system is a system that processes signals collected from brain tissue, so it can also be called a signal processing system or an EEG signal processing system.

[0103] like Figure 1As shown, the brain-computer interface system 10 includes a hardware signal acquisition device 100 (also referred to as a hardware signal acquisition module), a signal processing device 200 (or referred to as a signal processing module), a decoding device 300 (or referred to as a decoding module), an effector 400, and a feedback device 500 (or referred to as a feedback module). After the signal processing device 200 performs signal processing, it can also store the processed signal for subsequent use. Based on this, the signal processing device 200 can be a signal processing and storage device. Similarly, the decoding device 300 is used to perform feature extraction on the processed signal, and then decode the extracted features. Therefore, the decoding device 300 can be a feature extraction and decoding device.

[0104] The hardware signal acquisition device 100 is connected to the signal processing device 200, the signal processing device 200 is connected to the decoding device 300, and the decoding device 300 is connected to the effector 400. The effector 400 can act on the brain tissue directly, or it can be connected to the feedback device 500 and act on the brain tissue through the feedback device 500.

[0105] The functions, inputs, outputs, etc. of the hardware signal acquisition device 100 , the signal processing device 200 , the decoding device 300 , the effector 400 , and the feedback device 500 are described in detail below.

[0106] The hardware signal acquisition device 100 is used to collect signals from brain tissue. The hardware signal acquisition device 100 can be set on the scalp and areas outside the scalp to collect signals in a non-invasive manner. The hardware signal acquisition device 100 can also be implanted in the scalp to collect signals in an invasive manner. As shown in Table 1, the input of the hardware signal acquisition device 100 can be a neural signal of brain tissue, and the output of the hardware signal acquisition device 100 can be a digital signal. The hardware signal acquisition device 100 converts the neural signal into an electrical signal, such as an analog electrical signal (also called an analog signal), through electrodes and circuits, and converts the electrical signal into a processable digital electrical signal (also called a digital signal). It should be noted that the above-mentioned analog signals and digital signals can be collectively referred to as electroencephalographic signals.

[0107] Table 1

[0108]

[0109]

[0110] The signal processing device 200 is used to process the digital signal output by the hardware signal acquisition device 100, so as to improve the signal quality. The signal processing device 200 also has a storage function for storing the processed digital signal for subsequent use. The input of the signal processing device 200 can be the digital signal output by the signal acquisition device 100, and the output of the signal processing device 200 can be the processed digital signal, which is usually a high-quality digital signal.

[0111] The decoding device 300 is used to decode the signal processed by the signal processing device 200, so as to obtain brain instructions or brain states, thereby realizing the perception of brain consciousness. The input of the decoding device 300 can be the processed digital signal output by the signal processing device 200, and the output of the decoding device can be brain instructions or brain states. Among them, brain instructions refer to instructions issued by brain tissue, which can usually be control instructions, such as brain instructions for controlling the movement of prostheses, and brain states refer to the active state or excited state of at least one area in the brain tissue. Different areas of brain tissue can correspond to different functions, for example, the cerebellum is responsible for regulating and controlling fine movements, postures and balance, and the cerebral cortex is usually related to higher brain functions, such as conscious thinking, action selection and control. In specific implementation, the decoding device 300 can extract features from the processed signal, and then decode based on the features to obtain brain instructions or brain states. It should be noted that the features extracted by the decoding device 300 can be different depending on the signal type.

[0112] The effector 400 is used to respond to brain instructions or brain states and perform actions corresponding to the brain instructions or brain states. The effector 400, as an object controlled by the decoding device 300, can usually be at least one of an external device or a stimulation device. Among them, the external device is sometimes also called a peripheral device or an external application (external application), including but not limited to a computer cursor, a manipulator (mechanical arm), a mechanical skeleton, a wheelchair, and a monitoring and security device. The stimulation device can also be called a stimulation device, and can include the following types: a muscle stimulation device and a nerve control device. The muscle stimulation device can be a muscle functional electrical stimulation (FES). FES can apply a low-frequency pulse current or send it into the human body after amplification through signal-current conversion to produce an immediate effect, artificially causing a human body paralyzed due to damage to the central nervous system to produce movement. The nerve control device can include but is not limited to deep brain stimulation (DBS), spinal cord stimulation (SCS), and peripheral nerve electrical stimulation (PNS). The input of the effector 400 may be a brain command or a brain state, and the output of the effector 400 may be an action command or an action state.

[0113] The feedback device 500 is used to provide system feedback to the brain tissue, establishing a closed loop between the user's brain action / brain state and the system response. The input of the feedback device 500 can be the action instruction or action state output by the effector 400, and the output of the feedback device 500 can be feedback on the execution result (i.e., system feedback).

[0114] It should be noted that the feedback device 500 is an optional device. For example, the effector 400 can also be directly fed back to the brain tissue in the form of visual feedback without passing through the feedback device 500. Taking the effector 400 as an example of a computer cursor, the movement of the computer cursor can be fed back to the brain tissue in the form of visual feedback. In some cases, the feedback device 500 can be used as a supplement to visual feedback. For example, a robotic arm controlled by a brain-computer interface can usually enable patients with quadriplegia to complete functional movements, but the visual feedback provided to the patient is limited. Grasping movements transmit signals in the form of tactile feedback through the feedback device 500, which is often faster, more accurate and more realistic.

[0115] However, the signals collected by the above-mentioned invasive brain-computer interface or non-invasive brain-computer interface are greatly affected by factors such as electrode inactivation, random noise, motion artifacts (interference signals caused by slight movement of the subject during brain-computer interaction, such as head shaking), physical and mental state, and are non-stationary. This leads to a decrease in the long-term stability of signal encoding and decoding, or even failure, making it difficult to meet business needs.

[0116] In view of this, the present application provides a signal processing method. The method can be applied to Figure 2 The brain-computer interface system 10 shown is different from Figure 1 The brain-computer interface system 10 shown, Figure 2 In the brain-computer interface system 10 shown, a feedback path to the signal processing device 200 is provided downstream of the signal processing device 200, wherein the feedback path may be a feedback path between the downstream effector 400 and the signal processing device 200, or a feedback path between the downstream decoding device 300 and the signal processing device 200, and the feedback path is used to transmit the feedback signal of the effector 400 or the decoding device 300 to the signal processing device 200. Accordingly, the signal processing device 200 may process the signal (also referred to as the first signal) collected from the brain tissue based on the feedback signal, instead of directly inputting the signal into the decoding device 300 for decoding, thereby ensuring the long-term stability of the signal encoding and decoding, avoiding the long-term stability of the signal encoding and decoding from being reduced or even failing due to the non-stationarity of the signal, and meeting business needs.

[0117] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the brain-computer interface system 10 provided in the present application is introduced below in conjunction with the accompanying drawings.

[0118] See also Figure 2 The schematic diagram of the architecture of a brain-computer interface system 10 is shown, and the brain-computer interface system 10 includes a hardware signal acquisition device 100, a signal processing device 200, a decoding device 300, an effector 400 and a feedback device 500. Among them, the hardware signal acquisition device 100 is connected to the signal processing device 200, the signal processing device 200 is connected to the decoding device 300, and the decoding device 300 is connected to the effector 400. The effector 400 can act directly on the brain tissue, or it can be connected to the feedback device 500 and act on the brain tissue through the feedback device 500. In addition, a feedback path to the signal processing device 200 is also provided downstream of the signal processing device 200. Figure 2 An example is given in which a feedback path is provided between the decoding device 300 and the signal processing device 200. The feedback path is used to provide a feedback signal to the signal processing device 200 to assist the signal processing device 200 in signal processing.

[0119] The following table 2 Figure 2The functions, inputs, and outputs of each device / module in the brain-computer interface system 10 shown are introduced.

[0120]

[0121] The functions, inputs and outputs of the hardware signal acquisition device 100, the effector 400 and the feedback device 500 can be referred to in Table 1 or Figure 1 Related content description. The input of the signal processing device 200 is further increased by the feedback signal of the decoding device 300 on the basis of Table 1, and the feedback signal may be a brain instruction or a brain state. Accordingly, the signal processing device 200 is used to process the first signal according to the feedback signal, so as to improve the decoding stability. Among them, the first signal may be a signal collected from the brain tissue at a first time (for example, the current time, the current moment), and the second signal may be a signal collected from the brain tissue at a second time (for example, a historical time, a historical moment), and the feedback signal is used to characterize the feedback of the decoding device 300 to the second signal. In other words, the signal processing device 200 can process the signal at the current time in combination with the brain instruction or brain state at the historical time (such as the previous moment), including selecting / adjusting the signal postprocessing (postprocessing) process to improve the decoding stability. The output of the decoding device 300 is increased by the output of the signal processing device 200 on the basis of Table 1, specifically outputting the brain instruction / brain state to the signal processing device 200, so that the signal processing device 200 performs signal processing in combination with the feedback signals such as the brain instruction / brain state.

[0122] When the feedback path is set between the effector 400 and the signal processing device 200, the feedback signal can be an action command or an action state. The signal processing device 200 can process the signal in combination with the action command or action state of the previous moment, including selecting / adjusting the signal post-processing process to improve decoding stability. The output of the effector 400 is added to the output of the signal processing device 200 on the basis of Table 1, specifically outputting the action command / action state to the signal processing device 200, so that the signal processing device 200 performs signal processing in combination with the feedback signal such as the action command / action state.

[0123] The signal processing device 200 is further used to store the trigger condition and the processing strategy corresponding to the trigger condition. Accordingly, when the signal processing device 200 determines that the trigger condition is satisfied according to the feedback signal, it processes the first signal according to the processing strategy corresponding to the trigger condition.

[0124] The trigger condition may be a condition related to a monitoring indicator. The monitoring indicator may be set according to business requirements, for example, the monitoring indicator may include decoding accuracy. The decoding accuracy may be determined according to the brain instruction or brain state at the second time obtained by decoding, as well as the target brain instruction and target brain state.

[0125] The processing strategy may include at least one of calibration, collaborative control, data integration, stable release channel screening, or paradigm update. Among them, calibration may include timed calibration and adaptive calibration. Similar to calibration, collaborative control may also include time-sharing collaborative control or adaptive collaborative control. The specific implementation of calibration, collaborative control, data integration, stable release channel screening, or paradigm update is described in detail below.

[0126] exist Figure 2 In the system shown, the first signal processed by the signal processing device 200 may be a digital signal. The digital signal may be acquired and processed by the hardware signal acquisition device 100. Specifically, the hardware signal acquisition device 100 is used to acquire neural signals from brain tissue, convert the neural signals into model signals, and then convert the analog signals into the digital signals through analog-to-digital conversion, such as an analog-to-digital converter (ADC). The signal processing device 200 obtains the digital signal from the hardware signal acquisition device 100 and performs signal processing.

[0127] In some possible implementations, see Figure 3 , the signal processing device 200 may include a signal preprocessing module 202 and a signal postprocessing module 204. The signal preprocessing module 202 and the signal postprocessing module 206 may perform signal transmission through a transmission module 206. The transmission module 206 may be a module that implements signal transmission based on wired or wireless technology. Wireless technology may include but is not limited to Bluetooth or wireless networks (such as Wi-Fi).

[0128] Among them, the signal preprocessing module 202 is used to preprocess the signal collected by the hardware signal acquisition device 100. The preprocessing may include at least one of analog signal preprocessing or digital signal preprocessing. The signal preprocessing module 202 can be integrated or deployed in the hardware signal acquisition device 100, so as to preprocess the signal at the perception end (such as the hardware signal acquisition device 100) to achieve signal compression.

[0129] like Figure 4As shown, the neural signals collected in the brain tissue can be converted into electrical signals that can be processed by the computing device through the analog front-end chip processing in the hardware signal acquisition device 100. AFE can integrate ADC, so the processable electrical signal converted by AFE can be a digital signal. The above digital signal can be preprocessed by at least one module in the following signal preprocessing module 202: filtering, signal-to-noise ratio enhancement (SNR Boost), adaptive threshold calculation (Adaptive Threshold), pulse detection (Spike Detection), false positive detection (false positives reduction, FP Reduction), signal compression (such as pulse compression Spike Compression). Among them, false positive detection refers to the detection of noise identified as a peak, and false positive detection can be achieved by waveform detection. The signal preprocessing module 202 can be integrated or deployed in the hardware signal acquisition device 100 by on-chip implementation. The input of the signal preprocessing module 202 can be a voltage signal converted by ADC, and the output can include but is not limited to pulse waveforms, pulse emission time points, pulse emission frequencies in specific time windows, specific spectrum signals and other modalities.

[0130] The preprocessed digital signal can be sent to the transmission module 206 according to the corresponding transmission protocol, and the transmission module 206 transmits the preprocessed digital signal to the signal post-processing module 204 based on the corresponding transmission protocol. Figure 4 This is just an example of the preprocessing process of digital signals. Analog signals can also be preprocessed. For example, the analog signal can enter an amplifier or a filter for preprocessing.

[0131] In this way, while brain-computer interface acquisition technology develops towards high throughput, high precision, high sampling rate and wireless transmission, it can also perform analog signal and / or digital signal preprocessing at the perception end, extract functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reduce communication bandwidth and power consumption pressure, and realize sensing and computing integration.

[0132] In some possible implementations, the signal processing device 200 may further include a storage module 208, also referred to as a storage system. The preprocessed signal may be transmitted to the storage module 208 through the transmission module 206 for storage for subsequent use. It should be noted that the digital signal converted by the ADC may also be sent directly to the transmission module 206 without preprocessing, and the transmission module 206 may transmit the digital signal to the signal post-processing module 204, and the transmission module 206 may also transmit the digital signal to the storage module 208 for subsequent processing. For example, the signal post-processing module 204 may read the stored digital signal from the storage module 208 for signal processing (e.g., post-processing).

[0133] The signal post-processing module 204 is used to process the brain instructions or brain states fed back by the decoding device 300, or the action instructions or action states fed back by the effector 400, Figure 3 The first signal is processed by taking the feedback signal of the decoding device 300 as an example. Figure 3 As shown, a feedback path is provided between the decoding device 300 and the signal post-processing module 204 of the signal processing device 200, and the feedback path is used to transmit the feedback signal of the decoding device 300 to the signal post-processing module 204 of the signal processing device 200. Accordingly, the signal post-processing module 204 can process the first signal based on the brain instruction or brain state (such as visual information, posture information) fed back by the decoding device 300, for example, performing timing or adaptive calibration, time-sharing or adaptive collaborative control, data integration, stable release channel screening, paradigm optimization, etc., to improve the long-term stability of decoding.

[0134] The above is an example of collecting signals from brain tissue, processing and decoding the signals to control the effector 400. In some possible implementations, the effector 400 may include a stimulation device. The stimulation device may generate stimulation signals to stimulate brain tissue to generate neural signals.

[0135] In some possible implementations, the stimulation device can independently generate a stimulation signal. Specifically, the stimulation device can obtain an external input signal of at least one modality collected by a prosthesis (such as a visual prosthesis or an auditory prosthesis), and obtain a coding result by encoding the external input signal of at least one modality, wherein the coding result includes parameters of a stimulation pattern matching the modality. The stimulation device can generate a stimulation signal based on the coding result.

[0136] Among them, the prosthesis can be a wearable device, including but not limited to smart glasses and smart head-mounted display devices. The wearable device can deploy a sensor of at least one modality, and the wearable device is used to collect an external input signal of at least one modality through the sensor of at least one modality. Taking the prosthesis as smart glasses as an example, the smart glasses can deploy an image sensor (such as a camera), which can collect environmental images and recognize the environmental images through an image recognition model to obtain obstacle information or traffic light status information. The stimulation device can obtain the above-mentioned obstacle information or traffic light status information, and encode the obstacle information or traffic light status information to obtain a coding result. Then, a stimulation signal is generated based on the coding result to stimulate the area responsible for vision in the brain tissue, so that the blind wearing the smart glasses can perceive the position and size of the obstacle or the status of the traffic light.

[0137] In some other possible implementations, the stimulation device can generate a stimulation signal in combination with the collected information. Specifically, when the decoding device decodes the set brain state or brain instruction, it can generate an internal input signal of at least one modality. For example, the brain state can be a state in which the user is about to have an epilepsy, and the decoding device can generate an internal input signal that suppresses epilepsy. Accordingly, the stimulation device can obtain an internal input signal of at least one modality output by the decoding device, encode the internal input signal of at least one modality to obtain an encoding result, and the encoding result includes parameters of a stimulation mode that matches the modality. The stimulation device can generate a stimulation signal based on the encoding result. The stimulation signal can stimulate the area in the brain tissue responsible for movement, thereby suppressing the user from having epilepsy.

[0138] It should be noted that the aforementioned brain-computer interface system 10 can realize brain-computer interaction through different modes. Figure 5As shown, the brain-computer interface system 10 can realize brain-computer interaction through active mode, passive mode or interactive mode. In active mode, the brain-computer interface system 10 can control peripherals (such as a robotic arm) through intentional thoughts. In passive mode, the brain-computer interface system 10 can receive specific sensory stimulation from the user and collect corresponding EEG signals (such as the first signal) to control peripherals. In interactive mode, the brain-computer interface system 10 can trigger relevant EEG signals to control peripherals based on specific sensory, cognitive or motor events in the external environment. Whether it is active mode, passive mode, or interactive mode, the signal processing device 200 in the brain-computer interface system 10 can receive feedback signals, such as feedback signals (brain instructions or brain states) from the decoding device 300 to determine the decoding accuracy, thereby determining whether the decoding is accurate or stable. When the decoding is inaccurate or the decoding is unstable, for example, the degree of decrease in decoding accuracy / stability exceeds a set threshold, the signal post-processing module 204 of the signal processing device 200 can be triggered to post-process the first signal to improve the decoding accuracy or decoding stability.

[0139] Based on the aforementioned brain-computer interface system 10, the present application provides a signal processing method. The signal processing method of the present application is introduced below in conjunction with an embodiment.

[0140] See also Figure 6 The flowchart of a signal processing method shown in FIG. 1 is a flowchart of a signal processing method, which can be executed by the signal processing device 200 in the brain-computer interface system 10. The method includes the following steps:

[0141] S602: The signal processing device 200 obtains a first signal.

[0142] The first signal is a signal collected from brain tissue. The first signal may be a digital signal. For example, the first signal may be a neural signal collected from brain tissue by the hardware signal collection device 100, and the neural signal may be converted into an analog signal and then converted into a processable digital signal through an ADC.

[0143] In some possible implementations, the first signal may also be a preprocessed signal, and the signal quality may be improved by preprocessing. The preprocessing may include analog signal preprocessing and / or digital signal preprocessing. The analog signal preprocessing may include at least one of amplification and filtering (analog filtering). The digital signal preprocessing may include at least one of filtering (digital filtering), signal-to-noise ratio enhancement, adaptive threshold calculation, pulse detection waveform detection (such as FP Reduction), and signal compression (such as Spike Compression).

[0144] The specific implementation of digital signal preprocessing can be found in Figure 4, the digital signal output by AFE through AFC can be filtered by a filter and enhanced by signal-to-noise ratio to improve the signal-to-noise ratio. When adaptive threshold calculation is required, the filtered signal (if not filtered, it is the raw data after ADC conversion) can be divided into two paths, one entering the pulse detection module and the other entering the adaptive threshold calculation module, and the threshold is determined by the two outputs. After the threshold is determined, pulse detection can be performed based on the threshold. Waveform detection or pulse compression can be performed for pulse signals. In this way, the highest possible compression rate can be achieved under the premise of high precision and low power consumption.

[0145] It should be noted that the above preprocessing process can be performed on the hardware signal acquisition device 100 side. Figure 3 As shown, the signal processing device 200 may include a signal preprocessing module 202 and a signal post-processing module 204. The signal preprocessing module 202 may be integrated or deployed in the hardware signal acquisition device 100, and the signal processing device 200 may preprocess the signal through the signal preprocessing module 202 at the perception end (such as the hardware signal acquisition device 100) to achieve signal compression. Even if the hardware signal acquisition device 100 performs signal acquisition based on high throughput, high precision, and high sampling rate, or adopts wireless transmission, it can also perform analog signal and / or digital signal preprocessing at the perception end, extract functional signals from the original bioelectric signals (analog electrical signals converted from neural signals), reduce communication bandwidth and power consumption pressure, and achieve integrated sensing and computing.

[0146] S604: The signal processing device 200 obtains a downstream feedback signal.

[0147] The downstream feedback signal refers to a signal fed back by a downstream device (downstream equipment) of the signal processing device 200. The downstream feedback signal may include a feedback signal of the decoding device 300 or a feedback signal of the effector 400. The feedback signal is used to represent feedback on a historical signal before the first signal.

[0148] In some possible implementations, the first signal may be a signal collected from brain tissue at a first time, and the second signal may be a signal collected from brain tissue at a second time, and the second time is earlier than the first time. Accordingly, the feedback signal is used to characterize the feedback of the second signal. Based on this, the feedback signal may include a brain instruction or brain state at a second time obtained by decoding the second signal by the decoding device 300, or the feedback signal may include an action instruction or action state output by the effector 400 in response to the brain instruction or brain state at the second time. Among them, the action state may be visual information or posture information. Depending on the application scenario (such as different control peripherals), the action instruction and action state may be different.

[0149] It should be noted that the first time and the second time may be a moment, or a time sequence (a time period, a control cycle), which is not limited in the embodiment of the present application.

[0150] S606: The signal processing device 200 processes the first signal according to the feedback signal.

[0151] Specifically, the signal processing device 200 can process the first signal at a fixed time. Alternatively, the signal processing device 200 can also process the first signal by a processing strategy corresponding to the trigger condition when determining that the trigger condition is met according to the feedback signal. Among them, the trigger condition can be a condition related to the monitoring indicator. The monitoring indicator can be set according to business needs. For example, the monitoring indicator may include decoding accuracy. The decoding accuracy can be determined based on the brain instructions or brain state at the second time obtained by decoding, as well as the target brain instructions and target brain state.

[0152] For example, when the decoding accuracy drops suddenly, it may be caused by electrode inactivation or changes in the release characteristics. The signal processing device 200 can reselect the acquisition channel whose release stability meets the requirements. For another example, when the decoding accuracy remains at a low level for a period of time, it may be considered that the existing task is too challenging / difficult for the user. The signal processing device 200 can redefine the task parameters and adjust the difficulty of the task. Based on this, the trigger condition may include that the decoding accuracy meets the set conditions, and the set conditions may include multiple conditions. Different trigger conditions may correspond to different processing strategies. Processing strategies may include calibration, collaborative control, data integration, stable release channel screening, or paradigm update (specifically updating task parameters).

[0153] The signal processing device 200 may store the correspondence between the trigger condition and the processing strategy. When the trigger condition is met, the signal processing device 200 may determine the processing strategy corresponding to the currently met trigger condition according to the correspondence between the trigger and the processing strategy.

[0154] Based on the above description, it can be known that the signal processing method of the present application can process the signal (such as the first signal) collected from the brain tissue based on the downstream feedback signal (such as the feedback signal of the decoding device 300), instead of directly inputting the signal into the decoding device 300 for decoding, thereby ensuring the long-term stability of the signal encoding and decoding, avoiding the long-term stability of the signal encoding and decoding caused by the non-stationarity of the signal. Decrease or even failure, and can meet business needs.

[0155] Next, the signal processing method of the present application is illustrated in combination with different application scenarios.

[0156] First, see Figure 7 A schematic diagram of a signal processing method is shown in FIG. Figure 7As shown, the post-processing module 204 in the signal processing device 200 may include a calibration module, such as a regular calibration module or an adaptive calibration module.

[0157] In this scenario, the brain-computer interface system 10 is used to realize brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 can be fed back to the calibration module. The process of the calibration module performing timing calibration or adaptive calibration is described below.

[0158] When the calibration module adopts timed calibration, the brain instructions or brain states fed back by the decoding device 300 can be obtained at a set frequency (or time interval), and the decoding accuracy (or error) can be determined according to the brain instructions or brain states fed back by the decoding device 300 and the locally stored target brain states or target brain instructions (such as the target brain states or brain instructions stored in the storage module 208). Then the calibration module can update the decoding parameters (such as the parameters of the decoding model) for decoding the first signal according to the decoding accuracy. In specific implementation, the calibration module can estimate the decoding parameters according to the decoding accuracy, and send the estimated decoding parameters (estimated decoding parameters) as updated decoding parameters to the decoding device 300, thereby updating the decoding parameters.

[0159] Furthermore, the calibration module can also adjust the frequency or time interval of the timing calibration according to the decoding accuracy. For example, if the decoding accuracy is greater than the set value within the set time period, the calibration module can reduce the frequency of the timing calibration, or increase the time interval of the timing calibration. For example, when the decoding accuracy is continuously greater than 0.95, the calibration module can increase the time interval of the timing calibration from once a day to once a week.

[0160] When the calibration module adopts adaptive calibration, the brain instruction or brain state fed back by the decoding device 300 can be obtained, and the decoding accuracy (or error) can be determined according to the brain instruction or brain state fed back by the decoding device 300 and the target brain state or target brain instruction stored locally. When the decoding accuracy is less than the set value (or the error is greater than another set value), the calibration module can update the decoding parameters for decoding the first signal according to the decoding accuracy (or error). Among them, the calibration module can determine the updated decoding parameters in an iterative manner until the decoding accuracy is greater than the set value (or the error is reduced to another set value).

[0161] It should be noted that the first signal may be a signal collected from brain tissue at a first time, and the second signal may be a signal collected from brain tissue at a second time, the second time being earlier than the first time. Accordingly, the brain instruction or brain state fed back by the decoding device 300 may be the brain instruction or brain state at the second time.

[0162] Secondly, see Figure 8 A schematic diagram of a signal processing method is shown in FIG. Figure 8 As shown, the post-processing module 204 in the signal processing device 200 may include a collaborative control module, also referred to as a shared control module, or a shared control parameter calculation module.

[0163] In this scenario, the brain-computer interface system 10 is used to realize brain-controlled peripherals. Among them, the brain instructions or brain states decoded by the decoding device 300 and the action instructions or action states output by the effector 400 (such as peripherals) can be fed back to the collaborative control module. The process of the collaborative control module performing collaborative control is described below.

[0164] Specifically, the collaborative control module can obtain the brain instructions or brain states fed back by the decoding device 300 to determine the user's motion intention. Specifically, the collaborative control module can perform motion planning based on the brain instructions or brain states to determine the user's motion intention. For ease of distinction, the present application will record the motion intention determined by motion planning based on the decoded brain instructions or brain states as the first motion intention. The collaborative control module can also obtain the action state of the action instruction fed back by the effector 400 (such as peripherals such as a mechanical arm). In some examples, the action state may include a posture. The collaborative control module can predict the user's motion intention based on the action instruction or action state. Specifically, the collaborative control module can combine the motion instruction or motion state of the current control cycle with the motion instruction or motion state of the historical control cycle stored in the storage module, and predict the user's motion intention based on an algorithm (such as a Kalman filter, an AI model). Among them, the motion intention predicted based on the motion instruction or motion state is also referred to as the second motion intention. The collaborative control module can determine the ratio of the first motion intention to the second motion intention, and issue the ratio to the decoding device 300.

[0165] The collaborative control module may compare the difference between the first motion intention and the second motion intention, determine the accuracy of the decoded user's motion intention, and adjust the ratio of the first motion intention to the second motion intention accordingly based on the accuracy.

[0166] When the collaborative control module adopts the time-sharing collaborative control mode, the collaborative control module can decide a motion intention as the final motion intention. In other words, the ratio of the first motion intention to the second motion intention is 100%:0% or 0%:100%. For example, if the difference between the first motion intention and the second motion intention is less than the preset threshold, it means that the accuracy is high, and the motion intention of the user determined by the decoding method is reliable. The first motion intention determined by the decoding method can be selected, that is, the ratio of the first motion intention to the second motion intention is 100%:0%. If the difference between the first motion intention and the second motion intention is greater than the preset threshold, it means that the accuracy is low, and the reliability of the motion intention of the user determined by the decoding method is low. The predicted second motion intention can be selected, that is, the ratio of the first motion intention to the second motion intention is 0%:100%. The coordinated control module sends the above ratio to the decoding device 300.

[0167] When the collaborative control module adopts the adaptive collaborative control mode, the collaborative control module can use multiple motion intentions to obtain the final motion intention. In other words, the ratio of the first motion intention to the second motion intention is x%: (100-x)%, where x is (0, 100). Specifically, the collaborative control module can obtain Nash equilibrium through a dynamic decision tree based on the paths corresponding to different motion intentions. For example, the collaborative control module can calculate the benefits brought by each decision (such as each ratio) based on the cost-benefit function, and calculate the Nash equilibrium point of the dynamic decision tree based on the benefits. The ratio corresponding to the Nash equilibrium point can be used as the ratio of the first motion intention to the second motion intention. The collaborative control module can send the ratio to the decoding device 300 so that the decoding device 300 decodes according to the ratio and controls the peripherals based on the decoding results.

[0168] It should be noted that, when performing collaborative control, the collaborative control module may obtain the motion intention based on the feedback signal of the current control cycle, or may obtain the motion intention based on the feedback signal of the historical control cycle, and this embodiment does not impose any limitation on this.

[0169] Next, see Fig. 9 A schematic diagram of a signal processing method is shown in FIG. Fig. 9 As shown, the post-processing module 204 in the signal processing device 200 may include a stable channel selection (also referred to as a stable channel selection, Stable Channel Selection) module.

[0170] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 can be fed back to the stable channel selection module. The process of the stable channel selection module performing stable channel selection (or stable channel screening) is described below.

[0171] In specific implementation, the stable release channel selection module can obtain the brain instructions or brain state fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions (such as the target brain state or brain instructions stored in the storage module 208).

[0172] When the decoding accuracy meets the first set condition, the first set condition may be that the decrease in the decoding accuracy in the set time period is greater than the set amplitude, indicating that there is a sudden and substantial decrease in the decoding accuracy (also called a plummet). Considering that the plummet in decoding accuracy may be caused by electrode inactivation or changes in the emission characteristics, the stable emission channel selection module may determine the emission stability of multiple acquisition channels, and send information of the acquisition channels whose emission stability meets the requirements (such as FilteredStable Signals) to the decoding device 300, so that the decoding device 300 decodes the first signal according to the acquisition channels whose emission stability meets the requirements.

[0173] The stable distribution channel selection module can evaluate all acquisition channels based on the set distribution stability judgment standard, so as to determine the acquisition channel whose stability meets the requirements, that is, the stable distribution channel. The stable distribution channel selection module selects a stable distribution channel and transmits the information of the selected stable distribution channel (such as the channel identifier) ​​to the decoding device 300.

[0174] Then, see Fig.10 A schematic diagram of a signal processing method is shown in FIG. Fig.10 As shown, the post-processing module 204 in the signal processing device 200 may include a task parameter update module, also called a paradigm update module.

[0175] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 can be fed back to the task parameter update module. The following is an explanation of the process of the task parameter update module performing task parameter update (or paradigm update, paradigm optimization).

[0176] Specifically, the task parameter update module can obtain the brain instructions or brain state fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain state fed back by the decoding device 300 and the locally stored target brain state or target brain instructions (such as the target brain state or brain instructions stored in the storage module 208).

[0177] When the decoding accuracy meets the second set condition, the second set condition may be that the decoding accuracy is continuously lower than the accuracy threshold in a set time period, indicating that the decoding accuracy is continuously low for a period of time. The reason why the decoding accuracy continues to be maintained at a low level may be that the existing task is too challenging / difficult for the user. Based on this, the task parameter update module can update the task parameters and send a first signal (also called Task Parameter-Modulated Signals) modulated based on the updated task parameters to the decoding device 300. The purpose of adjusting the task parameters is to adjust the difficulty of the task. Taking the robot arm control as an example, adjusting the task parameters may be to adjust the movement accuracy of the robot arm, for example, from 1 millimeter (millimeter, mm) to 1 centimeter (centimeter, cm).

[0178] Next, see Fig.11 A schematic diagram of a signal processing method is shown in FIG. Fig.11 As shown, the post-processing module 204 in the signal processing device 200 may include a data integration module, also called a pre-trained model loading module.

[0179] In this scenario, the brain-computer interface system 10 is used to implement brain-controlled peripherals. The brain instructions or brain states decoded by the decoding device 300 can be fed back to the data integration module. The following is an explanation of the process of data integration (pre-training model loading, fine tuning) performed by the data integration module.

[0180] Specifically, in the offline state, in order to improve the generalization of decoding, the data integration module may consider integrating data across days, across users (subjects), and multimodal data to pre-train the decoding model, and retain the parameters of the pre-trained model. The parameter may include the weight of the pre-trained model. In some examples, the parameter may also include hyperparameters of the pre-trained model, such as optimizers and learning rates. During online decoding, the data integration module can obtain the brain instructions or brain states fed back by the decoding device 300, and determine the decoding accuracy based on the brain instructions or brain states fed back by the decoding device 300 and the locally stored target brain states or target brain instructions (such as the target brain states or brain instructions stored in the storage module 208).

[0181] When the decoding accuracy meets the third set condition, the third set condition may be that the decoding accuracy is continuously lower than the accuracy threshold in a set time period, indicating that the decoding accuracy is continuously low for a period of time, and the data integration module may consider loading the pre-trained model and fine-tuning it based on a small amount of data on that day. Specifically, the data integration module may use the first signal to fine-tune the pre-trained model, obtain updated decoding parameters, and send the updated decoding parameters to the decoding device 300. The pre-trained model is obtained by pre-training the decoding model in the decoding device 300 based on historical data (such as cross-day, cross-user, and multi-modal data).

[0182] Based on the aforementioned brain-computer interface system 10, the present application also provides another signal processing method. The following is an introduction to another signal processing method of the present application in conjunction with an embodiment.

[0183] Fig.12 Flow chart of a signal processing method shown in FIG. 1 , which can be performed by an effector 400 in a brain-computer interface system 10 , wherein the effector 400 includes a stimulation device, and the method includes the following steps:

[0184] S1202. The stimulation device obtains an external input signal of at least one modality collected by the prosthesis.

[0185] A prosthesis is also called a prosthetic implant. It is a man-made device used to replace a body part lost due to trauma, disease or congenital disorder. Prostheses can be divided into different types according to the body part they replace, including but not limited to visual prostheses, auditory prostheses, lower limb prostheses, and upper limb prostheses. The prosthesis can be in the form of a wearable device, for example, a visual prosthesis can include but is not limited to smart glasses.

[0186] The wearable device may deploy a sensor of at least one modality, and the modality may include but is not limited to a visual modality, an auditory modality, or a tactile modality. The wearable device is used to collect an external input signal of at least one modality through a sensor of at least one modality. Taking the prosthesis as an example of smart glasses, the smart glasses may deploy an image sensor (such as a camera), which may collect environmental images, and recognize the environmental images through an image recognition model to obtain obstacle information or traffic light status information, and the stimulation device may obtain the above-mentioned obstacle information or traffic light status information. Based on this, the external input signal of at least one modality may include obstacle information or traffic light status information. Among them, the obstacle information includes obstacle size, obstacle location, and obstacle type (fixed or movable). When the obstacle type is movable, the obstacle information may also include moving speed and moving direction. The traffic light status information may include traffic light color and traffic light switching waiting time.

[0187] S1204: The stimulation device encodes an external input signal of at least one modality to obtain an encoding result.

[0188] The encoding result includes the parameters of the stimulation mode that matches the modality. The stimulation mode includes but is not limited to light stimulation mode, electrical stimulation mode and ultrasonic stimulation mode. The stimulation device can store the correspondence between the modality and the stimulation mode. When an external input signal of at least one modality is obtained, the stimulation mode that matches the modality can be determined according to the above correspondence. For example, when the input signal is a visual modality, the stimulation mode can be an electrical stimulation mode. The stimulation device can encode the external input signal according to the corresponding stimulation mode through an encoder to obtain the encoding result.

[0189] S1206. The stimulation device generates a stimulation signal based on the encoding result.

[0190] Specifically, the stimulation device can generate a stimulation signal of a corresponding stimulation mode based on the parameters of the stimulation mode in the encoding result. The stimulation signal can be used to stimulate brain tissue, and the brain tissue can generate a neural signal when receiving the stimulation signal output by the stimulation device, thereby achieving brain perception.

[0191] Still taking the example of the smart glasses scenario, the stimulation device generates a stimulation signal based on the encoding result. The stimulation signal can stimulate the area of ​​the brain tissue responsible for vision, so that the blind person wearing the smart glasses can perceive the position and size of obstacles or the status of traffic lights, and then avoid obstacles or pass through when the traffic light is green.

[0192] Fig.12 The embodiment shown is described by taking the stimulation device independently generating the stimulation signal as an example. In other possible implementations of the present application, the stimulation device may generate the stimulation signal in combination with the collected information. The following is described in combination with another embodiment.

[0193] See also Fig.13 Flow chart of a signal processing method shown in FIG. 1 , which can be performed by an effector 400 in a brain-computer interface system 10 , wherein the effector 400 includes a stimulation device, and the method includes the following steps:

[0194] S1302 : The stimulation device receives an internal input signal of at least one modality generated by the decoding device 300 .

[0195] Specifically, when the decoding device 300 decodes a set brain state or brain instruction, it can generate an internal input signal of at least one modality. For example, the brain state can be a state in which the user is about to have an epilepsy, and the decoding device 300 can generate an internal input signal to suppress epilepsy.

[0196] S1304. The stimulation device encodes an internal input signal of at least one modality to obtain an encoding result.

[0197] The encoding result includes parameters of the stimulation mode that matches the modality. The stimulation mode includes but is not limited to a light stimulation mode, an electrical stimulation mode, and an ultrasonic stimulation mode. The stimulation device can store the correspondence between the modality and the stimulation mode, and when an internal input signal of at least one modality is obtained, the stimulation mode that matches the modality can be determined according to the above correspondence.

[0198] S1306. The stimulation device generates a stimulation signal based on the encoding result.

[0199] The stimulation device can generate a stimulation signal of a corresponding stimulation mode based on the parameters of the stimulation mode in the encoding result. The stimulation signal can be used to stimulate brain tissue. The brain tissue can generate a neural signal when receiving the stimulation signal output by the stimulation device, and brain control can be achieved based on the neural signal. For example, the stimulation signal can stimulate the area of ​​the brain tissue responsible for movement, thereby inhibiting the user from having epilepsy.

[0200] Based on the aforementioned method embodiment, the present application also provides a signal processing device 200. Fig.14 As shown, the device 200 includes:

[0201] An acquisition unit 1402 is used to acquire a first signal, where the first signal is a signal collected from brain tissue, and to acquire a downstream feedback signal;

[0202] The processing unit 1404 is configured to process the first signal according to the feedback signal.

[0203] Exemplarily, the acquisition unit 1402 and the processing unit 1404 can be implemented by software. When implemented by software, the acquisition unit 1402 and the processing unit 1404 can be applications running on a computing device. The application can be provided to users in the form of virtualized services. Virtualization services may include virtual machine (VM) services, bare metal server (BMS) services, and container services. Among them, the VM service can be a service that virtualizes a virtual machine (VM) resource pool on multiple physical hosts (such as computing devices) through virtualization technology to provide users with VMs on demand for use. The BMS service is a service that virtualizes a BMS resource pool on multiple physical hosts to provide users with BMS on demand for use. The container service is a service that virtualizes a container resource pool on multiple physical hosts to provide users with containers on demand for use. VM is a simulated virtual computer, that is, a logical computer. BMS is a high-performance computing service that can be elastically scalable, and its computing performance is no different from that of a traditional physical machine, and it has the characteristics of secure physical isolation. Container is a kernel virtualization technology that can provide lightweight virtualization to achieve the purpose of isolating user space, processes and resources. It should be understood that the VM service, BMS service and container service in the above virtualization services are only used as specific examples. In actual applications, the virtualization service can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0204] The acquisition unit 1402 and the processing unit 1404 may also be implemented by hardware. When implemented by hardware, the acquisition unit 1402 may be implemented by a communication interface, such as a transceiver, and the processing unit 1404 may be a processor, or a device implemented by an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0205] In some possible implementations, the downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

[0206] In some possible implementations, the first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to characterize feedback to a second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

[0207] In some possible implementations, the feedback signal includes the brain instructions or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instructions or action state output by the effector in response to the brain instructions or brain state at the second time.

[0208] In some possible implementations, the processing unit 1404 is specifically configured to:

[0209] When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

[0210] In some possible implementations, the processing strategy includes at least one of the following:

[0211] Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

[0212] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit 1404 is specifically configured to:

[0213] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0214] A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

[0215] In some possible implementations, the feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing unit 1404 is specifically configured to:

[0216] determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state;

[0217] A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

[0218] In some possible implementations, the feedback signal includes a brain instruction or a brain state at the second time, and the processing unit 1404 is specifically configured to:

[0219] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0220] When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

[0221] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit 1404 is specifically configured to:

[0222] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0223] When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

[0224] In some possible implementations, the feedback signal includes a brain instruction or a brain state, and the processing unit 1404 is specifically configured to:

[0225] Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state;

[0226] When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

[0227] In some possible implementations, the first signal is a digital signal, and the acquiring unit 1402 is specifically configured to:

[0228] acquiring neural signals collected from the brain tissue;

[0229] converting the neural signal into an analog signal;

[0230] The analog signal is converted into the digital signal by analog-to-digital conversion.

[0231] In some possible implementations, the processing unit 1404 is further configured to:

[0232] The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

[0233] In some possible implementations, the signal preprocessing module 202 is integrated or deployed in the hardware signal acquisition device 100, and the processing unit 1404 is specifically used for:

[0234] The analog signal or the digital signal is preprocessed on the hardware signal acquisition device 100 side through the signal preprocessing module 202 .

[0235] In some possible implementations, the neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0236] In some possible implementations, the prosthesis is a wearable device, the wearable device is deployed with a sensor of at least one modality, and the wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

[0237] In some possible implementations, the neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

[0238] The present application also provides a computing device 1500. Fig.15 As shown, computing device 1500 includes: bus 1502, processor 1504, memory 1506 and communication interface 1508. Processor 1504, memory 1506 and communication interface 1508 communicate through bus 1502. Computing device 1500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 1500.

[0239] The bus 1502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 The bus 1502 may include a path for transmitting information between various components of the computing device 1500 (eg, the memory 1506, the processor 1504, and the communication interface 1508).

[0240] The processor 1504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0241] The memory 1506 may include a volatile memory, such as a random access memory (RAM). The memory 1506 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to implement the aforementioned signal processing method. Specifically, the memory 1506 stores instructions for the signal processing device 200 or the brain-computer interface system 10 to execute the signal processing method.

[0242] The communication interface 1508 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1500 and other devices or communication networks.

[0243] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0244] like Fig.16 As shown, the computing device cluster includes at least one computing device 1500. The memory 1506 in one or more computing devices 1500 in the computing device cluster may store the same signal processing apparatus 200 or brain-computer interface system 10 for executing instructions of the signal processing method.

[0245] In some possible implementations, one or more computing devices 1500 in the computing device cluster may also be used to execute some instructions of the signal processing device 200 or the brain-computer interface system 10 for executing the signal processing method. In other words, the combination of one or more computing devices 1500 may jointly execute the instructions of the signal processing device 200 or the brain-computer interface system 10 for executing the signal processing method.

[0246] It should be noted that the memory 1506 in different computing devices 1500 in the computing device cluster can store different instructions for executing partial functions of the signal processing device 200 or the brain-computer interface system 10.

[0247] The present application also provides a chip. The chip may include a processor and a communication interface, wherein the communication interface is used to obtain a first signal, wherein the first signal is a signal collected from brain tissue, and the processor is used to execute computer-readable instructions to perform the aforementioned signal processing method.

[0248] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned signal processing method applied to the signal processing device 200 or the brain-computer interface system 10.

[0249] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the above-mentioned signal processing method.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A signal processing method, characterized in that: The method comprises: Acquiring a first signal, wherein the first signal is a signal collected from brain tissue, and acquiring a downstream feedback signal; The first signal is processed according to the feedback signal.

2. The method according to claim 1, characterized in that The downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

3. The method according to claim 1 or 2, characterized in that: The first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback to the second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

4. The method according to claim 3, characterized in that: The feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

5. The method according to any one of claims 1 to 4, characterized in that: The processing of the first signal according to the feedback signal includes: When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

6. The method according to claim 5, characterized in that The processing strategy includes at least one of the following: Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

7. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing of the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

8. The method according to claim 5, characterized in that The feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing of the first signal by a processing strategy corresponding to the trigger condition includes: determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state; A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

9. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

10. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

11. The method according to claim 5, characterized in that The feedback signal includes a brain instruction or a brain state. When determining that a trigger condition is satisfied according to the feedback signal, processing the first signal by a processing strategy corresponding to the trigger condition includes: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

12. The method according to any one of claims 1 to 11, characterized in that: The first signal is a digital signal, and obtaining the first signal includes: acquiring neural signals collected from the brain tissue; converting the neural signal into an analog signal; The analog signal is converted into the digital signal by analog-to-digital conversion.

13. The method according to claim 12, characterized in that The method further comprises: The analog signal or the digital signal is preprocessed to achieve signal compression.

14. The method according to claim 13, characterized in that The preprocessing of the analog signal or the digital signal comprises: The analog signal or the digital signal is preprocessed on the hardware signal acquisition device side.

15. The method according to any one of claims 12 to 14, characterized in that The neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

16. The method according to claim 15, characterized in that The prosthesis is a wearable device, and the wearable device is deployed with a sensor of at least one modality. The wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

17. The method according to any one of claims 12 to 14, characterized in that The neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

18. A signal processing device, characterized in that: The device comprises: An acquisition unit, configured to acquire a first signal, wherein the first signal is a signal collected from brain tissue, and acquire a downstream feedback signal; A processing unit is used to process the first signal according to the feedback signal.

19. The device according to claim 18, characterized in that The downstream feedback signal includes a feedback signal from a decoding device or a feedback signal from an effector.

20. The device according to claim 18 or 19, characterized in that The first signal is a signal collected from the brain tissue at a first time, the feedback signal is used to represent feedback to the second signal, and the second signal is a signal collected from the brain tissue at a second time, and the second time is earlier than the first time.

21. The device according to claim 18, characterized in that The feedback signal includes the brain instruction or brain state at the second time obtained by decoding the second signal by a decoding device, or the feedback signal includes the action instruction or action state output by the effector in response to the brain instruction or brain state at the second time.

22. The device according to any one of claims 18 to 20, characterized in that The processing unit is specifically used for: When it is determined according to the feedback signal that a trigger condition is met, the first signal is processed using a processing strategy corresponding to the trigger condition.

23. The device according to claim 22, characterized in that The processing strategy includes at least one of the following: Calibration, collaborative control, data integration, stable release channel screening or paradigm update.

24. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; A decoding parameter for decoding the first signal is updated according to the decoding accuracy.

25. The device according to claim 22, characterized in that The feedback signal includes at least one of a brain instruction or a brain state and at least one of an action instruction or an action state, and the processing unit is specifically used for: determining a first movement intention of the user according to the brain instruction or the brain state, and predicting a second movement intention of the user according to the action instruction or the action state; A ratio between the first motion intention and the second motion intention is determined, and the ratio is sent to a decoding device.

26. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state at a second time, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the first set condition, the emission stability of multiple acquisition channels is determined, and information of the acquisition channels whose emission stability meets the requirement is sent to the decoding device, so that the decoding device decodes the first signal according to the acquisition channels whose emission stability meets the requirement.

27. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the second set condition, the task parameters are updated, and a first signal modulated based on the updated task parameters is sent to the decoding device.

28. The device according to claim 22, characterized in that The feedback signal includes a brain instruction or a brain state, and the processing unit is specifically used for: Determining decoding accuracy according to the brain command or brain state in the feedback signal and the target brain command or target brain state; When the decoding accuracy meets the third set condition, the pre-trained model is fine-tuned using the first signal to obtain updated decoding parameters, and the updated decoding parameters are sent to the decoding device, wherein the pre-trained model is obtained by pre-training the decoding model in the decoding device based on historical data.

29. The device according to any one of claims 18 to 28, characterized in that The first signal is a digital signal, and the acquisition unit is specifically used for: acquiring neural signals collected from the brain tissue; converting the neural signal into an analog signal; The analog signal is converted into the digital signal by analog-to-digital conversion.

30. The device according to claim 29, characterized in that The processing unit is also used for: The analog signal or the digital signal is preprocessed by a signal preprocessing module to achieve signal compression.

31. The device according to claim 30, characterized in that The signal preprocessing module is integrated or deployed in a hardware signal acquisition device, and the processing unit is specifically used for: The analog signal or the digital signal is preprocessed by the signal preprocessing module on the hardware signal acquisition device side.

32. The device according to any one of claims 18 to 31, characterized in that The neural signal includes a signal generated by a stimulation signal output by a stimulation device when the brain tissue is stimulated, and the stimulation signal is generated based on an encoding result of an external input signal of at least one modality acquired by a prosthesis, and the encoding result includes parameters of a stimulation pattern matching the modality.

33. The device according to claim 32, characterized in that The prosthesis is a wearable device, and the wearable device is deployed with a sensor of at least one modality. The wearable device is used to collect an external input signal of the at least one modality through the sensor of the at least one modality.

34. The device according to any one of claims 18 to 31, characterized in that The neural signal includes a signal generated when the brain tissue is stimulated by a stimulation signal output by a stimulation device, and the stimulation signal is generated based on an encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, and the encoding result includes parameters of a stimulation pattern matching the modality.

35. A chip, characterized in that: The chip includes a processor and a communication interface, the communication interface is used to obtain a first signal, the first signal is a signal collected from brain tissue, and the processor is used to execute computer-readable instructions to perform the signal processing method according to any one of claims 1 to 17.

36. A brain-computer interface system, characterized in that: The system includes a signal processing device and a hardware signal acquisition device. The signal processing device is used to execute the signal processing method described in any one of items 1 to 17 to process the signal output by the hardware signal acquisition device to achieve brain-computer interaction.

37. The system according to claim 36, characterized in that The signal processing device includes a signal preprocessing module, which is integrated and deployed in the hardware signal acquisition device, and the signal preprocessing module is used to: The signal output by the hardware signal acquisition device is preprocessed on the hardware signal acquisition device side.

38. The system according to claim 36 or 37, characterized in that The system further comprises: A stimulation device is used to generate a stimulation signal based on the encoding result of an external input signal of at least one modality acquired by a prosthesis, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

39. The system according to claim 36 or 37, characterized in that The system further comprises: A stimulation device is used to generate a stimulation signal based on the encoding result of an internal input signal of at least one modality generated when a decoding device decodes a set brain state or brain instruction, wherein the encoding result includes parameters of a stimulation pattern matching the modality, and the stimulation signal is used to stimulate brain tissue.

Images