Control method, implantable closed-loop neural stimulation system, and storage medium

Abstract

This application relates to the field of medical device technology, and discloses a control method, an implantable closed-loop neurostimulation system, and a storage medium. The method includes: acquiring multimodal data of a user; the multimodal data includes: the user's electroencephalogram (EEG) signals and the user's perception data collected by a terminal device; determining a collaborative control index based on the multimodal data; the collaborative control index indicating the user's state information; generating a first control command when the collaborative control index is within a first index range, and sending the first control command to a first external device; the first control command instructs the first external device to output a first control signal according to a target ratio of preset parameters; generating a second control command when the collaborative control index is within a second index range; the second control command instructs the terminal device to display preset interactive content.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and more specifically, to a control method, an implantable closed-loop neurostimulation system, and a storage medium. Background Technology

[0002] Implantable closed-loop neurostimulation systems, as advanced medical devices, have great development prospects. Currently, implantable closed-loop neurostimulation systems are usually equipped with a programmable terminal, which allows users to remotely control the system. This programmable terminal is similar to a dedicated "remote control".

[0003] However, existing programmable terminals typically only serve as passive instruction output devices. Their simple structure allows them to perform basic system control, and their control methods and functions are very limited, making it difficult to adapt to more complex and dynamic intelligent application scenarios. Summary of the Invention

[0004] This application provides a control method, an implantable closed-loop neurostimulation system, and a storage medium to address the problem that the control methods and functions of programmable terminals in the prior art are too limited.

[0005] To address the aforementioned problems, this application discloses a control method applied to a terminal device, the control method comprising:

[0006] Acquire user's multimodal data; wherein, the multimodal data includes: the user's electroencephalogram (EEG) signals and the user's perceptual data collected by the terminal device;

[0007] Based on the multimodal data, a collaborative control index is determined; wherein the collaborative control index indicates the user's status information;

[0008] When the collaborative control index is within the first index range, a first control command is generated and sent to a first external device; wherein, the first control command is used to instruct the first external device to output a first control signal according to a target ratio of preset parameters, and the target ratio is less than 1.

[0009] When the collaborative control index is within the range of the second index, a second control instruction is generated; wherein the second control instruction is used to instruct the terminal device to display preset interactive content.

[0010] This application also discloses a terminal device, the terminal device comprising:

[0011] An acquisition module is used to acquire the user's multimodal data; wherein, the multimodal data includes: the user's electroencephalogram (EEG) signals and the user's perceptual data collected by the terminal device;

[0012] The indicator module is used to determine collaborative control indicators based on the multimodal data; wherein the collaborative control indicators indicate the user's status information.

[0013] A first instruction module is configured to generate a first control instruction when the collaborative control index is within a first index range, and send the first control instruction to a first external device; wherein, the first control instruction is configured to instruct the first external device to output a first control signal according to a target ratio of a preset stimulation parameter, wherein the target ratio is less than 1.

[0014] The second instruction module is used to generate a second control instruction when the collaborative control indicator is within the range of the second indicator; wherein the second control instruction is used to instruct the terminal device to display preset interactive content.

[0015] This application also discloses an implantable closed-loop neurostimulation system, including a first external device and a terminal device as described above, wherein the first external device is a neurostimulator.

[0016] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements one or more of the methods described in this application.

[0017] This application also discloses a computer program product, including a computer program that, when executed by a processor, implements one or more of the methods described in this application.

[0018] The beneficial effects of the technical solutions provided in this application are:

[0019] In this embodiment, by acquiring multimodal user data, especially combining EEG signals with sensory data collected by the terminal device, more comprehensive and accurate user state information can be obtained than that from a single modality. Based on the multimodal data, collaborative control indicators are determined, enabling the terminal device to move beyond simple command output and acquire the ability to analyze and evaluate user state. Furthermore, by determining whether the collaborative control indicators fall within a first or second indicator range, different control strategies can be automatically and dynamically selected: when intervention with external devices is required, a first control command is generated and sent to the first external device, and by instructing the first external device to output a first control signal according to a target ratio of preset parameters (where the ratio value is less than 1), fine-grained control is achieved, improving the adaptability and security of the control; when direct interaction with the user is required, a second control command is generated to instruct the terminal device to display preset interactive content, thereby expanding the interactive functions of the terminal device. Compared to existing programmable terminals, the terminal device using the control method of this application can perform multimodal perception, intelligent state assessment, and execute differentiated adaptive control accordingly, possessing not only multiple control methods but also more functions. Attached Figure Description

[0020] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0021] Figure 1 A flowchart of the control method provided in the embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0023] Figure 3 This is a schematic diagram of the structure of the implantable closed-loop neurostimulation system provided in the embodiments of this application. Detailed Implementation

[0024] The embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the embodiments described below with reference to the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions of the embodiments of this application.

[0025] Those skilled in the art will understand that, unless otherwise stated, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the terms “comprising” and “including” as used in the embodiments of this application mean that the corresponding feature can be implemented as the presented feature, information, data, step, operation, element, and / or component, but do not exclude implementation as other features, information, data, step, operation, element, component, and / or combinations thereof supported by the art. It should be understood that when we say that an element is “connected” or “coupled” to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. Furthermore, “connected” or “coupled” as used herein can include wireless connection or wireless coupling. The term “multiple” refers to two or more; therefore, in the embodiments of this application, “multiple” can also be understood as “at least two.” The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / ", unless otherwise specified, generally indicates that the related objects before and after it are in an "or" relationship.

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0027] To facilitate understanding of the technical solution of this application, the following terms will be introduced.

[0028] An electroencephalogram (EEG) is formed by summing the postsynaptic potentials that occur synchronously among a large number of neurons during brain activity. It records the changes in electrical waves during brain activity and is a comprehensive reflection of the electrophysiological activity of brain nerve cells on the surface of the cerebral cortex or scalp. It can also be called an electroencephalogram or brainwave.

[0029] An implantable closed-loop neurostimulation system can collect electroencephalogram (EEG) signals through electrodes placed near the epileptogenic focus, perform real-time analysis, and predict or monitor epileptic seizures. When abnormalities in the patient's EEG signals are detected, electrical stimulation is automatically applied to the cortex or target brain region via electrodes to inhibit excessive synchronized firing of brain neurons, thereby suppressing epileptic seizures. This electrical stimulation, also known as an electrical stimulation signal, is the electrical signal used to stimulate the brain.

[0030] This application proposes a control method applied to a terminal device. In some embodiments, the terminal device includes smart electronic devices such as mobile phones, computers, and wearable devices. In some embodiments, the terminal device can be used as part of an implantable closed-loop neurostimulation system, replacing an existing programmable terminal. This implantable closed-loop neurostimulation system can be used for neuroscience research, brain-computer interface research, and treatment of targeted diseases.

[0031] like Figure 1 As shown, the control method includes:

[0032] Step 101: Obtain the user's multimodal data.

[0033] In this step, multimodal refers to the way information is expressed, communicated, and understood using multiple different forms or sensory channels, and multimodal data is generated accordingly. In the embodiments of this application, multimodal data can refer to data including different data types or different data sources. For example, multimodal data includes: the user's electroencephalogram (EEG) signals and the user's sensory data collected by the terminal device.

[0034] In some embodiments, the terminal device is communicatively connected to a first external device that can monitor the user's electroencephalogram (EEG) signals and transmit the EEG signals to the terminal device. For example, the first external device may be an implantable closed-loop neurostimulation system or a neurostimulator.

[0035] In some embodiments, perceived data may refer to physical or environmental data related to the user's state collected by the terminal device (or its connected sensors). For example, perceived data includes, but is not limited to: user motion posture data (such as stillness, walking, trembling) collected by the terminal device's accelerometer or gyroscope; ambient sound data or user voice collected by the microphone; image data collected by the camera (used to analyze the user's facial expressions or body movements); heart rate or blood oxygen data collected by the photoplethysmography (PPG) sensor; and location data collected by the positioning module. It is understood that all perceived data can be of one modality or multiple modalities. For example, image data and audio data collected by the terminal device are modal data of two different modalities.

[0036] Step 102: Determine the collaborative control index based on the multimodal data; wherein, the collaborative control index indicates the user's status information.

[0037] In this step, the collaborative control index can be understood as a quantified or semi-quantified parameter calculated based on multimodal data fusion, representing a high-level abstraction of the user's current overall state. In essence, the user's state information refers to a specific state category related to the device's control logic, identified by the collaborative control index. For example, this state information could be a focused state, a relaxed state, a state indicating an intention to initiate movement, a fatigued state, or a state in a noisy environment.

[0038] For example, in scenarios where terminal devices work in conjunction with implantable closed-loop neurostimulation systems, or where terminal devices control implantable closed-loop neurostimulation systems, the user's electroencephalogram (EEG) signals can, to some extent, characterize the user's current condition, such as whether the user is currently experiencing an epileptic seizure or is about to have one. Similarly, the user's perceptual data can also characterize their condition to some extent. Therefore, collaborative control indicators determined based on multimodal data can also indicate the user's condition or health status.

[0039] In some embodiments, the process of determining the system control parameters may include:

[0040] Data preprocessing: Filtering, denoising, and feature extraction are performed on the EEG signals to obtain EEG features; denoising and feature extraction (such as motion amplitude, heart rate variability, sound intensity, etc.) are performed on the sensory data to obtain sensory data features.

[0041] Feature fusion and computation: EEG features and sensory data features are input into a pre-defined computational model (e.g., a rule-based decision tree, machine learning model, or neural network) to obtain a co-control index. After training, this model can map these heterogeneous features into one or more "co-control indices." For example, the co-control index can be a scalar value, such as a numerical value ranging from 0 to 1, or it can be a vector.

[0042] Step 103: If the collaborative control index is within the range of the first index, generate the first control command and send the first control command to the first external device.

[0043] In this step, the first control command is used to instruct the first external device to output the first control signal according to the target ratio of the preset parameters, where the target ratio is less than 1.

[0044] In this embodiment, when the collaborative control index falls within a preset first index range, it indicates that the user is in a state where adjustments to external devices are needed, but gradual intervention is required. The first index range is a pre-defined threshold range corresponding to a specific control strategy. In some embodiments, when the collaborative control index ranges from 0 to 1, the first index range can be a higher value within that range; for example, the first index range can be from 0.5 to 0.7, but is not limited to this.

[0045] Preset parameters: These refer to the parameters stored inside the first external device (such as an implantable closed-loop neurostimulation system or neurostimulator) that correspond to a certain intervention mode. For example, the standard set values ​​for the amplitude, frequency, and pulse width of the stimulation current. These standard set values ​​are used to output according to the values ​​when the target condition (such as epilepsy) occurs to achieve the intervention mode.

[0046] Target ratio: is a value less than 1, for example, it can be 0.3, 0.5 or 0.7, but is not limited to these.

[0047] In some embodiments, when the synergistic control index is in a moderate range (e.g., 0.4-0.7), it may be necessary to control the first external device to perform a certain function as specified. For example, if the first external device is an implantable closed-loop neurostimulation system, and the synergistic control index is in a moderate range, it indicates that the user has a high risk of developing the disease. In this case, the implantable closed-loop neurostimulation system is controlled to output a very small electrical stimulation signal to the user to intervene in advance of the onset of the disease. The preset parameter can be the intensity of the electrical stimulation signal required when the disease occurs. Therefore, the first control signal is a smaller electrical stimulation signal to achieve pre-onset intervention.

[0048] Step 104: If the collaborative control index is within the range of the second index, generate the second control command.

[0049] In this step, the second control command is used to instruct the terminal device to display preset interactive content.

[0050] When the collaborative control indicator falls within a preset second indicator range, it indicates that the user is in a state where no external device intervention is required. The second indicator range is a pre-defined threshold range corresponding to a specific control strategy. In some embodiments, when the collaborative control indicator ranges from 0 to 1, the first indicator range can be a higher value within that range; for example, the first indicator range can be from 0.3 to 0.5, but it is not limited to this.

[0051] In some embodiments, when the synergistic control index is in a low-level range (e.g., 0.4-0.7), no external device is required; the terminal device itself can achieve the desired result. For example, the first external device is an implantable closed-loop neurostimulation system. If the synergistic control index is in a low-level range, it indicates that the user has a low risk of developing the disease. In this case, preset interactive content is displayed on the terminal device, allowing the user to adjust themselves through the preset interactive content, thereby intervening in the onset of the disease or improving their psychological state.

[0052] In some embodiments, the terminal device is equipped with a display device (such as a screen). After generating the second control command, the terminal device will display preset content on its screen. The preset interactive content may include: current system status report (such as normal collaboration indicators), user status feedback (such as detecting that you are in a relaxed state), simple questionnaire, next operation prompts, interactive graphical interface for user active control, and various display contents for mood relaxation, etc. This approach expands the role of the terminal device, making it not only a sender of control commands, but also a provider of information and an initiator of interaction, enriching the control dimensions.

[0053] To facilitate understanding, a non-limiting example scenario will be used below. Suppose that this method is applied to a mobile phone, and the first external device is an implantable closed-loop neurostimulation system. The mobile phone can communicate with the implantable closed-loop neurostimulation system, which can provide electrical stimulation to the user to treat epilepsy.

[0054] S100: The mobile phone receives EEG signals from the implanted closed-loop neurostimulation system via Bluetooth, and at the same time uses its own sensors to collect the user's image data and audio data.

[0055] S200: The processor within the terminal device calculates a collaborative control index based on EEG signals, image data, and audio data. The higher this index, the higher the user's risk of developing the disease.

[0056] S300: If the coordinated control index is within a low to medium range of the first index (e.g., 0.3-0.5), it indicates that the user has not yet developed symptoms but has a high risk of developing symptoms. At this time, the terminal device generates a first control command and sends it to the implantable closed-loop neurostimulation system. This command instructs the implantable closed-loop neurostimulation system to output an electrical stimulation signal (first control signal) at 60% (target ratio) of the standard stimulation parameters to provide moderate intervention, rather than directly applying maximum stimulation (stimulation intensity at the time of symptom onset).

[0057] S400: If the collaborative control index is in a very low range of the second index (e.g., 0.1-0.3), it means that the user has not developed the disease and the risk of developing the disease is very low. At this time, the terminal device generates a second control command to adjust the user's mood by displaying preset interactive content, without the need for an implantable closed-loop neurostimulation system.

[0058] In this embodiment, by acquiring multimodal user data, especially combining EEG signals with sensory data collected by the terminal device, more comprehensive and accurate user state information can be obtained than that from a single modality. Based on the multimodal data, collaborative control indicators are determined, enabling the terminal device to move beyond simple command output and acquire the ability to analyze and evaluate user state. Furthermore, by determining whether the collaborative control indicators fall within a first or second indicator range, different control strategies can be automatically and dynamically selected: when intervention with external devices is required, a first control command is generated and sent to the first external device, and by instructing the first external device to output a first control signal according to a target ratio of preset parameters (where the ratio value is less than 1), fine-grained control is achieved, improving the adaptability and security of the control; when direct interaction with the user is required, a second control command is generated to instruct the terminal device to display preset interactive content, thereby expanding the interactive functions of the terminal device. Compared to existing programmable terminals, the terminal device using the control method of this application can perform multimodal perception, intelligent state assessment, and execute differentiated adaptive control accordingly, possessing not only multiple control methods but also more functions.

[0059] In some embodiments, determining collaborative control indices based on multimodal data includes:

[0060] EEG signals and sensory data are time-aligned, and feature fusion is performed after time alignment to obtain fused features;

[0061] The fused features are processed by a state recognition model to output collaborative control indicators.

[0062] It should be noted that the data of different modalities in multimodal data come from different sources and usually have time synchronization problems. Therefore, in this embodiment of the application, time alignment is performed before feature processing of multimodal data, and the same time axis is used to measure the data of different modalities.

[0063] For example, EEG signals are typically acquired by dedicated bioelectrical acquisition devices (such as implantable closed-loop neurostimulation systems), while sensory data is acquired by sensors in terminal devices. These two methods may have issues such as different acquisition frequencies, inconsistent start times, or transmission delays. In this embodiment, time alignment refers to adjusting the EEG signal data stream and the sensory data stream to a unified time reference using a time synchronization algorithm. Time alignment methods may include, for example, using hardware synchronization signals (such as a common clock pulse); or using software algorithms (such as dynamic time warping or sliding window matching) to align modal data from different modalities to the same sampling time point or time window.

[0064] In this embodiment, feature fusion includes: after completing time alignment, extracting features from the aligned EEG signals and sensory data respectively, and then performing fusion based on the extracted features.

[0065] In some embodiments, feature fusion may include any of the following methods:

[0066] Feature-level fusion: The extracted EEG feature vectors and sensory feature vectors are directly concatenated into a higher-dimensional joint feature vector.

[0067] Decision-level fusion: Preliminary state assessments are performed based on EEG and sensory features respectively, and then the assessment results are weighted or fused by voting.

[0068] Model-level fusion: Using a well-designed neural network (such as a multi-branch input network) to automatically learn the association between two types of features and fuse them.

[0069] The fused features obtained through feature fusion can provide more comprehensive and robust inputs for the subsequent calculation of collaborative control indicators.

[0070] In some embodiments, the state recognition model is a pre-trained computational model that maps the fused features obtained in the previous step into "cooperative control indicators" that can characterize the user's state.

[0071] This state recognition module includes, but is not limited to, machine learning models and deep learning modules. For example, machine learning models can be support vector machines, random forests, gradient boosting decision trees, etc. These machine learning models use labeled historical fusion feature data (corresponding to known user state labels) to learn during the training phase and establish a mapping relationship between features and collaborative control indicators.

[0072] Understandably, through model training, the goal is to make the collaborative control indicators output by the state recognition model reflect the real user state as accurately as possible.

[0073] In this embodiment, by introducing time alignment, the temporal consistency between EEG signals and sensory data is ensured during analysis, eliminating noise and errors that may be introduced by asynchronous data and providing a reliable data foundation for subsequent analysis. Based on this, feature fusion is performed, breaking down information barriers between different modalities and forming complementary and mutually reinforcing fused features, resulting in a more comprehensive and accurate representation of the user's state. Furthermore, a pre-trained state recognition model is used to intelligently process the fused features. This model can learn and generalize complex data patterns, thereby outputting more accurate and stable collaborative control indicators.

[0074] In some embodiments, the state recognition model processes the fused features and outputs a collaborative control index, including:

[0075] Obtain personalized calibration parameters associated with the user; wherein, the personalized calibration parameters are used to indicate the contribution weight of different modal data in the multimodal data to the output collaborative control index;

[0076] Based on the personalized calibration parameters, the corresponding feature components in the fusion feature are weighted and fused to obtain the weighted fusion feature.

[0077] Based on the weighted fusion characteristics, output collaborative control indicators.

[0078] It should be noted that data from different modalities can be considered as data from different dimensions. Comprehensively considering data from different dimensions to determine collaborative control indicators can make these indicators more accurate. However, the contribution of data from different dimensions or modalities to the output collaborative control indicators may vary. Furthermore, for different individuals, the contribution of data from the same dimension or modality to the output collaborative control indicators may also differ. Therefore, to maximize the utilization rate of data from different dimensions or modalities, this application's embodiments introduce personalized standard parameters to calculate collaborative control indicators.

[0079] In some embodiments, personalized calibration parameters can be a set of values ​​pre-determined and stored for a specific user. Their function is to indicate the contribution weight of different modalities in the multimodal data to the output collaborative control index. For example, for user A, their motor intentions may be more clearly reflected in their EEG signals, so the contribution weight of the EEG signal modality in user A's personalized calibration parameters would be higher; while for user B, their limb tremors (sensory data) might be a more critical indicator, so the contribution weight of the sensory signal modality in user B's personalized calibration parameters would be higher.

[0080] The methods for obtaining personalized calibration parameters include, but are not limited to, the following:

[0081] The system guides users to perform a series of standard actions or enter different preset states (such as attempting movement or remaining still), simultaneously recording their multimodal data and expected state labels. By analyzing this data offline, the system calculates the optimal weight combination that makes the model output best match the user's true state and stores it as the user's personalized calibration parameters.

[0082] In some embodiments, the original fusion features are converted into weighted fusion features, which emphasize modal information that is more discriminative to the current user, while weakening modal information that may be noisy or weakly correlated, thereby forming a more personalized and purer state representation.

[0083] In some embodiments, weighted fused features are input into the core computational unit (such as a classifier or regressor) of the state recognition model. Since the input features have been personalized and weighted, the model can infer based on the feature combination that best suits the user, thereby outputting more accurate collaborative control indicators that better reflect the individual characteristics of the user.

[0084] In this embodiment, by acquiring personalized calibration parameters associated with the user, individual differences or differences in data from different modalities are incorporated into the control logic as core considerations, enabling the model to adapt to individual or data-dependent variations. Furthermore, based on the personalized calibration parameters, the corresponding feature components in the fused features are weighted and fused to generate weighted fused features that better reflect the user's true state. Finally, based on the weighted fused features, collaborative control indicators are output, ensuring that the final output is based not only on multimodal data but also on high-quality feature representations that have undergone personalized calibration.

[0085] In some embodiments, before obtaining personalized calibration parameters associated with the user, the method further includes:

[0086] If there are preset facial expression features associated with the user in the perceived data, the personalized calibration parameters are updated; in the updated personalized calibration parameters, the contribution weight of the modality data to which the facial expression features belong is increased.

[0087] It should be noted that the perceived data can be data containing user facial expression information, such as image data or video streams collected by the camera or infrared sensor of the terminal device.

[0088] Preset facial expression features refer to quantified features extracted from image data using image processing algorithms (such as facial landmark detection and neural networks for expression classification) that are associated with a specific emotional or intentional state. In other words, these preset facial expression features are the characteristics of a preset expression. The preset expression can be one or more pre-defined expressions, such as smiling, frowning, glaring, or panicking, but is not limited to these.

[0089] It is understandable that when there is data related to user facial expressions in the perceived data, or when user facial expressions are considered as a factor in the output of collaborative control indicators, one or more specific facial expressions of the user may be closely related to the user's state. For example, when a user displays a frown or a panicked expression, it is almost possible to determine that the user's state is poor, regardless of other factors. Therefore, in the process of calculating collaborative control indicators using personalized calibration parameters, updating the personalized calibration parameters first and then using the updated results to calculate the collaborative control indicators can further improve the accuracy of the calculation results.

[0090] In this embodiment, by analyzing preset facial expression features in the perceived data in real time, it is possible to capture strong state indication signals that a user may express through facial expressions at a specific moment. Upon detecting such features, personalized calibration parameters are proactively updated, and the contribution weight of the modality data to which the facial expression features belong is increased. This dynamic adjustment allows the generation process of collaborative control indicators to rely not only on long-term individualized models but also on short-term behavioral information, further improving the accuracy of collaborative control indicators.

[0091] In some embodiments, the method further includes:

[0092] Record the coordinated control indicators and the generated control commands;

[0093] Obtain device feedback information after the first external device executes the first control command, and / or user input information received after the terminal device displays preset interactive content;

[0094] The model parameters of the state recognition model are updated based on the recorded collaborative control indicators, control commands, equipment feedback information, and user input information.

[0095] It should be noted that the model parameters of the state recognition model play a decisive role in the model's performance. In this embodiment, the model parameters of the state recognition model can be updated each time the terminal device is used, using actual usage data and feedback information, in order to maintain or improve the performance of the state recognition model.

[0096] For example, during each execution of the control method, the state recognition model is trained according to the same processing operations used in model training, and the model parameters are updated.

[0097] In some embodiments, in the scenario of generating the first control command, the state or effect data after the command is executed can be obtained from a first external device (such as an implantable closed-loop neurostimulation system). For example, this may include: the amplitude of the stimulation signal, information on changes in the electroencephalogram (EEG) signal after the stimulation is executed, etc.

[0098] In some embodiments, in the scenario of generating the second control command, the user can view preset interactive information from the human-computer interaction interface of the terminal device and perform human-computer interaction, inputting or operating on the terminal device. For example, this may include: user confirmation, cancellation, rating, text input, or direct status feedback (such as feeling good, needing adjustment) provided through touch screen, voice, etc.

[0099] In some embodiments, recorded collaborative control indicators and control commands can be used as historical input-output pairs for the model. By combining the acquired device feedback information and user input information, an "effect evaluation" for the decision can be constructed. For example, if device feedback shows that the stimulus achieved the expected physiological response and the user input is positive, the decision can be considered a "correct" or "effective" sample; otherwise, it may be considered a sample that needs correction. Using the dataset with effect evaluation constructed above, the model parameters of the state recognition model are optimized and updated.

[0100] In this embodiment, a complete decision-making scheme can be established by recording collaborative control indicators and generated control commands. Then, by obtaining device feedback and user input after execution, each decision is given a practical effect label, thus transforming open-loop control into an evaluable closed loop. Finally, by updating the model parameters of the state recognition model based on this information, the internal mapping relationships of the model can be continuously corrected, maintaining or improving the model's performance.

[0101] In some embodiments, the first control command is also used to instruct at least one of the following:

[0102] Instruct the first external device to adjust the event detection level to a target level with higher sensitivity;

[0103] Instruct the first external device to perform the target task: wherein the target task includes: preparatory tasks before outputting the first control signal.

[0104] It should be noted that the event detection level can refer to the different sensitivity settings of the functional module integrated within the first external device (such as an implantable closed-loop neurostimulation system) for real-time monitoring of specific physiological events (e.g., epileptic events, abnormal tremors, arrhythmia events, etc.). Understandably, each level of the first external device corresponds to a set of preset detection thresholds and algorithm parameters. Lower levels (low sensitivity) may only detect very significant events to reduce false alarms, while higher levels (high sensitivity) can detect weaker or earlier event signs.

[0105] In some embodiments, when adjusting to a target setting with higher sensitivity, the first control command may include a sub-command that commands the first external device to immediately or within a specified time switch the current setting of its event detection module to the "target setting" with higher sensitivity.

[0106] In some embodiments, when the collaborative control index is within the range of the first index, it often means that the user is in a critical state requiring intervention (e.g., there is a risk of movement impairment but it has not yet fully occurred). At this time, pre-adjusting the event detection sensitivity can put the external device into a "high alert" state, enabling it to more sensitively capture subsequent, more subtle physiological changes, providing earlier warnings and decision-making basis for possible subsequent interventions (such as outputting an adjusted first control signal). The target task can, for example, refer to a series of operations that need to be completed by the first external device before the output action to ensure the final safe or effective output of the first control signal.

[0107] For example, preparatory tasks before outputting the first control signal may include:

[0108] Equipment self-test and status confirmation: Check whether the impedance of the stimulation electrode is within the normal range, whether the battery power is sufficient, and whether the circuit connection is stable.

[0109] Parameter calculation: Based on the "target ratio" specified in the first control command, the precise stimulation parameters are calculated in advance.

[0110] Safety circuit preheating or startup: Activates the safety monitoring circuit in the first external device, such as overcurrent protection and charge balance monitoring module, so that it enters the working state.

[0111] In this embodiment, instructing the adjustment of event detection sensitivity and instructing the preparation tasks before execution output can move the intelligent decision-making of the terminal device forward, improve the system's proactive early warning and preventive intervention level, and optimize execution efficiency and reliability.

[0112] In some embodiments, the preset interactive content includes at least one of guided text and graphics, guided audio, and breathing training animation.

[0113] It should be noted that guiding text and image content can refer to information presented in the form of static or dynamic combinations of text and images, which aims to guide users' emotions or behaviors.

[0114] For example, guiding text and image content may include:

[0115] Status feedback graphics and text: Presented with minimal text (such as "You are in a relaxed state detected") and corresponding icons (such as a calm wave icon).

[0116] Instructional guide with illustrations and text: Through step-by-step illustrations and text descriptions, guide users to complete a non-therapeutic self-regulation activity, such as a simple diagram of shoulder and neck relaxation steps.

[0117] Understandably, guiding text and image content is intuitive and easy to understand, and can quickly convey clear feedback or guidance information to users.

[0118] Guided audio can refer to pre-recorded or synthesized speech or other guiding sounds played through the audio output module of a terminal device.

[0119] For example, guiding audio may include:

[0120] Voice prompts: such as "Please maintain your current calm breathing rhythm".

[0121] Ambient sounds or music: Play specific background sounds (such as natural wind sounds, white noise) or soothing music clips that help you relax or focus.

[0122] Understandably, audio interaction can free up users' visual attention and enhance the convenience and immersion of the interaction.

[0123] Breathing training animations can refer to interactive content that visually guides users to practice regular breathing through dynamic graphics, pattern changes, or progress bars on a terminal device screen.

[0124] For example, breathing training animations may include:

[0125] Graphics of expansion and contraction: such as an animation simulating a lung or a flower that slowly expands with an inhalation command and slowly shrinks with an exhalation command.

[0126] Understandably, breathing training animations transform abstract breathing rhythms into intuitive visual targets for following.

[0127] In this embodiment, the diverse limitations on the specific forms of the preset interactive content elevate the execution effect of the second control command from a simple "information display" to various interactive information suitable for multiple scenarios. Whether through visual communication via guiding graphic content or auditory guidance via guiding audio, the rich interactive forms collectively expand the depth and breadth of the terminal device as a human-computer interface.

[0128] In some embodiments, the user perception data collected by the terminal device includes at least one of the following:

[0129] Facial features extracted from image data captured by the camera of the terminal device;

[0130] Acoustic features extracted from audio data collected by the microphone of the terminal device.

[0131] It should be noted that terminal devices can use their image sensors (such as cameras) to capture static images or continuous video streams containing the user's face, and then process the acquired image data in real time or near real time, using computer vision algorithms (such as face detection) to extract facial features from the raw image data.

[0132] In some embodiments, facial features may include:

[0133] Geometric features: such as the position and distance of key points like eyebrows, corners of the mouth, and corners of the eyes.

[0134] Appearance features: such as the texture and color of specific facial areas (forehead, around the eyes).

[0135] Advanced semantic features: such as identified facial expression categories (e.g., calm, focused, confused) and their confidence scores.

[0136] Terminal devices can also use their sound sensors (such as microphones) to collect ambient sounds, which may include the user’s voice, breathing sounds, or specific non-verbal sounds (such as coughing or sighing).

[0137] In some embodiments, the terminal device processes the acquired audio data and extracts acoustic features that reflect the user's state or intention from the original audio using digital signal processing technology or audio analysis models.

[0138] In some embodiments, acoustic features may include:

[0139] Voice features: such as tone of voice, speech rate, volume, and spectral features (such as fundamental frequency and formants) when a user speaks.

[0140] Non-speech features: such as the frequency, depth, and rhythm of breathing.

[0141] In this embodiment, facial features provide an intuitive representation of a user's expressions and potential emotions, while acoustic features provide another dimension of the user's emotions. By introducing these two features, the perceptual data is greatly enriched.

[0142] Based on the same principle as the method provided in the embodiments of this application, the embodiments of this application also provide a terminal device, such as... Figure 2 As shown, it includes:

[0143] The acquisition module 201 is used to acquire the user's multimodal data; wherein, the multimodal data includes: the user's electroencephalogram (EEG) signals and the user's sensory data collected by the terminal device;

[0144] The indicator module 202 is used to determine the collaborative control indicators based on multimodal data; wherein, the collaborative control indicators indicate the user's status information.

[0145] The first instruction module 203 is used to generate a first control instruction when the collaborative control index is within the range of the first index, and send the first control instruction to the first external device; wherein, the first control instruction is used to instruct the first external device to output a first control signal according to the target ratio of the preset stimulation parameters, and the target ratio is less than 1;

[0146] The second instruction module 204 is used to generate a second control instruction when the collaborative control indicator is within the range of the second indicator; wherein the second control instruction is used to instruct the terminal device to display preset interactive content.

[0147] In some embodiments, the indicator module 202 includes:

[0148] The fusion unit is used to time-align EEG signals with sensory data, and then perform feature fusion after time alignment to obtain fused features;

[0149] The model unit is used to process the fused features through the state recognition model and output collaborative control indicators.

[0150] In some embodiments, the model unit is specifically used for:

[0151] Obtain personalized calibration parameters associated with the user; wherein, the personalized calibration parameters are used to indicate the contribution weight of different modal data in the multimodal data to the output collaborative control index;

[0152] Based on the personalized calibration parameters, the corresponding feature components in the fusion feature are weighted and fused to obtain the weighted fusion feature.

[0153] Based on the weighted fusion characteristics, output collaborative control indicators.

[0154] In some embodiments, the terminal device further includes:

[0155] The personalized calibration parameter update module is used to update the personalized calibration parameters when there are preset facial expression features associated with the user in the perceived data; in the updated personalized calibration parameters, the contribution weight of the modality data to which the facial expression features belong is increased.

[0156] In some embodiments, the terminal device further includes:

[0157] The first information collection module is used to record collaborative control indicators and generated control commands;

[0158] The second information collection module is used to acquire device feedback information after the first external device executes the first control command, and / or user input information received by the terminal device after displaying preset interactive content;

[0159] The model parameter update module is used to update the model parameters of the state recognition model based on the recorded collaborative control indicators, control commands, equipment feedback information, and user input information.

[0160] In some embodiments, the first control command 203 is also used to instruct at least one of the following:

[0161] Instruct the first external device to adjust the event detection level to a target level with higher sensitivity;

[0162] Instruct the first external device to perform the target task: wherein the target task includes: preparatory tasks before outputting the first control signal.

[0163] In some embodiments, the preset interactive content includes at least one of guided text and graphics, guided audio, and breathing training animation.

[0164] In some embodiments, the user perception data collected by the terminal device includes at least one of the following:

[0165] Facial features extracted from image data captured by the camera of the terminal device;

[0166] Acoustic features extracted from audio data collected by the microphone of the terminal device.

[0167] The terminal device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method embodiments are not described in detail here to avoid repetition.

[0168] The terminal device provided in this application, by acquiring multimodal data of the user, especially combining EEG signals with sensory data collected by the terminal device, can obtain more comprehensive and accurate user state information than single-modal data. Based on the multimodal data, it determines collaborative control indicators, enabling the terminal device to move beyond being a simple command output end and instead possess the ability to analyze and evaluate the user's state. Furthermore, by determining whether the collaborative control indicators fall within a first or second indicator range, it can automatically and dynamically select different control strategies: when intervention with an external device is required, a first control command is generated and sent to the first external device, and by instructing the first external device to output a first control signal according to a target ratio of preset parameters (where the ratio value is less than 1), fine-grained control is achieved, improving the adaptability and safety of the control; when direct interaction with the user is required, a second control command is generated to instruct the terminal device to display preset interactive content, thereby expanding the interactive functions of the terminal device. Compared to existing programmable terminals, terminal devices using the control method of this application can perform multimodal perception, intelligent state assessment, and execute differentiated adaptive control accordingly. They not only have multiple control methods but also more functions, solving the technical problems of existing technologies having a single control method and being unable to adapt to complex dynamic intelligent application scenarios.

[0169] The terminal device in this application embodiment can execute the control method provided in this application embodiment. The implementation principle is similar. The actions performed by each module and unit in the terminal device in each embodiment of this application are corresponding to the steps in the control method in each embodiment of this application. For detailed functional descriptions of each module of the terminal device, please refer to the descriptions in the corresponding control methods shown above, which will not be repeated here.

[0170] Based on the same principles as the methods shown in the embodiments of this application, the embodiments of this application also provide an implantable closed-loop neurostimulation system, which includes the terminal device and the first external device provided in the above embodiments, wherein the first external device is a neurostimulator.

[0171] In an alternative embodiment, an implantable closed-loop neurostimulation system is also provided, such as Figure 3 As shown, Figure 3The implantable closed-loop neurostimulation system 3000 shown includes a processor 3001 and a memory 3003. The processor 3001 and memory 3003 are connected, for example, via a bus 3002. Optionally, the implantable closed-loop neurostimulation system 3000 may further include a transceiver 3004, which can be used for data interaction between the implantable closed-loop neurostimulation system and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 3004 is not limited to one type, and the structure of this implantable closed-loop neurostimulation system 3000 does not constitute a limitation on the embodiments of this application.

[0172] Processor 3001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 3001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0173] Bus 3002 may include a pathway for transmitting information between the aforementioned components. Bus 3002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 3002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0174] The memory 3003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium capable of carrying or storing computer programs and capable of being read by a computer, without limitation herein.

[0175] The memory 3003 is used to store computer programs that execute the embodiments of this application, and the execution is controlled by the processor 3001. The processor 3001 is used to execute the computer programs stored in the memory 3003 to implement the steps shown in the foregoing method embodiments.

[0176] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the steps and corresponding content of the aforementioned method embodiments.

[0177] This application also provides a computer program product, including a computer program that, when executed by a processor, can implement the steps and corresponding content of the aforementioned method embodiments.

[0178] The terms "first," "second," "third," "fourth," "1," "2," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in a sequence other than that shown in the illustrations or text descriptions.

[0179] It should be understood that although arrows indicate various operation steps in the flowcharts of this application's embodiments, the order in which these steps are implemented is not limited to the order indicated by the arrows. Unless explicitly stated herein, in some implementation scenarios of this application's embodiments, the implementation steps in each flowchart can be executed in other orders as required. Furthermore, some or all steps in each flowchart, based on the actual implementation scenario, may include multiple sub-steps or multiple stages. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage can also be executed at different times. In scenarios where execution times differ, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and this application's embodiments do not limit this.

[0180] The above description is only an optional implementation method for some implementation scenarios of this application. It should be noted that for those skilled in the art, other similar implementation methods based on the technical concept of this application without departing from the technical concept of this application also fall within the protection scope of the embodiments of this application.

Claims

1. A terminal device, characterized in that, The terminal device includes: An acquisition module is used to acquire the user's multimodal data; wherein, the multimodal data includes: the user's electroencephalogram (EEG) signals and the user's perceptual data collected by the terminal device; The indicator module is used to determine collaborative control indicators based on the multimodal data; wherein the collaborative control indicators indicate the user's status information. The first instruction module is configured to generate a first control instruction when the collaborative control index is within a first index range, and send the first control instruction to a first external device; wherein, the first control instruction is configured to instruct the first external device to output a first control signal according to a target ratio of a preset stimulation parameter, wherein the target ratio is less than 1; the first control instruction is also configured to instruct the first external device to adjust the event detection level to a target level with higher sensitivity. The second instruction module is used to generate a second control instruction when the collaborative control indicator is within the range of the second indicator; wherein the second control instruction is used to instruct the terminal device to display preset interactive content; the preset interactive content includes at least one of guided graphic content, guided audio, and breathing training animation; The indicator module includes: The fusion unit is used to time-align the EEG signals with the sensory data, and then perform feature fusion after time alignment to obtain fused features; A model unit is used to acquire personalized calibration parameters associated with the user; wherein, the personalized calibration parameters are used to indicate the contribution weight of data from different modalities in the multimodal data to the output of the collaborative control index; based on the personalized calibration parameters, the corresponding feature components in the fusion feature are weighted and fused to obtain a weighted fusion feature; based on the weighted fusion feature, the collaborative control index is output; The personalized calibration parameters are obtained in the following way: The system guides users to perform a series of standard actions or be in different preset states, and simultaneously records their multimodal data and expected state labels. The system analyzes the recorded multimodal data and expected state labels to obtain the optimal weight combination that best matches the user's actual state when the model unit outputs the data. The optimal weight combination is then stored as the user's personalized calibration parameters.

2. The terminal device according to claim 1, characterized in that, The terminal device further includes: The personalized calibration parameter update module is used to update the personalized calibration parameters when there are preset facial expression features associated with the user in the perceived data; wherein, in the updated personalized calibration parameters, the contribution weight of the modal data to which the facial expression features belong is increased.

3. The terminal device according to claim 1, characterized in that, The terminal device also includes: The first information collection module is used to record the collaborative control indicators and the generated control commands; The second information collection module is used to acquire device feedback information after the first external device executes the first control command, and / or user input information received by the terminal device after displaying the preset interactive content; The model parameter update module is used to update the model parameters of the state recognition model based on the recorded collaborative control indicators, control commands, equipment feedback information, and user input information.

4. The terminal device according to claim 1, characterized in that, The first control command is further used to instruct the first external device to perform a target task: wherein the target task includes a preparatory task before outputting the first control signal.

5. The terminal device according to claim 1, characterized in that, The user's perception data collected by the terminal device includes at least one of the following: Facial features extracted from image data captured by the camera of the terminal device; Acoustic features extracted from audio data collected by the microphone of the terminal device.

6. An implantable closed-loop neurostimulation system, characterized in that, It includes a first external device and a terminal device as described in any one of claims 1 to 5, wherein the first external device is a neurostimulator.

Images