Device control method, electronic device, storage medium and computer program product

Abstract

The embodiment of the invention discloses a device control method, an electronic device, a storage medium and a computer program product, and the method comprises the steps: judging whether the actual attention of an auxiliary object meets a preset attention requirement or not based on an electroencephalogram signal collected by an electroencephalogram signal collection device; under the condition that the judgment result is yes, determining a first target action corresponding to the electroencephalogram signal; otherwise, prompting the auxiliary object to improve attention; wherein the electroencephalogram signal is generated due to an intention associated with the first target action in the consciousness of the auxiliary object; control information is generated according to the first target action, and the control information is used for controlling movement of the peripheral auxiliary equipment to execute the first target action. According to the scheme, whether the attention of the auxiliary object is concentrated or not can be judged according to the electroencephalogram signals collected for the auxiliary object, and therefore the electroencephalogram signals which are high in analysis precision and suitable for determining the first target action can be collected when the attention of the auxiliary object is concentrated.

Description

Technical Field

[0001] This invention relates to the field of medical rehabilitation, and more specifically, to a control method, system, electronic device, storage medium, and computer program product for peripheral assistive devices. Background Technology

[0002] In recent years, with the development of robotics and communication technologies, control methods based on brain-computer interfaces (BCIs) for peripheral assistive devices have been increasingly applied in the field of medical rehabilitation. For patients who have lost motor function due to limb injuries, BCI-based control systems can help them control the movements of peripheral assistive devices such as wheelchairs, robotic arms, and exoskeletons. With the assistance of these devices, patients can perform rehabilitation training independently, significantly reducing the time and effort required from caregivers.

[0003] To make it easier for paralyzed patients to control peripheral assistive devices, a method has emerged that uses display devices to provide visual stimulation to patients, thereby generating the electroencephalogram (EEG) signals needed to issue control commands. This visual stimulation input method can reduce the difficulty of issuing commands to some extent. However, this approach requires patients to stare at the screen for extended periods, which can easily cause fatigue and a decline in their attention. When patients have low attention, the accuracy of EEG signal recognition also decreases, thereby reducing their ability to control peripheral assistive devices.

[0004] Currently, no effective solution has been proposed for the above problems. Summary of the Invention

[0005] The present invention addresses the aforementioned problems. It provides a device control method, an electronic device, a storage medium, and a computer program product. This solution can determine whether the attention of an assistive subject is focused based on the collected electroencephalogram (EEG) signals, thereby enabling the acquisition of highly accurate EEG signals suitable for determining a first target action when the subject's attention is focused.

[0006] According to one aspect of the present invention, a device control method is provided for controlling a peripheral assistive device that an assistive subject wishes to operate. The peripheral assistive device is communicatively connected to an electroencephalogram (EEG) signal acquisition device, which acquires the EEG signals of the assistive subject. The method includes: determining, based on the EEG signals acquired by the EEG signal acquisition device, whether the actual attention of the assistive subject meets a preset attention requirement; if the determination result is yes, determining a first target action corresponding to the EEG signal; otherwise, prompting the assistive subject to increase attention; wherein the EEG signal is generated due to the intention associated with the first target action in the consciousness of the assistive subject; and generating control information based on the first target action, wherein the control information is used to control the movement of the peripheral assistive device to execute the first target action.

[0007] Optionally, the method further includes: displaying icons corresponding to multiple actions via a display device, wherein the flashing frequency of the icons corresponding to different actions is different, and each icon corresponds to a text description of the corresponding action; wherein the brain signals acquired by the brain signal acquisition device are generated when the assisted subject observes the icon corresponding to the action to be performed, and the assisted subject's visual nerves are stimulated by the flashing of the observed icon; when determining the first target action corresponding to the brain signal, the flashing frequency corresponding to the icon observed by the assisted subject is determined based on the brain signal, and the action corresponding to the icon with the displayed frequency is determined as the first target action.

[0008] Optionally, while displaying icons corresponding to multiple actions, the transparency of the icons can be adjusted according to the attention of the auxiliary object.

[0009] Optionally, determining whether the actual attention of the assisting subject meets the preset attention requirements includes: determining the current attention value of the assisting subject based on the EEG signal; if the ratio of the current attention value to the preset attention reference value is higher than a preset ratio threshold, determining that the actual attention of the assisting subject meets the preset attention requirements; otherwise, determining that the actual attention of the assisting subject does not meet the preset attention requirements; and adjusting the transparency of the icon based on the attention of the assisting subject includes: increasing the transparency of the icon if it is determined that the actual attention of the assisting subject does not meet the preset attention requirements.

[0010] Optionally, the increased transparency is associated with the ratio of the current attention value to the preset attention reference value.

[0011] Optionally, the electromyography (EMG) signal acquisition device, which is connected to the peripheral assistive device, acquires EMG signals when the assisted object performs a specific action; identifies the specific action represented by the EMG signal, and determines the second target action corresponding to the identified specific action based on a pre-configured correspondence between multiple specific actions and multiple target actions; and generates control information based on the first target action, including generating control information based on the first target action and the second target action, wherein the control information is used to control the movement of the peripheral assistive device to perform a combination of the first target action and the second target action.

[0012] Optionally, identifying the specific action represented by the electromyographic signal includes: identifying the action type and action parameters of the specific action; and determining the second target action corresponding to the identified specific action based on the pre-configured correspondence between multiple specific actions and multiple target actions, including: determining the target action set corresponding to the identified action type based on the pre-configured correspondence between multiple action types and multiple target action sets; and selecting an action from the target action set as the second target action based on the identified action parameters.

[0013] Optionally, for each of the multiple target action sets, the target action set can be associated with other target action sets besides itself through multiple action types, and there is a one-to-one correspondence between the multiple action types and other target action sets; according to the pre-configured correspondence between the multiple action types and the multiple target action sets, the target action set corresponding to the identified action type is determined, including: determining the first target action set where the first target action is located, and according to the one-to-one correspondence between the multiple action types and the target action sets, determining the second target action set associated with the first target action set through the identified action type, the second target action set being the target action set subsequently used to determine the second target action.

[0014] Optionally, based on the pre-configured correspondence between various specific actions and various target actions, different action types are distinguished by different joints, and / or by different ways of movement.

[0015] Optionally, each action in the target action set is pre-configured with a correspondence between the action and the action parameters, which include at least one of the following: number of actions, action amplitude, and action frequency.

[0016] Optionally, the peripheral auxiliary equipment includes multiple joints, and there is a correspondence between the multiple joints and multiple preset action sets. Each preset action set includes one or more actions that the corresponding joint can perform. The multiple target action sets are at least a portion of the multiple preset action sets, and the target action set to which the first target action belongs is different from the target action set to which the second target action belongs.

[0017] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the device control method described above.

[0018] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which are used to execute the above-described device control method when running.

[0019] The above technical solution determines the actual attention of the assisted subject based on the subject's EEG signals and promptly prompts the assisted subject to increase attention when the actual attention does not meet the preset attention requirements. This can remind the assisted subject to concentrate when their attention declines. The EEG signals generated when the assisted subject is concentrating are more likely to match the first target action that the assisted subject needs to convey, which helps to reduce the error rate of determining the first target action based on EEG signals.

[0020] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0021] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0022] Figure 1 A schematic flowchart of a device control method according to an embodiment of the present invention is shown;

[0023] Figure 2 A schematic diagram of the display interface of a display device according to an embodiment of the present invention is shown;

[0024] Figure 3A schematic diagram of an electromyographic signal recognition pattern according to an embodiment of the present invention is shown;

[0025] Figure 4 A schematic block diagram of a robotic arm control system according to an embodiment of the present invention is shown;

[0026] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0028] To at least partially solve the above-mentioned technical problems, embodiments of the present invention provide a device control method, an electronic device, a storage medium, and a computer program product. This solution can determine whether the attention of the assistive subject is focused based on the electroencephalogram (EEG) signals collected from the assistive subject, and can determine the first target action to be performed by the peripheral assistive device based on the EEG signals of the assistive subject when the subject's attention is focused. Therefore, when the assistive subject's attention is focused, the brain signals generated can have high analytical accuracy, thereby allowing for a more accurate determination of the first target action.

[0029] Please see Figure 1 The diagram shown is a schematic flowchart of a device control method according to an embodiment of the present invention. According to one aspect of the present invention, a device control method is provided for controlling a peripheral assistive device that an assistive subject wishes to operate. The peripheral assistive device is communicatively connected to an electroencephalogram (EEG) signal acquisition device, which is used to acquire the EEG signals of the assistive subject. The method includes steps S110, S120, and S130.

[0030] For example, a peripheral assistive device can be any device capable of being moved by control information to perform a corresponding single action or combination of actions. A peripheral assistive device can be, for example, an exoskeleton robotic arm, or a traditional six-axis industrial robotic arm. This embodiment of the invention does not limit the specific type of peripheral assistive device. The device control method of this embodiment can be executed by a control device for the peripheral assistive device. This control device can be a component inherent to the peripheral assistive device itself, or it can be a device that is independent of the peripheral assistive device but communicatively connected to it. The peripheral assistive device can be communicatively connected to an EEG signal acquisition device and an EMG signal acquisition device respectively through the control device. The control device can be, for example, a microcontroller (MCU), a central processing unit (CPU), etc.

[0031] In one specific embodiment, the EEG signal acquisition device includes a reference electrode, a ground electrode, and an EEG acquisition electrode. The reference electrode can be placed at the top of the subject's head, and the ground electrode can be placed at the midpoint of the line connecting the frontal pole midpoint (Fpz) and the frontal midpoint (Fz). The EEG acquisition electrode can be placed in the frontal and occipital regions of the subject, referring to the electrode placement positions of the "EEG 10-20 system" specified by the International Society for Electroencephalography (ESE). More specifically, the EEG acquisition electrodes can be placed at the frontal midpoint (Fz), left frontal pole (Fz1), right frontal pole (Fz2), vertex (Pz), left vertex (P3), right vertex (P4), occipital point (Oz), left occipital (O1), and right occipital (O2), respectively, acquiring EEG signals from a total of 9 channels. It should be noted that the electrode placement in this embodiment is for illustrative purposes only and is not intended to limit the scope of the invention. The electrode placement should satisfy the condition that "when the subject intends to perform a different target action each time, the EEG signals collected by the entire EEG acquisition electrode are different from each other each time (at least some channels of the EEG signals are different from each other)".

[0032] In step S110, based on the EEG signals acquired by the EEG signal acquisition device, it is determined whether the actual attention of the assisted subject meets the preset attention requirements.

[0033] For example, in the acquired EEG signals, the EEG signal used to determine the first target action that the peripheral assistive device needs to perform and the EEG signal used to determine whether the actual attention of the assisted subject meets the preset attention requirements can be EEG signals from the same channel, or they can be EEG signals from different channels. In a specific embodiment, the acquired EEG signals may include EEG signals from a first channel and EEG signals from a second channel. The EEG signals from the first channel include the EEG signals acquired by the EEG acquisition electrodes located at Fz, Fz1, and Fz2, and the EEG signals from the second channel include the EEG signals acquired by the EEG acquisition electrodes located at Pz, P3, P4, Oz, O1, and O2. It can be understood that in this embodiment, the first channel includes three channels corresponding one-to-one with the three electrode placement positions, and the second channel includes six channels corresponding one-to-one with the six electrode placement positions. At least one channel of the first channel can be used to determine the actual attention of the assisted subject. The EEG signals from the second channel can be used to determine the first target action. This embodiment of the invention does not limit the specific channel of the EEG signal used to determine actual attention, nor does it limit the placement of the electrodes that can collect the EEG signal used to determine actual attention. Existing or future EEG signals that can be used to determine actual attention can be used in step S110 of this embodiment of the invention.

[0034] In a specific embodiment, the attention value corresponding to the EEG signal of each channel in the first channel can be calculated using the following formula:

[0035]

[0036] In the formula, `attention` represents the attention value corresponding to the channel, `P(α)` represents the energy of the EEG signal in the α frequency band, `P(β)` represents the energy of the EEG signal in the β frequency band, and `P(θ)` represents the energy of the EEG signal in the θ frequency band. The energy in each frequency band is equal to the sum of the squares of the sampling points of the EEG signal in that frequency band during the sampling period. For example, the actual attention of the assisted subject can be represented by the average of the attention values ​​corresponding to each channel of the first channel, or by the sum of the attention values ​​corresponding to each channel of the first channel, or by the maximum / minimum value among the attention values ​​corresponding to each channel of the first channel. In some embodiments, the actual attention of the assisted subject meeting the preset attention requirement may include: the value of the actual attention of the assisted subject is greater than or equal to a preset attention reference value. In other embodiments, the actual attention of the assisted subject meeting the preset attention requirement may include: the ratio between the value of the actual attention of the assisted subject and the preset attention reference value is greater than or equal to a preset ratio threshold. In some other embodiments, the fact that the actual attention of the assistive object meets the preset attention requirements may include: the numerical value of the assistive object's actual attention meets the preset numerical value requirement, and the duration of meeting the numerical value requirement meets the preset duration requirement. In this embodiment of the invention, the preset attention reference value may be a fixed value, or attention detection experiments may be performed on the assistive object in advance to determine the attention value of the assistive object when its attention is focused, and this attention value may be used as the preset attention reference value.

[0037] In step S120, if the judgment result is yes, the first target action corresponding to the EEG signal is determined; otherwise, the assisting subject is prompted to increase attention; wherein, the EEG signal is generated due to the intention of the assisting subject in his / her consciousness to be associated with the first target action.

[0038] For example, when the subject's actual attention meets the preset attention requirements, the first target action can be determined based on the EEG signal. Specifically, when the subject intends to perform a target action by consciously controlling a peripheral assistive device, an intention associated with that target action will be formed in their mind. It is understood that when this intention is formed, the subject's brain can generate a corresponding EEG signal, and the EEG signals corresponding to different target actions are different. In some embodiments, the subject can generate the corresponding EEG signal by imagining and forming an intention associated with the target action in their mind without the aid of external devices. In other words, the subject does not need to stimulate the brain to generate the EEG signal corresponding to the target action that the peripheral assistive device is expected to perform through external devices. In other embodiments, a display device can be used to display one or more icons that correspond one-to-one with one or more target actions. For each icon, Chinese characters representing the action corresponding to that icon can be further displayed. Each icon can be displayed in a different color, and / or each icon can be displayed at a different flashing frequency. The assistive subject gazes at an icon that corresponds to the target action that they expect to perform through conscious control of the peripheral assistive device. Because of the different colors and / or frequencies of the icons being gazed at, the brain can generate electroencephalogram (EEG) signals that are generated by the specific colors and / or frequencies of the icons stimulating the optic nerves.

[0039] For example, when the assisted subject's actual attention does not meet the preset attention requirements, attention prompts can be output through external indicator lights, speakers, display devices, etc., to prompt the assisted subject to improve their attention. For example, the control device can acquire EEG signals from a connected EEG signal acquisition device and perform feature recognition on the acquired EEG signals. The feature recognition methods used can be, for example, Canonical Correlation Analysis (CCA), Task-Related Component Analysis (TRCA), Temporal Discriminant Component Analysis (TDCA), etc. Before performing feature recognition on the EEG signals, the control device can optionally perform downsampling and / or filtering and / or notch filtering on the EEG signals. For example, it can downsample to 250Hz to reduce the complexity of the EEG signals, or it can perform filtering at 3-45Hz and / or notch filtering at 50Hz. The control device can determine the first target action corresponding to the acquired EEG signal from among multiple target actions according to a pre-configured one-to-one correspondence between multiple target actions and various EEG signals, based on the category of the identified EEG signal. The first target action is the target action to be executed by the peripheral assistive device.

[0040] In step S130, control information is generated based on the first target action, wherein the control information is used to control the movement of peripheral auxiliary equipment to perform the first target action.

[0041] By way of example and not limitation, the control device or peripheral auxiliary equipment can convert control information into control signals and send the control signals to the actuator of the peripheral auxiliary equipment (e.g., a robotic arm). When the actuator receives the control signal, it can autonomously plan a suitable motion path within the range of physical constraints to perform the first target action.

[0042] The above technical solution determines the actual attention of the assisted subject based on the subject's EEG signals and promptly prompts the assisted subject to increase attention when the actual attention does not meet the preset attention requirements. This can remind the assisted subject to concentrate when their attention declines. The EEG signals generated when the assisted subject is concentrating are more likely to match the first target action that the assisted subject needs to convey, which helps to reduce the error rate of determining the first target action based on EEG signals.

[0043] Optionally, the method further includes: displaying icons corresponding to multiple actions via a display device, wherein the flashing frequency of the icons corresponding to different actions is different, and each icon corresponds to a text description of the corresponding action; wherein the brain signals acquired by the brain signal acquisition device are generated when the assisted subject observes the icon corresponding to the action to be performed, and the assisted subject's visual nerves are stimulated by the flashing of the observed icon; when determining the first target action corresponding to the brain signal, the flashing frequency corresponding to the icon observed by the assisted subject is determined based on the brain signal, and the action corresponding to the icon with the displayed frequency is determined as the first target action.

[0044] For example, when an assistive subject gazes at any icon, the flashing of that icon can stimulate the subject's optic nerve, thereby generating a corresponding electroencephalogram (EEG) signal in the subject's brain. Different flashing frequencies produce different EEG signals. Therefore, by acquiring the EEG signals, the flashing frequency corresponding to the icon observed by the assistive subject can be determined, thereby identifying the target action corresponding to that icon. For example, the display device can also display a text description of the action corresponding to each icon; preferably, the text description is displayed near the corresponding icon. Please refer to [link to relevant documentation]. Figure 2 The diagram shown is a schematic representation of the display interface of a display device according to an embodiment of the present invention. Figure 2In the illustrated embodiment, five target action sets are pre-configured, each set including three target actions. For ease of understanding and description, the five target action sets are denoted as set A, set B, set C, set D, and set E, respectively. Set A includes target movements such as wrist flexion, wrist repositioning, and wrist extension, with corresponding icons flashing at frequencies of 8Hz, 9Hz, and 10Hz, respectively. Set B includes target movements such as elbow flexion, elbow repositioning, and elbow extension, with corresponding icons flashing at frequencies of 8.6Hz, 9.6Hz, and 10.6Hz, respectively. Set C includes target movements such as shoulder adduction, shoulder repositioning, and shoulder abduction, with corresponding icons flashing at frequencies of 8.4Hz, 9.4Hz, and 10.4Hz, respectively. Set D includes target movements such as forearm internal rotation, forearm stillness, and forearm external rotation, with corresponding icons flashing at frequencies of 8.2Hz, 9.2Hz, and 10.2Hz, respectively. Set E includes target movements such as upper arm internal rotation, upper arm stillness, and upper arm external rotation, with corresponding icons flashing at frequencies of 8.8Hz, 9.8Hz, and 10.8Hz, respectively. Figure 2 In the displayed interface, each icon is accompanied by a text description of the corresponding target action.

[0045] The above technical solution stimulates the visual nerves of the assisted subject by displaying icons with different flashing frequencies. This helps ensure that the assisted subject's brain can generate EEG signals to determine the first target action, which can reduce the difficulty for the assisted subject to control peripheral assistive devices through consciousness.

[0046] Optionally, while displaying icons corresponding to multiple actions, the transparency of the icons can be adjusted according to the attention of the auxiliary object.

[0047] For example, the assistive subject can concentrate on observing the icon corresponding to the target action for a period of time (e.g., 3 seconds), and the EEG signal acquisition device can collect the EEG signals evoked by the cerebral cortex of the assistive subject during this period. The control device can extract and decode the EEG signals to determine the assistive subject's attention. For example, the assistive subject's attention is used to adjust the transparency of the icon. In some embodiments, the transparency can change in real time with changes in the assistive subject's actual attention, and correspondingly, the change in transparency can reflect the assistive subject's actual attention. When attention decreases, the control device can control the display device to increase the transparency of the icon; the more transparent the icon, the lower the assistive subject's attention. In other embodiments, each icon can be displayed according to a first preset transparency in the initial state. If the actual attention does not meet the preset attention requirement, the icon is displayed according to a second preset transparency. Typically, the second preset transparency is greater than the first preset transparency. The transparency value ranges from 0 to 1, where 0 represents completely opaque and 1 represents completely transparent. In other embodiments, each icon can be displayed with a first preset transparency in the initial state. If the actual attention does not meet the preset attention requirements, the maximum transparency of the icon is a second preset transparency. The smaller the current attention value of the auxiliary object, the more the transparency of the icon is increased based on the second preset transparency.

[0048] The above technical solution can visually prompt the assisted user to concentrate by adjusting the transparency of the icon, and also helps to reflect the user's mental fatigue level.

[0049] Optionally, determining whether the actual attention of the assisting subject meets the preset attention requirements includes: determining the current attention value of the assisting subject based on the EEG signal; if the ratio of the current attention value to the preset attention reference value is higher than a preset ratio threshold, determining that the actual attention of the assisting subject meets the preset attention requirements; otherwise, determining that the actual attention of the assisting subject does not meet the preset attention requirements; and adjusting the transparency of the icon based on the attention of the assisting subject includes: increasing the transparency of the icon if it is determined that the actual attention of the assisting subject does not meet the preset attention requirements.

[0050] It is understood that the attention of the auxiliary object can be quantified as an attention value. The larger the current attention value, the larger the ratio between it and the preset attention reference value, indicating that the attention of the auxiliary object is more focused. The quantification method of the attention value can be referred to the relevant description in the foregoing embodiments, and will not be repeated here. The preset ratio threshold can be less than or equal to 1; in one embodiment, the preset ratio threshold is 0.5. In some embodiments, regardless of whether the actual attention of the auxiliary object meets the preset attention requirements, the transparency is negatively correlated with the current attention value; the larger the current attention value, the smaller the transparency. In other embodiments, when the actual attention of the auxiliary object does not meet the preset attention requirements, the transparency of the icon can be increased to a fixed preset value. In still other embodiments, when the actual attention of the auxiliary object does not meet the preset attention requirements, the transparency of the icon can be controlled to increase as the actual attention decreases. For example, when the actual attention of the auxiliary object meets the preset attention requirements, the transparency of the icon can be a fixed value, or it can decrease as the actual attention increases.

[0051] It's understandable that people tend to concentrate more when observing a target object that isn't clearly visible. Therefore, when the user's attention is not currently focused on the auxiliary object, increasing the transparency of the icon can not only prompt the user to focus their attention in a timely manner, but also help the user focus their attention to some extent.

[0052] Optionally, the increased transparency is associated with the ratio of the current attention value to the preset attention reference value.

[0053] For example, the smaller the current attention value, the greater the increased transparency can be. In a specific embodiment, by conducting attention testing experiments on the assistive subject beforehand, a preset attention reference value can be determined when the subject is focused. Transparency is equal to 1 - current attention value / preset attention reference value. When the actual current attention value is greater than the preset attention reference value, the ratio between the current attention value and the preset attention reference value is 1. This scheme can use the real-time transparency of the icon to provide feedback on the assistive subject's level of attention concentration, thereby helping the assistive subject and medical staff understand the subject's brain activity.

[0054] Optionally, the electromyography (EMG) signal acquisition device, which is connected to the peripheral assistive device, acquires EMG signals when the assisted object performs a specific action; identifies the specific action represented by the EMG signal, and determines the second target action corresponding to the identified specific action based on a pre-configured correspondence between multiple specific actions and multiple target actions; and generates control information based on the first target action, including generating control information based on the first target action and the second target action, wherein the control information is used to control the movement of the peripheral assistive device to perform a combination of the first target action and the second target action.

[0055] For example, an electromyography (EMG) signal acquisition device may include a ground electrode and a reference electrode, and the positions of the ground electrode and reference electrode of the EMG signal acquisition device are consistent with the positions of the ground electrode and reference electrode of the electroencephalogram (EEG) signal acquisition device. Alternatively, the EMG signal acquisition device may share the ground electrode and reference electrode with the EEG signal acquisition device. The EMG signal acquisition electrodes of the EMG signal acquisition device may be placed on one or more specific sites of the assistive subject, provided that the specific sites can generate different EMG signals when the assistive subject performs different specific actions. In a specific embodiment, the specific site is the wrist joint, and the electrodes of the EMG acquisition device are respectively placed on the palmaris longus and extensor carpi radialis muscles of the assistive subject. When the assistive subject performs wrist flexion, ulnar deviation, radial deviation, and extension actions, the EMG acquisition electrodes can acquire different EMG signals respectively.

[0056] For example, "performing a specific action" can exist in two scenarios: First, the assistive device can complete the specific action, generating muscle signals corresponding to that action during the process. Second, the assistive device cannot complete the specific action, but can control the muscles in the corresponding area to exert force based on the desired action, also generating electromyographic (EMG) signals corresponding to the specific action. The assistive device can also control peripheral assistive devices to perform a target action by executing a specific action. Specifically, the control device can acquire EMG signals from a connected EMG signal acquisition device and analyze the acquired EMG signals to identify the specific action performed by the assistive device based on the analysis results. The control device can determine the second target action corresponding to the identified specific action according to a pre-configured correspondence between multiple specific actions and multiple target actions; this second target action is another target action to be performed by the peripheral assistive device.

[0057] For example, for any two specific actions among a pre-configured set of specific actions, the action types of the two specific actions may be different, or the action parameters of the two specific actions may be different, or both the action types and action parameters of the two specific actions may be different. In some embodiments, the action types of the various specific actions are all different from each other; in this case, the action parameters of the various specific actions may be the same or different. In other embodiments, at least two specific actions with the same action type exist among the various specific actions; in this case, the action parameters of the at least two specific actions are different from each other. In still other embodiments, the action types of the various specific actions are all the same; in this case, the action parameters of the various specific actions are all different. For example, the signal characteristics of the electromyographic signals corresponding to the various specific actions are different. The control device performs feature analysis on the acquired electromyographic signals to determine the specific action to be performed by the assisted object based on the signal characteristics obtained from the feature analysis. Signal characteristics include, but are not limited to, amplitude characteristics, time-domain / frequency-domain characteristics, and energy distribution characteristics of the electromyographic signals corresponding to each muscle group.

[0058] For example, if at least some of the specific movements among a variety of specific movements have different movement types, an electromyography (EMG) acquisition device can be used to acquire EMG signals of muscle groups corresponding one-to-one with at least two channels. In some embodiments, the control device can compare the signal energy of the EMG signals from each channel and determine the movement type corresponding to the specific movement based on the comparison result. Different movement types may correspond to different comparison results. In a specific embodiment, EMG signals can be acquired through channels corresponding one-to-one with two muscle groups at the wrist, namely the palmaris longus and the extensor carpi radialis. The electromyographic signal (EMG) signal under the channel corresponding to the palmaris longus muscle is denoted as Pc1, and the EMG signal under the channel corresponding to the extensor carpi radialis muscle is denoted as Pc2. When the acquired EMG signals meet the condition that Pc1 > 3Pc2, the movement type of the specific action can be determined to be wrist flexion; when the acquired EMG signals meet the condition that 3Pc1 < Pc2, the movement type of the specific action can be determined to be wrist extension; when the acquired EMG signals meet the condition that Pc1 < Pc2 < 3Pc1, the movement type of the specific action can be determined to be radial wrist deviation; and when Pc2 < Pc1 < 3Pc2, the movement type of the specific action can be determined to be ulnar wrist deviation. It is understood that the above embodiments are only examples and not intended to limit the scope. In the embodiments of the present invention, if at least some of the specific actions among multiple specific actions have different movement types, the muscle group used to provide the EMG signals only needs to satisfy the condition that "when performing specific actions of different movement types, the movement type of the assisted object can be distinguished by the signal characteristics of the acquired EMG signals".

[0059] For example, for any two specific actions of a plurality of specific actions, if the two specific actions have the same action type but different action parameters, the signal characteristics such as signal frequency, signal amplitude, and number of energy peaks of the electromyographic signals acquired by the electromyographic signal acquisition device may be different when the assisted object performs the two specific actions respectively. Action parameters may be, for example, action frequency, action amplitude, or number of actions, etc., and this embodiment of the invention does not specifically limit the action parameters. For example, the number of second target actions can be one or more. When there are multiple second target actions, if at least two specific actions require sharing a muscle group, electromyographic signals can be acquired in at least two time windows to identify the specific actions performed by the assisted object respectively. If the muscle groups corresponding to the at least two specific actions are different from each other, the electromyographic signals of the muscle groups corresponding to the at least two actions can be acquired in the same time window to simultaneously determine the actions performed by the assisted object. For example, there is a pre-configured correspondence between the plurality of specific actions and the plurality of target actions, and a unique target action corresponding to each identified specific action can be determined. It is understandable that the subject performing a specific action is the auxiliary object, while the subject performing the target action is the external auxiliary equipment. The first target action and the second target action are usually different target actions.

[0060] For example, the control device can generate a first motion control command based on a determined first target action, and can generate a second motion control command based on a determined second target action. In some embodiments, the control device can generate control information based on the first motion control command, the second motion control command, and a pre-configured execution strategy. The execution strategy can be a composite execution strategy, in which case the control information can instruct the peripheral auxiliary device to execute the first target action and the second target action simultaneously. Alternatively, the execution strategy can be a sequential execution strategy, in which case the control information can instruct the peripheral auxiliary device to execute the first target action and the second target action in a preset order. The sequential execution strategy may include, for example, executing the first target action first and then the second target action, or vice versa. When there are multiple second target actions, the sequential execution strategy may also include executing each second target action sequentially according to a determined order. In other embodiments, the control device is a device independent of the peripheral auxiliary device. The peripheral auxiliary device itself has a pre-configured execution strategy, and the control device can generate control information based on the first motion control command and the second motion control command, the control information instructing the peripheral auxiliary device to execute the first target action and the second target action. In some embodiments, the assisted user or medical personnel can send an execution strategy to the control device via an interactive device. Specifically, by sending an execution strategy to the control device, the user can instruct the simultaneous execution of a first target action and a second target action, or the execution of the first target action and the second target action in a preset order. When the execution strategy sent by the user is a sequential execution strategy, the sequential execution strategy sent by the user may include a preset order; alternatively, if the control device or peripheral auxiliary device is pre-configured with a preset order, the sequential execution strategy sent by the user may not include the preset order. Exemplarily and not limitingly, the control device or peripheral auxiliary device can convert control information into control signals and send the control signals to the actuator (e.g., a robotic arm) of the peripheral auxiliary device. Upon receiving the control signal, the actuator can autonomously plan a suitable motion path within the range of physical constraints to execute a combination of the first target action and the second target action according to the corresponding execution strategy.

[0061] The aforementioned technical solution determines the first and second target actions by collecting EEG and EMG signals from the assisted individual. This allows for the rapid determination of the combined actions the assisted individual intends to perform using the peripheral assistive device, eliminating the need to individually identify each action in the combined action based solely on EEG or EMG signals. This significantly reduces the time required to determine the combined action. Furthermore, using both EMG and EEG signals to convey the action intention effectively reduces the possibility of EEG signals failing to match the target actions due to prolonged concentration when the assisted individual needs to convey multiple action intentions. Using both EMG and EEG signals to determine the first and second target actions improves the efficiency and accuracy of action determination, enhancing the assisted individual's experience. On the other hand, the aforementioned technical solution can also generate control information based on the first and second target actions to control the movement of the peripheral assistive device. The movement of the peripheral assistive device is flexible and diverse. Especially when using this solution to assist the assisted individual in motor rehabilitation, it helps to control the flexible movement of the peripheral assistive device according to the individual's actual situation, thereby effectively assisting the assisted individual in motor rehabilitation.

[0062] Optionally, identifying the specific action represented by the electromyographic signal includes: identifying the action type and action parameters of the specific action; and determining the second target action corresponding to the identified specific action based on the pre-configured correspondence between multiple specific actions and multiple target actions, including: determining the target action set corresponding to the identified action type based on the pre-configured correspondence between multiple action types and multiple target action sets; and selecting an action from the target action set as the second target action based on the identified action parameters.

[0063] For example, as described above, at least two specific actions of the same type may exist among multiple specific actions. For at least two specific actions of the same type, target actions that correspond one-to-one with these at least two specific actions can be grouped into an action set, i.e., a target action set. Multiple specific actions may include at least two action types, and correspondingly, multiple target actions can be divided into at least two target action sets that correspond one-to-one with the at least two action types, wherein each target action set of at least one target action set may include at least two target actions. It is understood that each action type may have a pre-configured correspondence with a target action set, and the target actions contained in the target action sets corresponding to each action type are different from each other. The control device can identify the action type of a specific action performed by the assisted object based on the acquired electromyographic signals, and determine the target action set corresponding to the identified action type from the at least two target action sets. For example, each target action in the target action set may have a pre-configured correspondence with an action parameter, and the action parameters corresponding to each target action in the same target action set are different. The control device can also identify the action parameters of a specific action performed by the assisted object based on the collected electromyographic signals. After determining the target action set based on the identified action type, the control device can determine the target action corresponding to the identified action parameters as the second target action in the determined target action set.

[0064] In the above technical solution, if various specific actions are distinguished only by action type, it is difficult to configure a corresponding action type for each target action when there are many target actions. By combining action type and action parameters, more specific actions can be distinguished by action parameters when the number of action types is limited. Thus, a correspondence between multiple target actions and specific actions can be configured one by one. By identifying the action type and action parameters of specific actions, the unique second target action corresponding to each specific action performed by the auxiliary object can be accurately determined among multiple target actions.

[0065] Optionally, for each of the multiple target action sets, the target action set can be associated with other target action sets besides itself through multiple action types, and there is a one-to-one correspondence between the multiple action types and other target action sets; according to the pre-configured correspondence between the multiple action types and the multiple target action sets, the target action set corresponding to the identified action type is determined, including: determining the first target action set where the first target action is located, and according to the one-to-one correspondence between the multiple action types and the target action sets, determining the second target action set associated with the first target action set through the identified action type, the second target action set being the target action set subsequently used to determine the second target action.

[0066] For example, for each set of target actions, associations between that set and other sets of target actions can be pre-configured. These associations can be established through action types. These associations are directional; each set of target actions can point to another set of target actions through one action type. It can be understood that other sets of target actions, besides the target set itself, correspond one-to-one with multiple action types. In a specific embodiment, the multiple target actions are divided into three sets, denoted as the first set, the second set, and the third set. The action types of these specific actions include wrist flexion and wrist extension. The first set can be associated with the second set through wrist flexion and with the third set through wrist extension. Similarly, the second set can be associated with the first set through wrist flexion and with the third set through wrist extension. Likewise, the third set can be associated with the first set through wrist flexion and with the second set through wrist extension.

[0067] It is understandable that each set of target actions can serve as an association starting point, pointing to another set of target actions as an association ending point through one action type. When the association starting points are different, the association ending points corresponding to the same action type may be the same or different. When the control device determines the first target action, it can determine the first set of target actions to which the first target action belongs; the first set of target actions is the current association starting point. When the first set of target actions is the current association starting point, all other set of target actions besides the first set of target actions are set of target actions associated with the first set of target actions. Among the set of target actions associated with the first set of target actions, the set of target actions that corresponds to the identified action type when the first set of target actions is the association starting point can be determined; this is the second set of target actions. The second set of target actions is the set of target actions used to determine the second target action.

[0068] In the above technical solution, when the associated starting point is different, the target action corresponding to the specific action may be different. This realizes the reuse of specific actions and helps to reduce the workload of configuring specific actions.

[0069] Please continue reading. Figure 2 As shown, Figure 2 The display interface also shows the relationships between set C and sets A, B, D, and E through four action types (which can be understood as...). Figure 3 The display interface can, if an action in set C is determined to be the first target action, display the action types associated with other sets through set C; if an action in another set, for example, set B, is determined to be the first target action, then the screen will display the different action types associated with other sets of set B, and the display position of the set containing the first target action can be adjusted to the center of the screen. Figure 2In the exemplary interface shown, set C can be associated with set A through wrist flexion, set B through ulnar wrist deviation, set D through radial wrist deviation, and set E through wrist extension. In this embodiment, the action parameter is the number of actions. When the assistive subject selects any target action in set C as the first target action via EEG signal, if the assistive subject's wrist moves once according to the wrist flexion action type, the determined second target action is wrist extension; if the assistive subject's wrist moves twice according to the wrist flexion action type, the determined second target action is wrist repositioning; if the assistive subject's wrist moves three times according to the wrist flexion action type, the determined second target action is wrist flexion. Similarly, when the assistive subject selects any target action in set C as the first target action via EEG signal, if the assistive subject's wrist moves once according to the ulnar wrist deviation action type, the determined second target action is elbow flexion; if the assistive subject's wrist moves twice according to the ulnar wrist deviation action type, the determined second target action is elbow repositioning; if the assistive subject's wrist moves three times according to the ulnar wrist deviation action type, the determined second target action is elbow extension. Similarly, when the subject selects any target action from set C as the first target action via EEG signals, if the subject's wrist moves once according to the radial wrist movement type, the determined second target action is forearm external rotation; if the subject's wrist moves twice according to the radial wrist movement type, the determined second target action is forearm stationary; if the subject's wrist moves three times according to the radial wrist movement type, the determined second target action is forearm internal rotation. Similarly, when the subject selects any target action from set C as the first target action via EEG signals, if the subject's wrist moves once according to the wrist extension movement type, the determined second target action is upper arm internal rotation; if the subject's wrist moves twice according to the wrist extension movement type, the determined second target action is upper arm stationary; if the subject's wrist moves three times according to the wrist extension movement type, the determined second target action is upper arm external rotation. After determining the first target action and the second target action, the display device can highlight the icons corresponding to the first target action and the second target action respectively. The highlighting method can be, for example, thickening the edge of the icon, distinguishing the color of the icon from the color of other icons, or stopping the icon from flashing.

[0070] It should be noted that, in Figure 2In the illustrated embodiments, sets A, B, D, and E can all be considered as the target action sets to which the first target action belongs. Although the correspondence between other sets and action types is not displayed on the display interface when these sets are respectively considered as the target action sets to which the first target action belongs, the assistive object is usually trained before executing the device control method of the present invention and can understand the correspondence between other target sets and action types when each target action set is considered as the set to which the first target action selected by the EEG signal belongs. Specifically, when any target action in set A is used as the first target action: set A is associated with set E through wrist flexion, with the number of actions corresponding to upper arm external rotation being 1, the number of actions corresponding to upper arm stillness being 2, and the number of actions corresponding to upper arm internal rotation being 3; set A is associated with set C through wrist extension, with the number of actions corresponding to shoulder adduction being 1, the number of actions corresponding to shoulder repositioning being 2, and the number of actions corresponding to shoulder abduction being 3; set A is associated with set B through wrist ulnar deviation, with the number of actions corresponding to elbow flexion being 1, the number of actions corresponding to elbow repositioning being 2, and the number of actions corresponding to elbow extension being 3; set A is associated with set D through wrist radial deviation, with the number of actions corresponding to forearm external rotation being 1, the number of actions corresponding to forearm stillness being 2, and the number of actions corresponding to forearm external rotation being 3.

[0071] Specifically, when any target action in set B is used as the first target action: set B is associated with set A through wrist flexion, with wrist extension corresponding to 1 action, wrist repositioning corresponding to 2 actions, and wrist flexion corresponding to 3 actions; set B is associated with set C through wrist radial deviation, with shoulder abduction corresponding to 1 action, shoulder repositioning corresponding to 2 actions, and shoulder adduction corresponding to 3 actions; set B is associated with set D through wrist ulnar deviation, with forearm external rotation corresponding to 1 action, forearm stillness corresponding to 2 actions, and forearm external rotation corresponding to 3 actions; set B is associated with set E through wrist extension, with upper arm internal rotation corresponding to 1 action, upper arm stillness corresponding to 2 actions, and upper arm external rotation corresponding to 3 actions. Specifically, when any target action in set D is used as the first target action: set D is associated with set A through wrist flexion, with the number of wrist extension actions being 1, the number of wrist repositioning actions being 2, and the number of wrist flexion actions being 3; set B is associated with set C through wrist radial deviation, with the number of shoulder abduction actions being 1, the number of shoulder repositioning actions being 2, and the number of shoulder adduction actions being 3; set B is associated with set D through wrist ulnar deviation, with the number of forearm external rotation actions being 1, the number of forearm stillness actions being 2, and the number of forearm external rotation actions being 3; set B is associated with set E through wrist extension, with the number of upper arm internal rotation actions being 1, the number of upper arm stillness actions being 2, and the number of upper arm external rotation actions being 3. Specifically, when any target action in set E is used as the first target action: set E is associated with set C through wrist flexion, with the number of actions corresponding to shoulder abduction being 1, shoulder repositioning being 2, and shoulder adduction being 3; set E is associated with set A through wrist extension, with the number of actions corresponding to wrist flexion being 1, wrist repositioning being 2, and wrist extension being 3; set E is associated with set D through radial wrist deviation, with the number of actions corresponding to forearm external rotation being 1, forearm stillness being 2, and forearm external rotation being 3; set E is associated with set B through ulnar wrist deviation, with the number of actions corresponding to elbow flexion being 1, elbow repositioning being 2, and elbow extension being 3.

[0072] Optionally, based on the pre-configured correspondence between various specific actions and various target actions, different action types are distinguished by different joints, and / or by different ways of movement.

[0073] For example, the joint corresponding to a specific movement type refers to the joint of the assistive device, such as the shoulder, elbow, wrist, and knee joints. The same joint can move in different ways; for example, the wrist joint can move in ways such as wrist extension, wrist flexion, radial deviation, and ulnar deviation. For any two specific movements with different types, these two movements can be performed by different joints with the same or different movements, or they can be performed by different movements of the same joint. Distinguishing specific types of movements by the joints and / or movements of the assistive device allows for a greater diversity of configured movements and more flexible configuration methods. The assistive device can control peripheral assistive devices to perform various target movements through a wide variety of limb movements.

[0074] Optionally, each action in the target action set is pre-configured with a correspondence between the action and the action parameters, which include at least one of the following: number of actions, action amplitude, and action frequency.

[0075] For example, for any set of target actions, when the action parameter of a specific action in the set of target actions is the number of actions, the control device can perform feature analysis on the electromyographic signals acquired during the signal sampling period to count the occurrence frequency of signal features corresponding to the determined action type, thereby determining the number of actions performed by the assistive object according to the corresponding action type. The control device may or may not include the electromyographic signals on which the action type is identified in the number of actions. When the action parameter of a specific action in the set of target actions is the amplitude of action, the control device can determine the specific action performed by the assistive object based on the energy of the electromyographic signals acquired during the signal sampling period. When the action parameter of a specific action in the set of target actions is the frequency of action, the control device can perform feature analysis on the electromyographic signals acquired during the signal sampling period to obtain the frequency of occurrence of signal features corresponding to the determined action type, thereby determining the frequency of actions performed by the assistive object according to the corresponding action type. For example, typically, specific actions within the same set of target actions are distinguished by the same action parameter, while the action parameters used to distinguish specific actions in different sets of target actions may be the same or different. The embodiments of the present invention provide a wide variety of action parameters for distinguishing specific actions, which helps to improve the diversity of specific actions.

[0076] Please see Figure 3 The diagram shown is a schematic representation of an electromyographic signal recognition pattern according to an embodiment of the present invention. The channel corresponding to the palmaris longus muscle is designated as channel 1, and the electromyographic signal of channel 1 is designated as Pc1. The channel corresponding to the extensor carpi radialis muscle is designated as channel 2, and the electromyographic signal of channel 2 is designated as Pc2. Figure 3In the left graph, the horizontal axis represents the change of the number of sampling points over time, and the vertical axis represents the signal amplitude. The waveform of the electromyographic (EMG) signal acquired by the EMG signal acquisition device when the assisted subject sequentially performs wrist flexion, radial deviance, ulnar deviance, and wrist extension movements is shown. Figure 3 The waveform shown in the left-middle figure. The control device can optionally filter the acquired electromyographic signals, for example, from 80Hz to 500Hz, and then perform notch filtering on the filtered electromyographic signals at 50Hz, 100Hz, 150Hz, 200Hz, 250Hz, 300Hz, 350Hz, 400Hz, 450Hz, and 500Hz to reduce power frequency noise. Figure 3 In the corresponding embodiment, the duration of a single movement by the assisted object according to any action type is 1 second. Therefore, the action type can be determined using the electromyographic (EMG) signal in the first second of the sampling period. For the signal energy of the EMG signal in each channel, the energy of the EMG signal in that channel is equal to the sum of the squares of the sampling point values ​​of the EMG signal within the sampling period (1 second in this embodiment). (Reference) Figure 3 The comparison of the energy of the electromyographic signals in the two channels shown in the left and middle diagrams of the auxiliary object during wrist flexion, radial deviation, ulnar deviation, and wrist extension can determine the type of specific action performed by the auxiliary object.

[0077] exist Figure 3 In the corresponding embodiment, since the maximum number of movements is 3, the sampling period is 3 seconds. The sampling period can be divided into 3 time windows based on the time of a single movement (i.e., 1 second). If the movement type is wrist flexion or ulnar wrist deviation, the electromyography (EMG) signal from channel 1 is used as the analysis object; if the movement type is wrist radial deviation or wrist extension, the EMG signal from channel 2 is used as the analysis object. For the EMG signal used as the analysis object, the signal energy value within the 3 time windows can be calculated, and the maximum energy value Pmax within the 3 time windows can be calculated. Then, the number of windows with an energy value greater than 0.5 × Pmax within the 3 time windows is counted, which is the number of movements. Figure 3 The right figure shows the peak energy distribution of the electromyographic (EMG) signals collected by the EMG signal acquisition device when the experimental subject performed one or more movements of a certain action type within each sampling time period of multiple sampling time periods in a single experiment. The horizontal axis represents the number of sampling points increasing over time, and the vertical axis represents the signal energy. Figure 3 In the right-hand diagram, each pair of adjacent dashed lines represents a sampling period. The number of peak values ​​within a single sampling period indicates the number of actions performed during that period. Figure 3 Taking the first three sampling periods shown in the right figure as an example, the number of actions for each period is 1, 3, and 2, respectively.

[0078] Optionally, the peripheral auxiliary equipment includes multiple joints, and there is a correspondence between the multiple joints and multiple preset action sets. Each preset action set includes one or more actions that the corresponding joint can perform. The multiple target action sets are at least a portion of the multiple preset action sets, and the target action set to which the first target action belongs is different from the target action set to which the second target action belongs.

[0079] For example, the actuator of a peripheral auxiliary device may include multiple joints, each of which can perform one or more actions. Taking a robotic arm as an example, the robotic arm may have shoulder joints, wrist joints, elbow joints, etc. Each joint of the actuator can respectively perform actions such as wrist flexion, wrist repositioning, wrist extension, elbow flexion, elbow repositioning, elbow extension, shoulder adduction, shoulder repositioning, shoulder abduction, forearm internal rotation, forearm stationary, forearm external rotation, upper arm internal rotation, upper arm stationary, upper arm external rotation, etc. The actions that the actuator can perform can be divided into multiple preset action sets, and each joint can have a corresponding relationship with at least one preset action set, and the actions in the preset action set corresponding to that joint can be performed by that joint. In some embodiments, the actions that each joint of the actuator of the peripheral auxiliary device can perform can be divided into multiple preset action sets according to preset rules, such as the execution duration of the action, the expected execution frequency of the action, etc. The target action set can be at least a portion of the entire preset action set of the peripheral auxiliary device. The first target action and the second target action belong to different sets of target actions, and the joints corresponding to the set of target actions of the first target action and the set of target actions of the second target action are different.

[0080] In the above technical solution, the assisted user can control the joints of the peripheral auxiliary device to perform corresponding actions through brain consciousness and limb movements. The peripheral auxiliary device can flexibly perform relatively complex actions and can perform a wide variety of combined actions, which can easily adapt to the user's control needs.

[0081] Optionally, the peripheral assistive device includes a pressure sensor and a transcutaneous electrical nerve stimulation (TENS) subsystem. The pressure sensor is used to detect the pressure on a first preset site of the assistive subject, and the TENS subsystem is used to electrically stimulate a second preset site of the assistive subject. The method further includes: determining the electrical stimulation intensity of the TENS subsystem based on the measurement value of the pressure sensor; and controlling the TENS subsystem to electrically stimulate the second preset site of the assistive subject according to the determined electrical stimulation intensity.

[0082] For example, at least some joints of the actuator of the peripheral assistive device may be equipped with three-dimensional pressure sensors. The actuator may be a wearable exoskeleton robotic arm. When the assistive object wears the actuator, the pressure sensors can measure the pressure changes of the corresponding joints relative to their original state during the movement of the actuator. In particular, in the exoskeleton robotic arm, this pressure change may be related to the muscle exertion of the upper limb movement part (i.e., the first preset part); in other types of robotic arms, it may be related to the movement state. The assistive object can adjust the way the pressure sensors are used according to actual needs. For example, the peripheral assistive device may also include a (transcutaneous) electrical nerve stimulation subsystem, which may include a microprocessor and an electrical stimulation plate. The microprocessor can acquire the measurement values ​​of the pressure sensors and calculate the electrical stimulation intensity applied to the second preset part of the assistive object based on the measurement values. Electrical stimulation is applied to different muscle groups of the assistive object through the electrical stimulation patch, and the intensity of the electrical stimulation can be adjusted in real time according to the real-time acquired measurement values ​​to make it more in line with the physiological laws of human movement. Generally, the larger the measurement value of the pressure sensor, the greater the intensity of the applied electrical stimulation. Specifically, the electrical stimulation patches can be placed on both sides of the muscles of the corresponding upper limb requiring rehabilitation training, such as the deltoid, biceps brachii, triceps brachii, palmaris longus, and extensor carpi radialis. In one specific embodiment, when the subject's upper limb function is limited, or when the subject determines that the limb to be electrically stimulated is the upper limb, the electrical stimulation patches can be placed at both ends of the muscles controlling upper limb movement. By applying positive or negative sine waves, square waves, or continuous stimulation, the activity induced in the cerebral cortex by limb movement can be simulated, thereby promoting brain neuroplasticity.

[0083] Please see Figure 4 The diagram shown is a schematic block diagram of a robotic arm control system according to an embodiment of the present invention. The robotic arm control system may include a visual cues subsystem, a brain-computer interface subsystem, a human physiological subsystem, a transcutaneous electrical nerve stimulation subsystem, a robotic subsystem, and a communication subsystem. The human physiological subsystem (i.e., the assistive object) includes the brain for providing electroencephalogram (EEG) signals and muscles for providing electromyographic (EMG) signals. Figure 4The physiological signals referred to here are electroencephalogram (EEG) and electromyogram (EMG) signals. The assisted user generates EEG signals by observing icons presented by the visual cueing subsystem for visual stimulation. The brain-computer interface subsystem includes an EEG analysis module and an EMG analysis module. The EEG analysis module uses feature extraction method 1 to determine the assisted user's actual attention from the EEG signals of the first channel, then uses conversion algorithm 1 to convert the actual attention into transparency and generate a corresponding transparency adjustment instruction (NF instruction) which is sent to the visual cueing subsystem. The visual cueing subsystem displays the icon according to the transparency indicated by the NF instruction to present the attention analysis results. The EEG analysis module can also use feature extraction method 2 (e.g., CCA, TRCA, TDCA) to identify the first target action corresponding to the acquired EEG signals from the second channel, and convert the identification result into a first motor control instruction using conversion algorithm 2. The EMG analysis module uses feature extraction method 3 to determine the specific action corresponding to the acquired EMG signals, thereby obtaining a second target action corresponding to the determined specific action. The electromyography (EMG) analysis module can convert the determined second target action into a second motor control command using conversion algorithm 3. The EEG interface subsystem can fuse the first and second motor control commands into control information for the robotic arm and send this control information to the robot subsystem via the communication subsystem. Specifically, the communication subsystem can connect the various subsystems. The EEG interface subsystem can send control information to the robot subsystem according to the client-server TCP / IP protocol. The robot subsystem can generate a path planning strategy based on the control information to control the robotic arm to perform combined movements, thereby assisting the limb movement of the currently worn robotic arm. The robot subsystem can also transmit the measurement values ​​of the pressure sensors at each joint of the robotic arm (i.e., the actuator) to the transcutaneous electrical stimulation (TES) subsystem based on the serial port protocol. The TES subsystem can calculate the current intensity for muscle electrical stimulation based on the pressure sensor measurements to electrically stimulate the muscles of the assisted subject.

[0084] Please see Figure 5 As shown, it is a schematic block diagram of an electronic device 500 according to an embodiment of the present invention. According to another aspect of the present invention, an electronic device is also provided, including: a processor 510 and a memory 520, wherein the memory 520 stores computer program instructions, which are executed by the processor 510 to perform the above-described device control method.

[0085] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored. When the program instructions are executed by a computer or processor, the computer or processor performs corresponding steps of the device control method described in the embodiments of the present invention, and is used to implement corresponding modules in the robotic arm control system described in the embodiments of the present invention, or corresponding modules in the robotic arm control system described above. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0086] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the device control method as described above.

[0087] Those skilled in the art will be able to understand the specific implementation and beneficial effects of the above-described robotic arm control system, electronic equipment, storage medium, and computer program products by reading the detailed description of the equipment control method above, and will not elaborate further here for the sake of brevity.

[0088] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0089] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0091] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0092] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0093] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0094] Furthermore, those skilled in the art will understand that although some embodiments herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0095] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules in a robotic arm control system according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0096] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0097] The above are merely specific embodiments or descriptions of the present invention, and the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A device control method, characterized in that, The method comprises controlling a peripheral assistive device for an assisted subject's desired operation, the peripheral assistive device being communicatively connected to an EEG signal acquisition device, the EEG signal acquisition device being used to acquire the EEG signals of the assisted subject; the method includes: Based on the brain signals acquired by the brain signal acquisition device, it is determined whether the actual attention of the assisted object meets the preset attention requirements; If the determination result is yes, the first target action corresponding to the EEG signal is determined; otherwise, the assistive subject is prompted to increase attention; wherein, the EEG signal is generated due to the intention in the assistive subject's consciousness associated with the first target action; Control information is generated based on the first target action, wherein the control information is used to control the movement of the peripheral auxiliary equipment to perform the first target action.

2. The method according to claim 1, characterized in that, Also includes: The display device shows icons corresponding to multiple actions, with different flashing frequencies for the icons corresponding to different actions, and each icon displays a text description of the corresponding action. The EEG signals acquired by the EEG signal acquisition device are generated when the assisted object observes the icon corresponding to the action it is to perform, and the optic nerve is stimulated by the flashing of the observed icon. When determining the first target action corresponding to the EEG signal, the flashing frequency of the icon observed by the assistive object is determined based on the EEG signal, and the action corresponding to the icon with that flashing frequency is determined as the first target action.

3. The method according to claim 2, characterized in that, During the display of icons corresponding to multiple actions, the transparency of the icons is adjusted according to the attention of the auxiliary object.

4. The method according to claim 3, characterized in that, Determining whether the actual attention of the auxiliary object meets the preset attention requirements includes: The current attention value of the assisted subject is determined based on the electroencephalogram (EEG) signals. If the ratio of the current attention value to the preset attention reference value is higher than a preset ratio threshold, it is determined that the actual attention of the assisted object meets the preset attention requirements. Otherwise, it is determined that the actual attention of the auxiliary object does not meet the preset attention requirements; Furthermore, adjusting the icon's transparency based on the attention of the assisting object includes: If the actual attention of the auxiliary object does not meet the preset attention requirements, the transparency of the icon is increased.

5. The method according to claim 4, characterized in that, The increased transparency is associated with the ratio of the current attention value to the preset attention reference value.

6. The method according to claim 1, characterized in that, Also includes: Electromyographic signals acquired by an electromyographic signal acquisition device that is communicatively connected to the peripheral auxiliary device when the assisted object performs a specific action; The specific action represented by the electromyographic signal is identified, and a second target action corresponding to the identified specific action is determined based on a pre-configured correspondence between multiple specific actions and multiple target actions. Furthermore, generating control information based on the first target action includes: Control information is generated based on the first target action and the second target action, wherein the control information is used to control the movement of the peripheral auxiliary equipment to perform a combination of the first target action and the second target action.

7. The method according to claim 6, characterized in that, Identifying the specific actions represented by the electromyographic signals includes: Identify the action type and action parameters of the specific action; Furthermore, based on a pre-configured correspondence between various specific actions and various target actions, a second target action corresponding to the identified specific action is determined, including: Based on the pre-configured correspondence between multiple action types and multiple target action sets, determine the target action set corresponding to the identified action type; The action selected from the target action set based on the identified action parameters is used as the second target action.

8. The method according to claim 7, characterized in that, For each of the plurality of target action sets, the target action set can be associated with other target action sets other than the target action set through the plurality of action types, and there is a one-to-one correspondence between the plurality of action types and the other target action sets; Based on a pre-configured correspondence between multiple action types and multiple target action sets, the target action set corresponding to the identified action type is determined, including: A first target action set is determined, and based on the one-to-one correspondence between the various action types and the target action set, a second target action set is determined that is associated with the first target action set by the identified action type. The second target action set is the target action set subsequently used to determine the second target action.

9. The method according to claim 7, characterized in that, Based on the pre-configured correspondence between various specific actions and various target actions, different action types are distinguished by different joints, and / or by different ways of movement.

10. The method according to claim 7, characterized in that, Each action set contains actions pre-configured with corresponding action parameters, which include at least one of the following: Number of movements, range of motion, and frequency of movement.

11. The method according to claim 7, characterized in that, The peripheral auxiliary equipment includes multiple joints, and there is a correspondence between the multiple joints and multiple preset action sets. Each preset action set includes one or more actions that the corresponding joint can perform. The plurality of target action sets are at least a portion of the plurality of preset action sets, and the target action set to which the first target action belongs is different from the target action set to which the second target action belongs.

12. An electronic device comprising a processor and a memory, characterized in that, The memory stores computer program instructions, which, when executed by the processor, are used to perform the device control method as described in any one of claims 1-11.

13. A storage medium on which program instructions are stored, characterized in that, The program instructions are used to execute the device control method as described in any one of claims 1-11 when the program is run.

Images