A rehabilitation device control system and method for electroencephalogram signal emotion recognition

Abstract

This specification provides a control system for a rehabilitation device that uses EEG signal emotion recognition. By setting a synchronization module, the image acquisition module and the EEG signal module synchronously acquire and record the data over time. The data is then identified, timestamped, and transmitted to a queue. The emotion queue is monitored, and when a target emotion is detected, its timestamp is determined. The system searches the event queue for target event information containing the timestamp. Based on the timestamps of subsequent events in the event queue, the duration of the target event is monitored, and control commands are generated according to the duration and a preset control strategy to control the movement device in the rehabilitation device to perform the corresponding action. The synchronized timestamps avoid processing time differences between different modalities, and by combining environmental images and EEG signal recognition to identify events that trigger the target emotion, the accuracy of identifying sensitive scenes is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a control system and method for a rehabilitation device that recognizes emotions based on electroencephalogram (EEG) signals. Background Technology

[0002] People with disabilities often rely on rehabilitation devices to compensate for lost or temporarily lost functions, such as wheelchairs and powered prostheses that allow them to move within a certain range. To address specific scenarios, the industry uses cameras to identify hazards. However, different users experience the same scenario differently, leading to varying psychological barriers. For example, some people are afraid of dogs, while others like them; some have claustrophobia and prefer open spaces, while others prefer quiet environments; some like rainy days, while others prefer sunny days; some enjoy noisy children, while others find it unpleasant. For ordinary people, they can choose whether to leave such environments, but for those using devices to compensate for some functions, it is often difficult to quickly and accurately control the rehabilitation device to move according to their emotions.

[0003] Therefore, a new approach is needed to improve the accuracy of identifying scenarios that are sensitive to people with disabilities. Summary of the Invention

[0004] This specification provides a rehabilitation device control system for EEG signal emotion recognition, which improves the accuracy of recognizing sensitive scenes for non-healthy individuals, thereby accurately controlling the rehabilitation device to automatically avoid sensitive scenes.

[0005] The system has: The synchronization module controls the image acquisition module and the EEG signal module to acquire and record signals synchronously over time. The image acquisition module acquires environmental images, performs event recognition on the environmental images, adds a timestamp recorded by the synchronization module, and then transmits the images to the event queue. The EEG signal module collects EEG signals, performs emotion recognition on the EEG signals, and transmits them to the emotion queue after adding the timestamp recorded by the synchronization module. The linkage processing module is used to create an emotion queue and an event queue, which are used to store the identified emotion information and event information, respectively. It also monitors the emotion queue, determines the timestamp of the target emotion when a target emotion is detected, searches for target event information within a preset time window before the timestamp in the event queue, monitors the duration of the target event based on the timestamps of subsequent events in the event queue, and generates control instructions according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

[0006] Optionally, the EEG signal module has a pre-identification unit and a buffer unit; The pre-identification unit is used to perform emotion pre-identification on EEG signals to obtain emotion information containing emotion value and emotion category, which is then transmitted in the form of emotion value key-value pairs and emotion category key-value pairs after being timestamped. The buffer unit buffers emotion categories based on emotion information within a continuous time period, and then transmits the buffered emotion categories and corresponding timestamps to the emotion queue.

[0007] Optionally, the buffer unit includes: a segmenter, a recognizer, a recorder, a controller, an assigner, and an outputter; The process of buffering emotion categories based on emotional information over a continuous time period includes: The segmenter sets an emotion unification zone, and sets a buffer segment within the emotion unification zone that covers the midpoint of the emotion unification zone; The segmenter is used to determine the segment in which the sentiment value corresponding to each timestamp belongs in real time. The recognizer determines whether two emotional information messages in consecutive time are in the same segment and change direction. When it is recognized that they are in different segments, it generates either an out-of-area behavior or an in-area behavior based on their change direction. The recorder is used to record the entry behavior and the emotional category before entering the area when the entry behavior is detected; The controller is used to make judgments and controls based on the recognition results of the recognizer and the recording results of the recorder, including: If the current emotion information is in the same segment as the emotion information of the previous timestamp, the assigner is triggered to read the emotion category in the previous emotion information, assign a value to the emotion category key in the current emotion information, and then pass it to the output device. If the current emotional information is in a different segment than the emotional information at the previous timestamp, then it is determined whether the behavior currently identified by the recognizer is an out-of-area behavior. If it is an out-of-area behavior, it is determined whether the recognizer has identified a historical in-area behavior. If there is no historical in-area behavior, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If there is a historical in-area behavior, it is determined whether the change direction of the in-area behavior is the same as the change direction of the current out-of-area behavior. If the change direction is the same, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If the change direction is different, then the assigner is triggered to read the emotional category before entering the area, assign a value to the emotional category key in the current emotional information, and then pass it to the output device. If it is not an outgoing behavior, the assigner is triggered to read the emotion category before entering the area, assign a value to the emotion category key in the current emotion information, and then pass it to the output device.

[0008] Optionally, the step of performing emotion pre-identification on the EEG signals to obtain emotion information including emotion value and emotion category includes: Acquire the first EEG signal as the baseline emotion, and acquire the second EEG signal as the target emotion; The brainwave similarity between the currently acquired EEG signal and the first and second EEG signals is calculated, which are the first similarity and the second similarity. The emotion value is the ratio of the second similarity to the sum of the first and second similarities. If the emotion value is less than 0.5, the emotion category is pre-identified as the baseline emotion; if the emotion value is greater than 0.5, the emotion category is pre-identified as the target emotion.

[0009] Optionally, the segmenter sets an emotion normalization zone, including: Set the endpoint values ​​of the emotion normalization zone to 0 and 1; The direction of change is determined by the increasing or decreasing trend of sentiment values ​​across consecutive timestamps.

[0010] Optionally, the system also includes a buffer segment adjustment unit for adjusting the endpoint values ​​of the buffer segment.

[0011] Optionally, the system also has a verification module with a sample set for verifying the stability of the rehabilitation control system; Adjust the endpoint values ​​of the buffer segment, including: Obtain the stability tendency of the user input, and adjust the endpoint values ​​of the buffer segment according to the stability tendency. If the user tends to increase stability, increase the distance between the endpoint values ​​of the buffer segment.

[0012] Optionally, the target emotion is a high-arousal emotion; The step of generating control commands based on the duration and a preset control strategy to enable the motion device controlled by the system to perform corresponding actions includes: If the duration exceeds the preset duration, a control command is generated to cause the motion device in the system to perform an avoidance action.

[0013] This manual also provides a control method for a rehabilitation device that recognizes emotions based on electroencephalogram (EEG) signals, including: The image acquisition module and the EEG signal module are synchronized to acquire and record the time. The image acquisition module acquires environmental images, performs event recognition on the environmental images, adds a timestamp recorded by the synchronization module, and then transmits them to the event queue. EEG signals are collected through the EEG signal module, and after emotion recognition is performed on the EEG signals and the timestamp recorded by the synchronization module is added, the signals are transmitted to the emotion queue. An emotion queue and an event queue are created to store identified emotion information and event information, respectively. The emotion queue is monitored, and when a target emotion is detected, the timestamp of the target emotion is determined. The target event information within a preset time window before the timestamp is searched in the event queue. The duration of the target event is monitored based on the timestamps of subsequent events in the event queue. Control commands are generated according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

[0014] This specification also provides a computer device, wherein the computer device includes: Processor; and, A memory that stores computer-executable instructions, which, when executed, cause the processor to perform any of the methods described above.

[0015] This specification also provides a computer-readable storage medium that stores one or more programs / instructions that, when executed by a processor, implement any of the methods described above.

[0016] This specification also provides a computer program product, wherein the computer program product includes: a computer program / instruction, which, when executed by a processor, implements any of the methods described above.

[0017] The various technical solutions provided in the embodiments of this specification, by setting up a synchronization module, control the image acquisition module and the EEG signal module to synchronously acquire and record time, identify and stamp the data, and then transmit it to a queue for monitoring the emotion queue. When a target emotion is detected, the timestamp of the target emotion is determined, and the target event information containing the timestamp is searched in the event queue. Based on the timestamps of subsequent events in the event queue, the duration of the target event is monitored, and control instructions are generated according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action. By using synchronized timestamps to avoid processing time differences between different modalities, and by combining environmental images and EEG signals to identify events that trigger the target emotion, the accuracy of identifying sensitive scenes is improved. Attached Figure Description

[0018] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0019] In the attached diagram: Figure 1 A schematic diagram of the structure of a rehabilitation device control system for EEG signal emotion recognition provided in some embodiments of this specification; Figure 2 A schematic diagram of the structure of an electroencephalogram (EEG) signal module provided for some embodiments of this specification; Figure 3 A schematic diagram illustrating the principle of a rehabilitation device control system for EEG signal emotion recognition, provided for some embodiments of this specification; Figure 4 A schematic diagram illustrating the effect of a rehabilitation device control system for EEG signal emotion recognition, provided for some embodiments of this specification; Figure 5 A schematic diagram illustrating the principle of a rehabilitation device control method for EEG signal emotion recognition provided in some embodiments of this specification; Figure 6 This is a schematic diagram of a computer-readable medium provided for some embodiments of this specification. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Figure 1 This specification provides a schematic diagram of the structure of a rehabilitation device control system for EEG signal emotion recognition, as shown in some embodiments. The system includes: Synchronization module 101 controls the image acquisition module and the EEG signal module to synchronously acquire and record time; for each synchronous acquisition, a time is recorded, and the recorded time can be the start time of the synchronous acquisition by the image acquisition module and the EEG signal module, or the end time of the acquisition by both, etc. Image acquisition module 102 acquires environmental images, performs event recognition on the environmental images, adds a timestamp recorded by the synchronization module, and then transmits the images to the event queue. The EEG signal module 103 collects EEG signals, performs emotion recognition on the EEG signals, and transmits them to the emotion queue after stamping them with the timestamp recorded by the synchronization module. The linkage processing module 104 is used to create an emotion queue and an event queue, which are used to store the identified emotion information and event information, respectively. It also monitors the emotion queue, determines the timestamp of the target emotion when a target emotion is detected, searches for target event information within a preset time window before the timestamp in the event queue, monitors the duration of the target event based on the timestamp of subsequent events in the event queue, and generates control instructions according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

[0022] This system employs a synchronization module to control the image acquisition module and the EEG signal module to synchronously acquire and record time data. After identification and timestamp, the data is transmitted to a queue for monitoring the emotion queue. When a target emotion is detected, its timestamp is determined, and the system searches the event queue for target event information containing the timestamp. Based on the timestamps of subsequent events in the event queue, the system monitors the duration of the target event and generates control commands according to the duration and a preset control strategy, controlling the movement device in the rehabilitation apparatus to perform the corresponding actions. By using synchronized timestamps to avoid processing time differences between different modalities, and by combining environmental images and EEG signals to identify events that trigger the target emotion, the system improves the accuracy of identifying sensitive scenarios.

[0023] The image acquisition module can capture images of the area surrounding the user's rehabilitation device.

[0024] Rehabilitation devices can be whole objects, such as wheelchairs or prostheses; movement devices can be the electromechanical structures within them; and control systems can be local computer devices or remote servers that send instructions to these electromechanical structures.

[0025] Emotion recognition from EEG signals can involve calculating the emotion value corresponding to the EEG signal and determining the emotion category corresponding to that emotion value.

[0026] In some embodiments, the baseline emotion and the target emotion can be a pair of opposite emotions, such as happiness and sadness, excitement and fear.

[0027] Considering that in real-world scenarios, individuals with limited self-emotional regulation abilities or those at risk from high arousal, such as patients with cardiovascular and cerebrovascular diseases, mental disorders, epilepsy and other neurological diseases, children with special needs, pregnant women, and patients in post-operative recovery, it is especially important to avoid risky situations that trigger high arousal. Therefore, emotions can be categorized based on arousal level, and emotional arousal can be used to measure the extent to which emotions are activated.

[0028] Therefore, in some embodiments, low-arousal emotion can be used as the baseline emotion and high-arousal emotion as the target emotion.

[0029] In practice, emotional states with high physiological activation levels, such as tension, excitement, fear, and grief, can be considered as high arousal emotions. EEG signals at these times can be collected and labeled as EEG signals of high arousal emotions. Emotional states with low physiological activation levels, such as calmness and relaxation, can be considered as low arousal emotions. EEG signals at these times can be collected and labeled as EEG signals of low arousal emotions.

[0030] In specific implementation, the change amplitude and fluctuation frequency of the electroencephalogram (EEG) signal can also be calculated. The emotional state with the change amplitude and fluctuation frequency of the EEG signal less than the preset threshold is regarded as a low-arousal emotion, and the emotional state with the change amplitude and fluctuation frequency of the EEG signal greater than the preset threshold is regarded as a high-arousal emotion. The specific threshold can be designed according to experience and preference and is not specifically limited herein.

[0031] In the above various embodiments, for determining the emotional category of the EEG signal, in actual implementation, as an example, some sample users can be selected, movie clips of different scene types (such as fear, joy, excitement, sadness) are played for them, the corresponding EEG signals are collected, and their emotions are surveyed through questionnaires. Combining the survey results and the scene types of the movie clips, the emotional type of the EEG signal collected at that time is determined.

[0032] The form of the event information can be various pre-defined event identifiers, such as wind blowing, rain falling, a dog approaching, etc. The event information transmitted by the image acquisition module can exist in the form of a data packet.

[0033] By searching for the target event information within a preset time window before the time stamp in the event queue, it can be determined which event the user is nervous about. For example, when the wind suddenly blows, the wind blowing event can be identified by performing event recognition on the environmental image. By using the method of recognizing the target emotion to search for the events at that time, there is no need to preset the events to be recognized, which has strong flexibility and applicability.

[0034] For another example, for some people who are afraid of dogs, they will be nervous after a dog suddenly appears around. Through the EEG signal, it can be known that the change in emotion to nervousness occurs synchronously with the appearance of the dog around. At this time, the causal relationship can be roughly determined. However, multiple events may be included in consecutive images, and only the events synchronized with the emotion change are used as the target events, which is more meaningful for reference.

[0035] After the target event is recognized, considering that the duration of the target event may be very long or very short. If the duration is very short, there is no need to change the actions of the motion device (such as stopping, turning, or backing), which will cause the device to repeatedly switch its motion state and be unstable.

[0036] Therefore, in this embodiment, the duration of the target event can be monitored, and a control instruction is generated according to the duration and the preset control strategy to make the motion device controlled by the system complete the corresponding action.

[0037] The control strategy can be that if the duration reaches the preset duration, the motion state of the motion device is changed.

[0038] When classifying emotions into two opposing categories, emotion recognition often relies on a threshold: emotions greater than this threshold are identified as the first type of emotion, and those less than this threshold are identified as the second type of emotion. This approach is easily affected by momentary fluctuations and has poor recognition accuracy. Therefore, we propose constructing a buffer unit to buffer the recognized emotion categories.

[0039] In the embodiments described in this specification, the EEG signal module has a pre-identification unit and a buffer unit; The pre-identification unit is used to perform emotion pre-identification on EEG signals to obtain emotion information containing emotion value and emotion category, which is then transmitted in the form of emotion value key-value pairs and emotion category key-value pairs after being timestamped. The buffer unit buffers emotion categories based on emotion information within a continuous time period, and then transmits the buffered emotion categories and corresponding timestamps to the emotion queue.

[0040] Pre-identification calculates the emotion value and determines the emotion category to which the emotion value belongs, while buffering retains the original emotion value and only modifies the category.

[0041] By using emotion value key-value pairs and emotion category key-value pairs, the emotion category can be modified through assignment.

[0042] In the embodiments of this specification, the step of performing emotion pre-recognition on EEG signals to obtain emotion information including emotion value and emotion category may include: Acquire the first EEG signal as the baseline emotion, and acquire the second EEG signal as the target emotion; The brainwave similarity between the currently acquired EEG signal and the first and second EEG signals is calculated, which are the first similarity and the second similarity. The emotion value is the ratio of the second similarity score to the sum of the first and second similarities. If the emotion value is less than 0.5, the emotion category is pre-identified as the baseline emotion; if the emotion value is greater than 0.5, the emotion category is pre-identified as the target emotion. Both the first and second similarities are values ​​between 0 and 1.

[0043] As a simple example, calculating EEG similarity involves segmenting the current EEG signal, calculating feature values ​​for each segment, constructing a multi-dimensional feature vector using these feature values, and then calculating the dot product of the feature vector of the current EEG signal and the feature vector of the first EEG signal. Dividing this product by the product of their magnitudes yields the similarity between the current and first EEG signals. Similarly, calculating the dot product of the feature vectors of the current and second EEG signals and dividing this product by their magnitudes yields the similarity between the current and second EEG signals. There are many existing technologies for calculating EEG similarity; in fact, any method for calculating the similarity of waveform signals (such as audio signals) can be referenced, so it will not be elaborated upon here.

[0044] Figure 2 This is a schematic diagram of the structure of an electroencephalogram (EEG) signal module provided for some embodiments of this specification.

[0045] exist Figure 2 In the brain signal module, there is a pre-identification unit and a buffer unit. The buffer unit has: a segmenter, a recognizer, a recorder, a controller, an assigner, and an output unit. The EEG signal is input to the pre-identification unit. The pre-identification unit calculates the similarity between the EEG signal and the EEG signal of the baseline emotion as the first similarity, and calculates the similarity between the EEG signal and the EEG signal of the target emotion as the second similarity. The emotion value is calculated as: second similarity / (sum of first and second similarities). Based on whether the emotion value is greater than 0.5, the category is initially determined, and the emotion value and emotion category are stamped with the acquisition timestamp and transmitted to the buffer unit.

[0046] The buffer unit buffers emotion categories based on emotional information within a continuous time period. Specifically, it includes: The segmenter sets an emotion unification zone and sets a buffer segment within the emotion unification zone that covers the midpoint of the emotion unification zone. The segmenter is used to determine the segment in which the sentiment value corresponding to each timestamp belongs in real time. The recognizer determines whether two emotional information pieces with adjacent timestamps are in the same segment and change direction. When they are in different segments, the recognizer generates either an out-of-area behavior or an in-area behavior based on the change direction. The recorder is used to record the entry behavior and the emotional category before entering the area when the entry behavior is detected; The controller is used to make judgments and controls based on the recognition results of the identifier and the recording results of the recorder, specifically including: If the current emotion information is in the same segment as the emotion information of the previous timestamp, the assigner is triggered to read the emotion category in the previous emotion information, assign a value to the emotion category key in the current emotion information, and then pass it to the output device. If the current emotional information is in a different segment than the emotional information at the previous timestamp, then it is determined whether the behavior currently identified by the recognizer is an out-of-area behavior. If it is an out-of-area behavior, it is determined whether the recognizer has identified a historical in-area behavior. If there is no historical in-area behavior, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If there is a historical in-area behavior, it is determined whether the change direction of the in-area behavior is the same as the change direction of the current out-of-area behavior. If the change direction is the same, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If the change direction is different, then the assigner is triggered to read the emotional category before entering the area, assign a value to the emotional category key in the current emotional information, and then pass it to the output device. If it is not an outgoing behavior, the assigner is triggered to read the emotion category before entering the area, assign a value to the emotion category key in the current emotion information, and then pass it to the output device.

[0047] Among them, leaving the buffer zone refers to leaving the buffer zone, and entering the buffer zone refers to entering the buffer zone.

[0048] The direction of change can be determined based on whether the emotion value increases or decreases. The direction of change refers to the endpoint side in the emotion normalization zone, i.e., which side the change is directed towards.

[0049] The emotional value at the midpoint of the emotional normalization zone is 0.5.

[0050] Specifically, the emotional category before entering the area can be directly determined based on the emotional value at the time point before entering the area. An emotional value less than 0.5 is the baseline emotion, and an emotional value greater than 0.5 is the target emotion.

[0051] The phrase "then passed to the output device" can include: providing the output device with the key-value pairs of the emotion values, the assigned emotion category key-value pairs, and the collection timestamp. The output device then processes these values.

[0052] In this embodiment of the specification, the segmenter sets an emotion unification zone, including: The direction of change is determined by the increasing or decreasing trend of sentiment values ​​across consecutive timestamps.

[0053] Figure 3 This is a schematic diagram illustrating the principle of a rehabilitation device control system for EEG signal emotion recognition, provided for some embodiments of this specification. Figure 4 This specification provides schematic diagrams illustrating the effect of a rehabilitation device control system for EEG signal emotion recognition, as shown in some embodiments. Figure 3 and Figure 4 To elaborate.

[0054] Figure 4 This is a diagram for ease of understanding. In reality, as long as the segments in which each point in the diagram are located remain unchanged, even if... Figure 4 Time compression on the horizontal axis does not affect the judgment; therefore, for Figure 4 The x-coordinate of the midpoint is not specifically restricted. Figure 4 In the diagram, the ordinate of point A corresponds to the baseline emotion, the ordinate of point D corresponds to the target emotion, the area between the parallel lines AD is the emotion normalization zone, the area between the parallel dashed lines BC is the buffer zone, the ordinate of point e is the midpoint of the emotion normalization zone, and the emotion value at point e is 0.5. The system only needs to determine which segment it belongs to based on the current emotion value.

[0055] The segmenter is used to determine the segment in real time where the emotion value corresponding to each timestamp is located. It identifies point t1 as the first time the EEG signal was collected and the corresponding emotion value, which is outside the buffer segment. Then the segmenter determines that point t2 is inside the buffer segment, and the recognizer judges that an entry behavior has occurred. When the recognizer recognizes the entry behavior, the recorder records the entry behavior and the emotion category of t1 before entry (as the baseline emotion).

[0056] At this point, the recognizer determines that the two emotions in consecutive time are in different segments, and continues to judge. It determines that the current (t2) is not an out-of-zone behavior. Therefore, the assigner reads the emotion category before entering the zone (t1) and assigns a value to the emotion category key of t2, so that the emotion category of t2 becomes the base emotion. After the output is output, the emotion category does not change, the motion device does not change, and it remains stable.

[0057] At time t3, the recognizer determines that the two emotions in consecutive time are in the same segment. The assignor reads the emotion category of t2 and assigns it to the emotion category of t3, making the emotion category of t3 the base emotion. After the output is output, the emotion category does not change, the motion device does not change, and it remains stable.

[0058] The same applies to t4, maintaining stability.

[0059] At time t5, the recognizer determines that the two emotions in consecutive time periods are in different segments, and continues to judge. If it determines that the current (t5) is an out-of-area behavior, it continues to judge. If it determines that the recognizer has identified a historical in-area behavior (the most recent in-area time in the figure is t1-t2), it continues to judge. If it determines that the direction of change of the in-area behavior is different from the direction of change of the current out-of-area behavior, it assigns the emotion category of t4 to the emotion category of t5, so that the emotion category of t5 maintains the baseline emotion. After the output device outputs, the emotion category does not change, the motion device does not change, and it remains stable.

[0060] The same applies to t6, maintaining stability.

[0061] At time t7, the recognizer determines that the two emotions in consecutive time are in different segments, and continues to judge. If it is determined that the current (t7) is not an out-of-zone behavior, the emotion category of t7 is assigned to the emotion category of t7, so that the emotion category of t7 maintains the baseline emotion. After the output is output, the emotion category does not change, the motion device does not change, and it remains stable.

[0062] The same applies to t8, maintaining stability.

[0063] At time t9, the recognizer determines that the two emotions in consecutive time intervals (t8 and t9) are in different segments, and continues to judge. If it determines that the current (t9) is an out-of-area behavior, it continues to judge. If it determines that the recognizer has identified a historical in-area behavior (t7), it continues to judge. If it determines that the direction of change of the most recent historical in-area behavior is the same as the direction of change of the current out-of-area behavior (both emotion values ​​increase), then no assignment is needed. The assignor is skipped and the output device is directly triggered to extract the timestamp and emotion category from the emotion information, associate them, and pass them to the emotion queue. If the emotion category changes compared to time t8, the motion device changes its action.

[0064] If we follow the traditional method of dividing different emotions into boundary points to determine the emotion category, the above process requires three switching steps, with the motion device changing its movements three times. However, the above embodiment only requires one switching step, which improves stability and avoids repeated switching near the boundary point.

[0065] Considering that the size of the buffer segment is used to balance stability and recognition accuracy in practical applications, and can be adjusted according to the actual situation, the system in this embodiment of the specification also has a buffer segment adjustment unit for adjusting the endpoint values ​​of the buffer segment.

[0066] In practical applications, the endpoint values ​​of the buffer segment can be adjusted manually by providing stability verification results to users.

[0067] Therefore, in the embodiments of this specification, the system also has a verification module, which has a sample set for verifying the stability of the rehabilitation device control system; Adjust the endpoint values ​​of the buffer segment, including: Obtain the stability tendency of the user input, and adjust the endpoint values ​​of the buffer segment according to the stability tendency. If the user tends to increase stability, increase the distance between the endpoint values ​​of the buffer segment.

[0068] Stability can also be determined by the number of times the emotion category is mistakenly switched during EEG signal fluctuations. For example, if a motion device repeatedly switches motion states more than a predetermined number of times within a preset time period due to changes in emotion category recognition, then the stability is deemed too low and should be increased.

[0069] In the embodiments described in this specification, the target emotion is a high-arousal emotion; The step of generating control commands based on the duration and a preset control strategy to enable the motion device controlled by the system to perform corresponding actions includes: If the duration exceeds a preset duration, a control command is generated to cause the motion device in the system to perform an avoidance action, such as deceleration, braking, or reversing the path.

[0070] Figure 5 The schematic diagram illustrates the principle of a control method for a rehabilitation device based on EEG signal emotion recognition, provided in some embodiments of this specification. This method is applicable to the systems shown in the various embodiments described above and includes: S501: Controls the image acquisition module and the EEG signal module to acquire and record the time synchronously; S502: The image acquisition module acquires environmental images, performs event recognition on the environmental images, adds the timestamp recorded by the synchronization module, and then transmits them to the event queue; S503: Collect brain signals through the brain signal module, perform emotion recognition on the brain signals, and transmit them to the emotion queue after stamping with the timestamp recorded by the synchronization module; S504: Create an emotion queue and an event queue to store identified emotion information and event information, respectively; monitor the emotion queue; when a target emotion is detected, determine the timestamp of the target emotion; search for target event information within a preset time window before the timestamp in the event queue; monitor the duration of the target event based on the timestamps of subsequent events in the event queue; and generate control instructions according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

[0071] Optionally, the emotion recognition of the EEG signal includes: performing emotion pre-recognition on the EEG signal to obtain emotion information containing emotion value and emotion category, and transmitting it in the form of emotion value key-value pairs and emotion category key-value pairs after adding a timestamp; buffering the emotion category according to the emotion information in a continuous time period, and transmitting the buffered emotion category and the corresponding timestamp to the emotion queue.

[0072] Optionally, the emotion category is buffered based on emotional information over a continuous period of time, including: Set up an emotion unification zone, and set up a buffer segment within the emotion unification zone that covers the midpoint of the emotion unification zone; Determine the segment in which the sentiment value corresponding to each timestamp falls in real time; Determine whether two emotional messages in consecutive time are in the same segment and change direction, and when they are in different segments, generate either an out-of-area behavior or an in-area behavior based on their change direction. When an entry behavior is detected, the entry behavior and the emotional category before entering the area are recorded. Judgments and controls are made based on the recognition results of the identifier and the recording results of the recorder, specifically including: If the current emotion information is in the same segment as the emotion information of the previous timestamp, the assigner is triggered to read the emotion category in the previous emotion information, assign a value to the emotion category key in the current emotion information, and then pass it to the output device. If the current emotional information is in a different segment than the emotional information at the previous timestamp, then it is determined whether the behavior currently identified by the recognizer is an out-of-area behavior. If it is an out-of-area behavior, it is determined whether the recognizer has identified a historical in-area behavior. If there is no historical in-area behavior, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If there is a historical in-area behavior, it is determined whether the change direction of the in-area behavior is the same as the change direction of the current out-of-area behavior. If the change direction is the same, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If the change direction is different, then the assigner is triggered to read the emotional category before entering the area, assign a value to the emotional category key in the current emotional information, and then pass it to the output device. If it is not an outgoing behavior, the assigner is triggered to read the emotion category before entering the area, assign a value to the emotion category key in the current emotion information, and then pass it to the output device.

[0073] Optionally, the step of performing emotion pre-identification on the EEG signals to obtain emotion information including emotion value and emotion category includes: Acquire the first EEG signal as the baseline emotion, and acquire the second EEG signal as the target emotion; The brainwave similarity between the currently acquired EEG signal and the first and second EEG signals is calculated, which are the first similarity and the second similarity. The emotion value is the ratio of the second similarity to the sum of the first and second similarities. If the emotion value is less than 0.5, the emotion category is pre-identified as the baseline emotion; if the emotion value is greater than 0.5, the emotion category is pre-identified as the target emotion.

[0074] Optionally, the setting of the emotion unification zone includes: Set the endpoint values ​​of the emotion normalization zone to 0 and 1; The direction of change is determined by the increasing or decreasing trend of sentiment values ​​across consecutive timestamps.

[0075] Optionally, the method further includes: adjusting the endpoint values ​​of the buffer segment.

[0076] Optionally, the method further includes: acquiring a sample set to verify the stability of the rehabilitation control system; Adjust the endpoint values ​​of the buffer segment, including: Obtain the stability tendency of the user input, and adjust the endpoint values ​​of the buffer segment according to the stability tendency. If the user tends to increase stability, increase the distance between the endpoint values ​​of the buffer segment.

[0077] Optionally, the target emotion is a high-arousal emotion; The step of generating control commands based on the duration and a preset control strategy to enable the motion device controlled by the system to perform corresponding actions includes: If the duration exceeds the preset duration, a control command is generated to cause the motion device in the system to perform an avoidance action.

[0078] Figure 6 This is a schematic diagram of a computer-readable medium provided for some embodiments of this specification.

[0079] accomplish Figure 5 The computer program for the method illustrated can be stored on one or more computer-readable media. The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. The program code for performing the operations of this invention can be written in any combination of one or more programming languages. The program code can be executed entirely on a user computing device, partially on a user computing device, as a standalone software package, partially on a user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be located via any type of network.

[0080] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A control system for a rehabilitation device based on EEG signal emotion recognition, characterized in that, have: The synchronization module controls the image acquisition module and the EEG signal module to acquire and record signals synchronously over time. The image acquisition module acquires environmental images, performs event recognition on the environmental images, adds a timestamp recorded by the synchronization module, and then transmits the images to the event queue. The EEG signal module collects EEG signals, performs emotion recognition on the EEG signals, and transmits them to the emotion queue after adding the timestamp recorded by the synchronization module. The linkage processing module is used to create an emotion queue and an event queue, which are used to store the identified emotion information and event information, respectively. It also monitors the emotion queue, determines the timestamp of the target emotion when a target emotion is detected, searches for target event information within a preset time window before the timestamp in the event queue, monitors the duration of the target event based on the timestamps of subsequent events in the event queue, and generates control instructions according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

2. The system according to claim 1, characterized in that, The EEG signal module has a pre-identification unit and a buffer unit; The pre-identification unit is used to perform emotion pre-identification on EEG signals to obtain emotion information containing emotion value and emotion category, which is then transmitted in the form of emotion value key-value pairs and emotion category key-value pairs after being timestamped. The buffer unit buffers emotion categories based on emotion information within a continuous time period, and then transmits the buffered emotion categories and corresponding timestamps to the emotion queue.

3. The system according to claim 2, characterized in that, The buffer unit includes: a segmenter, a recognizer, a recorder, a controller, an assigner, and an output device; The process of buffering emotion categories based on emotional information over a continuous time period includes: The segmenter sets an emotion unification zone, and sets a buffer segment within the emotion unification zone that covers the midpoint of the emotion unification zone. The segmenter is used to determine the segment in which the sentiment value corresponding to each timestamp belongs in real time. The recognizer determines whether two emotional information messages in consecutive time are in the same segment and change direction. When it is recognized that they are in different segments, it generates either an out-of-area behavior or an in-area behavior based on their change direction. The recorder is used to record the entry behavior and the emotional category before entering the area when the entry behavior is detected; The controller is used to make judgments and controls based on the recognition results of the recognizer and the recording results of the recorder, including: If the current emotion information is in the same segment as the emotion information of the previous timestamp, the assigner is triggered to read the emotion category in the previous emotion information, assign a value to the emotion category key in the current emotion information, and then pass it to the output device. If the current emotional information is in a different segment than the emotional information at the previous timestamp, then it is determined whether the behavior currently identified by the recognizer is an out-of-area behavior. If it is an out-of-area behavior, it is determined whether the recognizer has identified a historical in-area behavior. If there is no historical in-area behavior, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If there is a historical in-area behavior, it is determined whether the change direction of the in-area behavior is the same as the change direction of the current out-of-area behavior. If the change direction is the same, then the output device is directly triggered to extract the timestamp and emotional category from the emotional information, associate them, and pass them to the emotional queue. If the change direction is different, then the assigner is triggered to read the emotional category before entering the area, assign a value to the emotional category key in the current emotional information, and then pass it to the output device. If it is not an outgoing behavior, the assigner is triggered to read the emotion category before entering the area, assign a value to the emotion category key in the current emotion information, and then pass it to the output device.

4. The system according to claim 2, characterized in that, The process of performing emotion pre-identification on EEG signals to obtain emotion information including emotion value and emotion category includes: Acquire the first EEG signal as the baseline emotion, and acquire the second EEG signal as the target emotion; The brainwave similarity between the currently acquired EEG signal and the first and second EEG signals is calculated, which are the first similarity and the second similarity. The emotion value is the ratio of the second similarity to the sum of the first and second similarities. If the emotion value is less than 0.5, the emotion category is pre-identified as the baseline emotion; if the emotion value is greater than 0.5, the emotion category is pre-identified as the target emotion.

5. The system according to claim 3, characterized in that, The segmenter sets an emotion normalization zone, including: Set the endpoint values ​​of the emotion normalization zone to 0 and 1; The direction of change is determined by the increasing or decreasing trend of sentiment values ​​across consecutive timestamps.

6. The system according to claim 5, characterized in that, The system also has a buffer segment adjustment unit for adjusting the endpoint values ​​of the buffer segment.

7. The system according to claim 6, characterized in that, The system also has a verification module with a sample set for verifying the stability of the rehabilitation control system. The adjustment of the endpoint values ​​of the buffer segment includes: Obtain the stability tendency of the user input, and adjust the endpoint values ​​of the buffer segment according to the stability tendency. If the user tends to increase stability, increase the distance between the endpoint values ​​of the buffer segment.

8. The system according to claim 7, characterized in that, The target emotion is a highly arousing emotion; The step of generating control commands based on the duration and a preset control strategy to enable the motion device controlled by the system to perform corresponding actions includes: If the duration exceeds the preset duration, a control command is generated to cause the motion device in the system to perform an avoidance action.

9. A control method for a rehabilitation device based on EEG signal emotion recognition, characterized in that, include: The image acquisition module and the EEG signal module are synchronized to acquire and record the time. The image acquisition module acquires environmental images, performs event recognition on the environmental images, adds a timestamp recorded by the synchronization module, and then transmits them to the event queue. EEG signals are collected through the EEG signal module, and after emotion recognition is performed on the EEG signals and the timestamp recorded by the synchronization module is added, the signals are transmitted to the emotion queue. An emotion queue and an event queue are created to store identified emotion information and event information, respectively. The emotion queue is monitored, and when a target emotion is detected, the timestamp of the target emotion is determined. The target event information within a preset time window before the timestamp is searched in the event queue. The duration of the target event is monitored based on the timestamps of subsequent events in the event queue. Control commands are generated according to the duration and a preset control strategy to control the movement device in the rehabilitation device to complete the corresponding action.

10. A computer device, characterized in that, The computer device includes: Processor; and, A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method of claim 9.

Images