User state management method and system based on brain-computer interface man-machine interaction system

Abstract

The invention provides a user state management method and system based on a brain-computer interface man-machine interaction system. The method comprises the steps that S1, user state information of a user in the man-machine interaction process is collected; s2, constructing a user state sequence according to the obtained user state information and time characteristics; s3, calculating a state excitation index according to the obtained user state sequence; s4, according to the obtained state excitation index, when the state excitation index meets an excitation characteristic condition, performing characteristic enhancement processing on the eye image sequence to obtain an enhanced eye image sequence; s5, extracting user state features according to the electroencephalogram signal sequence and the enhanced eye image sequence; s6, according to the obtained user state features, the trained state analysis model is adopted to analyze the state features, and a user state recognition result is obtained; and S7, performing corresponding control on the man-machine interaction system according to the user state recognition result. According to the invention, the accuracy and the intelligent level of further control of the man-machine interaction system by identifying the user state can be improved.

Description

Technical Field

[0001] This invention relates to the field of brain-computer interface human-computer interaction user state detection technology, and in particular to a user state management method and system based on a brain-computer interface human-computer interaction system. Background Technology

[0002] Brain-computer interface (BCI) is a technology that establishes a direct communication pathway between the brain and external devices. In non-invasive BCI human-computer interaction systems, users collect brain signals by wearing special devices, and then control the interactive system through these brain signals.

[0003] During brain-computer interface (BCI) human-computer interaction, monitoring and managing the user's state is a crucial factor affecting the user's experience. Currently, monitoring the user's state during human-computer interaction typically involves directly analyzing fluctuations in electroencephalogram (EEG) signals, or using facial image data from the interaction process, such as facial or eye images, to assist in analyzing the user's current mental state.

[0004] Existing technologies for user state analysis typically rely on single EEG signals or facial images to determine user state. However, in actual human-computer interaction, EEG signals also serve as control signals, and relying on a single EEG signal cannot accurately distinguish the user's state or command behavior. While facial images can reflect a user's expressions or actions to some extent, individual behavioral habits or facial expression habits (such as fidgeting, head shaking, or unconscious movements) can interfere with normal user state analysis and judgment (in extreme cases, the user may not be actively engaging in human-computer interaction but their emotional state may still be misjudged due to unconscious movements). Consequently, the technical means of determining the user's current state in actual human-computer interaction still suffer from insufficient accuracy. Summary of the Invention

[0005] To address the aforementioned problems, this invention aims to provide a user state management method and system based on a brain-computer interface human-computer interaction system.

[0006] The objective of this invention is achieved through the following technical solution: Firstly, a user state management method based on a brain-computer interface human-computer interaction system is proposed, including the following steps: S1 collects user status information during human-computer interaction, including user EEG data and user facial image data, wherein the user EEG signal data includes at least the user's forehead EEG signal data. S2 constructs a user state sequence based on the acquired user state information according to time characteristics, wherein the user state sequence includes an EEG signal sequence and an eye image sequence; S3 calculates the state excitation index based on the obtained user state sequence; S4 Based on the obtained state excitation index, when the state excitation index meets the excitation feature condition, the eye image sequence is subjected to feature enhancement processing to obtain an enhanced eye image sequence. S5 extracts user state features based on EEG signal sequences and enhanced eye image sequences, including EEG features, facial expression features, and confidence features. S6 uses a trained state analysis model to analyze the obtained user state features to obtain user state recognition results. S7 controls the human-computer interaction system accordingly based on the user status recognition results to adjust the user status.

[0007] Preferably, step S1 specifically includes: The system collects multi-channel EEG signal data from users during human-computer interaction using EEG devices. The collected multi-channel EEG signal data includes the user's frontal EEG signal data and is a continuous EEG signal sequence. The system uses a camera to collect facial image data of the user during human-computer interaction, and the collected facial image data is a continuous video image sequence.

[0008] Preferably, step S2 specifically includes: Based on the obtained user face image data, the position of the user's eyes is located, and the eye image is segmented from the user face image based on the located eye position to obtain an eye image sequence; Extracting the EEG signal sequence from the user's forehead from multi-channel EEG signal data; A user state sequence is constructed by aligning the EEG signal sequence and eye image sequence to the same time axis.

[0009] Preferably, step S3 specifically includes: The obtained user state sequence is processed by adding a data window and framing to obtain user state sequence segments corresponding to different times. Frequency band analysis was performed on the EEG signal sequence segments corresponding to each time point to extract the power of the EEG signal sequence segment in the Beta and Gamma bands. Based on the summation and normalization of the obtained Beta and Gamma band power, the EEG fluctuation characteristic values ​​at the corresponding time point were obtained. And based on the characteristic values ​​of brainwave fluctuations at each moment, brainwave fluctuation sequences are formed; For each time-series segment of the eye image, optical flow is used to track pixel motion in the eye image. The average pixel motion amplitude value within the sequence segment is obtained and normalized to obtain the eye change feature value at the corresponding time. And based on the characteristic values ​​of eye changes at each time point, an eye change feature sequence is formed; At the current moment Based on this, and according to the set reference time length The corresponding EEG wave sequence segments were extracted from the EEG wave sequence and the eye change feature sequence, respectively. and eye change feature sequence segments ; Based on the obtained EEG wave sequence segments and eye change feature sequence segments Calculate the state excitation index at the current moment, using the following function:

[0010] In the formula, Describe the current moment The state excitation index, express Eye change feature values ​​at time points, This represents the average eye change feature value at each time step in the eye image sequence segment. This represents the characteristic value of brainwave fluctuations at time t. Represents the time variable, and its range is to , Indicates the current moment. This indicates the set reference time length.

[0011] Preferably, step S4 includes: Based on the obtained state excitation index ,when When, then the eye image corresponding to the current time moment in the eye image sequence. Feature enhancement processing is performed to obtain an enhanced eye image sequence, where This indicates the set threshold for triggering a state.

[0012] Preferably, in step S4, feature enhancement processing is performed on the eye image at the current moment, specifically including: Extracting target eye image Pixel values ​​of each pixel ; Based on the target eye image Perform a copy operation to obtain a copy of the eye image. ; For eye image copy Perform Gaussian blur processing to obtain a Gaussian blurred copy of the eye image. ; Feature enhancement processing is performed on the target eye image, and the feature enhancement function used is:

[0013] In the formula, Represents the pixel after feature enhancement processing Pixel value at; Represents the pixels in the target eye image Pixel value at that location, Represents the pixels in the Gaussian blurred copy of the eye image. Pixel value at that location, This represents the set enhancement coefficient, where , Indicates the current time The state excitation index; The enhanced eye image at the current moment is obtained based on the pixel values ​​of each pixel after feature enhancement processing; The enhanced eye images are then used to replace the image data at the corresponding time points in the eye image sequence to obtain the enhanced eye image sequence.

[0014] Preferably, step S5 includes: Feature extraction is performed on the obtained EEG signal sequence to obtain EEG features. The EEG characteristics include at least one of the following: frequency domain characteristics, time domain characteristics, and entropy characteristics of the EEG sequence. Feature extraction is performed on the obtained enhanced eye image sequence to obtain expression features. The facial expression features include at least one of the following: eye opening and closing features, blinking features, and periorbital movement features. The confidence feature is obtained based on the state excitation index. ; User state features are constructed based on the obtained EEG features, facial expression features, and confidence features. .

[0015] Preferably, step S6 includes: Based on the obtained user state features, the state feature sequence is input into the trained state analysis model, which analyzes the user state to obtain the user state recognition result. The state analysis model comprises an input layer, a branch layer, a feature fusion layer, a decision layer, and an output layer. The input layer acquires user state features, outputting EEG features, facial expression features, and confidence features to their respective branch layers. The branch layers include parallel EEG, facial expression, and confidence branches. Each branch extracts its corresponding features through a multi-layer fully connected layer structure and outputs them to the feature fusion layer. The feature fusion layer fuses the high-level feature vectors from the three branches and outputs the fused feature vector to the decision layer. The decision layer further extracts the fused feature vector to obtain the final feature vector and uses a classifier to classify and identify the final feature vector, obtaining the probability distribution for each user state. The output layer outputs the user state identification result based on the probability distribution of the user state.

[0016] Preferably, step S7 includes: Based on the user status recognition results, when the user status recognition results are abnormal, the human-computer interaction system will be intervened, including pausing, stopping, adjusting or controlling environmental parameters, in order to adjust the user status.

[0017] Secondly, a user status management system based on a brain-computer interface human-computer interaction system is proposed, characterized by comprising a data acquisition module, a processing module, and a control module; wherein, The acquisition module is used to collect user status information during human-computer interaction. The user status information includes user EEG data and user facial image data. The user EEG signal data includes at least the user's forehead EEG signal data. The processing module is used to construct a user state sequence based on the acquired user state information according to time features. The user state sequence includes an EEG signal sequence and an eye image sequence. It calculates a state arousal index based on the obtained user state sequence. Based on the obtained state arousal index, when the state arousal index meets the arousal feature conditions, it performs feature enhancement processing on the eye image sequence to obtain an enhanced eye image sequence. It extracts user state features based on the EEG signal sequence and the enhanced eye image sequence, including EEG features, facial expression features, and confidence features. Based on the obtained user state features, it uses a trained state analysis model to analyze the state features and obtain the user state recognition result. The control module is used to control the human-computer interaction system according to the user status recognition results in order to adjust the user status.

[0018] The beneficial effects of this invention are as follows: The user state management method and system for brain-computer interface human-computer interaction systems proposed in this invention simultaneously collects the user's electroencephalogram (EEG) signals and facial image data as the basis for judging the user's state. Based on the collected EEG signal data and facial image data, the method extracts the forehead EEG signal sequence and the eye image sequence, aligns them to the same time axis to obtain the user state sequence, and calculates the state arousal index based on the user state sequence to represent the consistency between the changes in the user's current eye features and the changes in EEG signal features. Based on the state arousal index, the method performs targeted feature enhancement processing on the eye image data when the user experiences real state fluctuations, thereby improving the representation level of eye features during real state fluctuations. This helps to improve the accuracy of subsequent extraction of facial expression features and further user state analysis based on eye image data. Furthermore, the method obtains multi-dimensional feature fusion user state features based on EEG features, facial expression features, and confidence features, and combines these with a state analysis model to identify user state features. This effectively improves the accuracy and robustness of user state identification and also helps to improve the accuracy and intelligence level of further human-computer interaction system control based on the identified user state. Attached Figure Description

[0019] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the user state management method based on a brain-computer interface human-computer interaction system according to an embodiment of the present invention. Figure 2 This is a design structure diagram of the state analysis model shown in an embodiment of the present invention; Figure 3 This is a framework diagram of a user status management system based on a brain-computer interface human-computer interaction system, as shown in an embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described in conjunction with the following application scenarios.

[0022] See Figure 1 It demonstrates a user state management method based on a brain-computer interface human-computer interaction system, including the following steps: S1 collects user status information during human-computer interaction, including user EEG data and user facial image data, wherein the user EEG signal data includes at least the user's forehead EEG signal data. S2 constructs a user state sequence based on the acquired user state information according to time characteristics, wherein the user state sequence includes an EEG signal sequence and an eye image sequence; S3 calculates the state excitation index based on the obtained user state sequence; S4 Based on the obtained state excitation index, when the state excitation index meets the excitation feature condition, the eye image sequence is subjected to feature enhancement processing to obtain an enhanced eye image sequence. S5 extracts user state features based on EEG signal sequences and enhanced eye image sequences, including EEG features, facial expression features, and confidence features. S6 uses a trained state analysis model to analyze the obtained user state features to obtain user state recognition results. S7 controls the human-computer interaction system accordingly based on the user status recognition results to adjust the user status.

[0023] The above-described embodiments of the present invention propose a user state management method for brain-computer interface human-computer interaction systems. This method simultaneously collects the user's electroencephalogram (EEG) signals and facial image data as the basis for determining the user's state. Based on the collected EEG and facial image data, a forehead EEG signal sequence and an eye image sequence are extracted and aligned to the same time axis to obtain a user state sequence. A state excitation index is calculated based on the user state sequence to represent the consistency between changes in the user's current eye features and changes in EEG signal features. Based on the state excitation index, targeted feature enhancement processing is performed on the eye image data when the user experiences real state fluctuations, thereby improving the representation level of eye features during real state fluctuations. This helps improve the accuracy of subsequent extraction of facial expression features and further user state analysis based on eye image data. Furthermore, a multi-dimensional feature fusion user state feature is obtained based on EEG features, facial expression features, and confidence features. Combined with a state analysis model, the user state feature is identified, which effectively improves the accuracy and robustness of user state identification. This also helps improve the accuracy and intelligence level of further human-computer interaction system control based on the identified user state.

[0024] In one scenario, the brain-computer interface (BCI) human-computer interaction system is a non-invasive BCI system. The system uses an EEG device to record the user's electroencephalogram (EEG) signals in real time during the interaction. The recorded EEG signals are used to control the BCI system and monitor the user's state. Simultaneously, the system is equipped with a camera, which is positioned over the user's face during interaction to capture real-time facial images. These recorded images are then used to further monitor the user's state.

[0025] Preferably, step S1 specifically includes: The system collects multi-channel EEG signal data from users during human-computer interaction using EEG devices. The collected multi-channel EEG signal data includes the user's frontal EEG signal data and is a continuous EEG signal sequence. The system uses a camera to collect facial image data of the user during human-computer interaction, and the collected facial image data is a continuous video image sequence.

[0026] By using an external camera, facial image data can be collected from the user during the operation of the brain-computer interface-based human-computer interaction system, serving as the basis for subsequent user status analysis and judgment. For the collection of the user's electroencephalogram (EEG) signals, the data set of the user's frontal EEG signals is selected as the basis to determine the user's current brain activity state.

[0027] Preferably, step S2 specifically includes: Based on the obtained user face image data, the position of the user's eyes is located, and the eye image is segmented from the user face image based on the located eye position to obtain an eye image sequence; Extracting the EEG signal sequence from the user's forehead from multi-channel EEG signal data; A user state sequence is constructed by aligning the EEG signal sequence and eye image sequence to the same time axis.

[0028] Specifically, based on the detected eye position, a rectangular image region of a specified size is segmented with the eye position as the center to serve as the eye image. The segmented eye image includes part of the user's eye region and the skin region around the eyes.

[0029] In one scenario, the method of identifying the eye position and segmenting the eye image based on the face image can be achieved by template matching or other existing image segmentation models, and this invention does not specifically limit the method.

[0030] In one scenario, the method for acquiring eye images can be to detect the center position of the eyes based on the acquired key points, and then use the center position of the eyes as a basis to segment a rectangular area from the face image with a length of twice the distance between the eyes (pixel distance) and a width of once the distance between the eyes (pixel distance) as the eye image. The center position of the eye image is aligned with the center point between the eyes.

[0031] Based on the obtained user state sequence, the user's EEG signal data and eye image data can be aligned to the same time axis, providing a basis for further consistency detection of facial expression changes and EEG fluctuations.

[0032] Preferably, the procedure before step S3 includes: The obtained EEG signal sequence is preprocessed, including filtering, artifact removal and noise interference processing, and the preprocessed EEG signal sequence is applied to subsequent processing.

[0033] The obtained EEG signal sequence can be processed to remove noise interference and improve the quality of the EEG signal sequence.

[0034] Considering that existing technologies rely on a single user eye image for user status analysis, user behavior habits or facial expressions (such as fidgeting, head shaking, or unconscious movements) can interfere with normal user status analysis and judgment (in extreme cases, the user's emotional state may be misjudged even without any actual human-computer interaction control due to the user's unconscious movements), the technical means of judging the user's current status based on the user's eye image may result in misjudgment or inaccurate identification during actual human-computer interaction.

[0035] Based on observations and analysis of real-world users of human-computer interaction systems, it was found that the timing of changes in facial expressions and brainwave fluctuations coincides when users experience real-world state fluctuations. Therefore, by first analyzing the consistency between changes in user facial expressions and brainwave fluctuations, we can use these as criteria to determine whether a user has experienced a real state change, thereby improving the accuracy of user state recognition.

[0036] Preferably, step S3 specifically includes: The obtained user state sequence is processed by adding a data window and framing to obtain user state sequence segments corresponding to different times; among them, the user state sequence segment corresponding to a time includes an EEG signal sequence segment and an eye image sequence segment; Frequency band analysis was performed on the EEG signal sequence segments corresponding to each time point to extract the power of the EEG signal sequence segment in the Beta and Gamma bands. Based on the summation and normalization of the obtained Beta and Gamma band power, the EEG fluctuation characteristic values ​​at the corresponding time point were obtained. And based on the characteristic values ​​of brainwave fluctuations at each moment, brainwave fluctuation sequences are formed; For each time-series segment of the eye image, optical flow is used to track pixel motion in the eye image. The average pixel motion amplitude value within the sequence segment is obtained and normalized to obtain the eye change feature value at the corresponding time. And based on the characteristic values ​​of eye changes at each time point, an eye change feature sequence is formed; At the current moment Based on this, and according to the set reference time length The corresponding EEG wave sequence segments were extracted from the EEG wave sequence and the eye change feature sequence, respectively. and eye change feature sequence segments ; Based on the obtained EEG wave sequence segments and eye change feature sequence segments Calculate the state excitation index at the current moment, using the following function:

[0037] In the formula, Describe the current moment The state excitation index, express Eye change feature values ​​at time points, This represents the average eye change feature value at each time step in the eye image sequence segment. This represents the standard deviation of the characteristic values ​​of eye changes at each time step in a segment of an eye image sequence. This represents the characteristic value of brainwave fluctuations at time t. This represents the average EEG fluctuation characteristic value at each moment in the EEG fluctuation sequence segment. This represents the standard deviation of the EEG eigenvalues ​​at each time step within the EEG eigenvalue sequence segment. This represents the set standard deviation threshold. Represents the time variable, and its range is to , Indicates the current moment. This indicates the set reference time length.

[0038] This process involves frequency band analysis of the EEG signal sequence segments at each time point. Based on the extracted EEG signal sequence segments, Beta and Gamma filters are used for bandpass filtering. The power magnitude at each sampling point is then calculated based on the filtered signal sequence segments, and the frequency band power of the corresponding filtered signal sequence segment is obtained based on the average power magnitude in the sequence segment. The obtained Beta and Gamma frequency band powers are then weighted and summed, and the sum is normalized to obtain the EEG fluctuation characteristic value at the corresponding time point.

[0039] Optionally, the normalization of the obtained power can be based on the power and sum values ​​within a time period T2, using that time period […]. T2~ The maximum and minimum values ​​within the range are used as the basis for normalization; or the minimum and maximum values ​​set based on experience are used as the basis for normalization to obtain the EEG fluctuation characteristic values. .

[0040] In one scenario, targeting the characteristic values ​​of brainwave fluctuations The acquisition can be ,in and These represent the Beta band power and Gamma band power at the current moment, respectively. and These represent the preset reference maximum power value and minimum power value, respectively.

[0041] Similarly, optionally, normalization processing can be performed on the average pixel motion amplitude value. This can be based on the average pixel motion amplitude value within a time period T2, using the average pixel motion amplitude value within that time period as a basis. T2~ The maximum and minimum values ​​are used as the basis for normalization; or the maximum and minimum values ​​set based on experience are used as the basis for normalization to obtain the characteristic values ​​of eye changes. .

[0042] In one scenario, to avoid the influence of blinking on the acquisition of eye change feature values, observation points are first marked in the periorbital region (non-eye region) of the eye image. The motion amplitude values ​​of each monitoring point are obtained using optical flow based on their positional changes. Then, the average motion amplitude value of each monitoring point is normalized to obtain the eye change feature values. .

[0043] In one scenario, taking a sampling frequency of 50Hz as an example, the length of the data window used in the data window frame processing is 10, that is, each data window corresponds to 0.2s; the data windows partially overlap, so that the interval between adjacent data frames is 0.1s.

[0044] In one scenario, a reference time length is set. The state excitation index is calculated based on the EEG fluctuation sequence and eye change feature sequence within 0.5 seconds prior to the current moment.

[0045] In one scenario, a standard deviation threshold is set. Specifically .

[0046] In the above embodiments of the present invention, based on the obtained user state sequence, brainwave fluctuation feature values ​​and eye change feature values ​​are extracted from the electroencephalogram (EEG) signal sequence and eye image sequence, respectively, to characterize the user's EEG fluctuations and eye feature changes at different times, reflecting the degree of EEG and eye changes at different times. Based on the obtained EEG fluctuation feature values ​​and eye change feature values, the user's current state arousal index is calculated using the proposed state arousal index calculation function. The state arousal index can accurately reflect the consistency between the user's current EEG changes and facial expression changes, providing a basis for subsequent enhancement processing of the user's eye images and user state analysis.

[0047] Preferably, step S4 includes: Based on the obtained state excitation index ,when When, then the eye image corresponding to the current time moment in the eye image sequence. Feature enhancement processing is performed to obtain an enhanced eye image sequence, where This indicates the set threshold for triggering a state.

[0048] In the actual analysis, it was further discovered that when a user's state fluctuates, the corresponding facial expression changes are not necessarily proportional to the state fluctuation. That is, even slight changes in a user's facial expression can still represent the user's representation of the state change. However, in conventional eye image preprocessing, indiscriminately sharpening or filtering the image cannot accurately distinguish and enhance useful features in the image, and may even have the opposite effect.

[0049] Preferably, the range of the set state excitation threshold is: , Preferably, in step S4, feature enhancement processing is performed on the eye image at the current moment, specifically including: Extract target eye image Pixel values ​​of each pixel ; Based on the target eye image Perform a copy operation to obtain a copy of the eye image. ; For eye image copy Perform Gaussian blur processing to obtain a Gaussian blurred copy of the eye image. ; Feature enhancement processing is performed on the target eye image, and the feature enhancement function used is:

[0050] In the formula, Represents the pixel after feature enhancement processing Pixel value at; Represents the pixels in the target eye image Pixel value at that location, Represents the pixels in the Gaussian blurred copy of the eye image. Pixel value at that location, This represents the set enhancement coefficient, where , Indicates the current time The state excitation index; The enhanced eye image at the current moment is obtained based on the pixel values ​​of each pixel after feature enhancement processing; The enhanced eye images are then used to replace the image data at the corresponding time points in the eye image sequence to obtain the enhanced eye image sequence.

[0051] The Gaussian kernel size used for Gaussian filtering of the eye image copy is [size missing]. or The standard deviation was set as .

[0052] Optionally, step S4 may also include: Based on the obtained state excitation index ,when At that time, no feature enhancement processing or conventional filtering processing is performed on the eye image corresponding to the current moment. This yields an updated sequence of enhanced eye images.

[0053] In the above embodiments of the present invention, when a user's brainwave fluctuations and facial expression fluctuations show consistent changes, it indicates that the user is experiencing a state fluctuation. Therefore, by judging the eye images at key moments based on the obtained state excitation index and performing targeted feature enhancement processing on the eye images, key features in the eye images can be sharpened when the user's state fluctuates, improving the representation level of eye image features (especially improving the representation level of subtle changes), which helps to improve the accuracy of subsequent user state recognition based on eye images. Conversely, when the user's state does not fluctuate, normal methods or fuzzy filtering are used to avoid amplifying or suppressing interfering features in the image (such as mere random movements or unconscious expressions), thereby reducing the possibility of misjudgment of the user's state.

[0054] Based on the state excitation index proposed in this invention, targeted feature enhancement processing is performed on eye images when users experience real-world state fluctuations. Compared with traditional non-targeted, one-size-fits-all image preprocessing methods (such as indiscriminate sharpening), this method avoids amplifying the user's own distracting facial expressions (such as random movements or unconscious expressions unrelated to human-computer interaction brain activity), preventing subsequent misjudgments of the user's true state changes based on these distracting expressions. Furthermore, it can also specifically enhance micro-expression changes under real-world state fluctuations (avoiding additional noise interference), thereby improving the recognition results of micro-expressions under state changes, thus enhancing the accuracy and robustness of user state recognition results and improving user state recognition performance.

[0055] Preferably, step S5 includes: Feature extraction is performed on the obtained EEG fluctuation sequence to obtain EEG features. The EEG characteristics include at least one of the following: frequency domain characteristics, time domain characteristics, and entropy characteristics of the EEG sequence. Feature extraction is performed on the obtained enhanced eye image sequence to obtain expression features. The facial expression features include at least one of the following: eye opening and closing features, blinking features, and periorbital movement features. The confidence feature is obtained based on the state excitation index. ; User state features are constructed based on the obtained EEG features, facial expression features, and confidence features. .

[0056] Optionally, to avoid the confidence feature from having a negative impact on the state recognition result under special circumstances, a minimum boundary is specifically set for the state excitation index, making... ,in Set to the minimum value, i.e. ,For example , .

[0057] Optionally, when extracting user state features, the EEG fluctuation sequence and enhanced eye image sequence over a past period can be used as a benchmark to extract the corresponding user state features based on the data sequence over this period. In one scenario, the selected time period length is 2 seconds, 3 seconds, etc.

[0058] Optionally, feature extraction can be performed based on the EEG fluctuation sequence. For frequency domain feature extraction, the power spectral density and band power of each channel in different rhythm frequency bands can be calculated after performing a short-time Fourier transform on the EEG fluctuation sequence (such as the relative power of θ, α, β, γ and the band power ratio). For time domain feature extraction, the mean, variance, skewness, kurtosis, etc. of the EEG fluctuation sequence can be statistically analyzed based on the amplitude changes of the EEG fluctuation sequence as time domain features. For entropy feature extraction, the approximate entropy, sample entropy, spectral entropy, fuzzy entropy, etc. of the EEG fluctuation sequence can be calculated as entropy features.

[0059] Optionally, feature extraction is performed based on the enhanced eye image sequence. For eye opening and closing feature extraction, key point detection can be performed on the eye image, and the corners and upper and lower edges of the eyes can be located based on the position of the key points. The aspect ratio of the eyes and the opening and closing angle of the eyes are calculated based on the data of the upper and lower edges of the eyes as eye opening and closing features. For blinking feature extraction, the blinking frequency and the duration of eye closure are calculated based on the changes in the aspect ratio of the eyes and the corresponding time axis features. For the extraction of periorbital movement features, a trained CNN neural network is used to perform user state recognition on the eye images to obtain the corresponding periorbital movement features, which include frowning, staring, closing the eyes, and no movement.

[0060] In one scenario, the feature extraction of the above-mentioned EEG sequence and enhanced eye image sequence can also be completed using other pre-trained feature extraction models or algorithms in the prior art to obtain the feature data required for subsequent user state recognition. This invention is not specifically limited to these methods.

[0061] Optionally, for the extraction of confidence features, the state excitation index at the current moment can be used as the confidence feature; alternatively, the maximum or average value of the corresponding state excitation index over a period of time starting from the current moment can be used as the confidence feature.

[0062] The above embodiments of the present invention construct user state features based on the obtained EEG features, facial expression features, and confidence features, which can obtain a multi-dimensional feature vector that accurately reflects the user's current state, and further perform subsequent user state recognition processing to improve the accuracy of user state recognition.

[0063] Preferably, step S6 includes: Based on the obtained user state features, the state feature sequence is input into the trained state analysis model, which analyzes the user state to obtain the user state recognition result. The state analysis model comprises an input layer, a branch layer, a feature fusion layer, a decision layer, and an output layer. The input layer acquires user state features, outputting EEG features, facial expression features, and confidence features to their respective branch layers. The branch layers include parallel EEG, facial expression, and confidence branches. Each branch extracts its corresponding features through a multi-layer fully connected layer structure and outputs them to the feature fusion layer. The feature fusion layer fuses the high-level feature vectors from the three branches and outputs the fused feature vector to the decision layer. The decision layer further extracts the fused feature vector to obtain the final feature vector and uses a classifier to classify and identify the final feature vector, obtaining the probability distribution for each user state. The output layer outputs the user state identification result based on the probability distribution of the user state.

[0064] In one scenario, see Figure 2 The design of each layer of the state analysis model is as follows: The design for the branching layers includes two fully connected layers: the EEG branch and the facial expression branch. The first fully connected layer has 64 neurons and uses the ReLU activation function; the second fully connected layer has 32 neurons and uses the ReLU activation function. The facial expression branch includes two fully connected layers: the third fully connected layer has 64 neurons and uses the ReLU activation function; the fourth fully connected layer has 32 neurons and uses the ReLU activation function. The confidence branch includes a fifth fully connected layer. The feature fusion layer has 16 nodes and uses ReLU activation. It concatenates the high-level feature vectors from the three branches, outputting an 80-dimensional fused feature vector to the decision layer. The decision layer consists of a sixth fully connected layer and a seventh fully connected layer connected sequentially. The sixth fully connected layer has 64 neurons and uses ReLU activation; the seventh fully connected layer has 32 neurons and uses ReLU activation. The final feature vector output from the seventh fully connected layer is input to a softmax layer to obtain the probability distribution for each user state. User states include focused, relaxed, tired, panicked, frustrated, and tense. The output layer selects and outputs the category with the highest probability based on the probability distribution of user states as the user state recognition result.

[0065] For training the state analysis model, user state features obtained in the same way and the corresponding user state recognition results can be used as the training set to complete the training of the state analysis model. The accuracy of the state analysis model results can be tested through the test set. Once the test is passed, the trained state analysis model is obtained.

[0066] In the above embodiments of the present invention, based on the obtained user state features, the aforementioned state analysis model is used to analyze the user state features. The constructed state analysis model, based on the extracted user state features, designs a multi-layer fully connected network to integrate and classify the feature vectors, thereby accurately analyzing the user's current state. Furthermore, during the feature fusion process, confidence features are also incorporated to further influence the consistency between EEG features and facial expression features, thereby improving the accuracy and reliability of user state recognition.

[0067] Preferably, step S7 includes: Based on the user status recognition results, when the user status recognition results are abnormal, the human-computer interaction system will be intervened, including pausing, stopping, adjusting or controlling environmental parameters, in order to adjust the user status.

[0068] In one scenario, when the user's state recognition result is fear, the human-computer interaction system is immediately adjusted to reduce the playback volume; if the user remains in a state of continuous fear, the content being played in the current human-computer interaction system is stopped.

[0069] In another scenario, when the user's state recognition result indicates tension, the human-computer interaction system is controlled to adjust environmental parameters, such as room temperature and ambient light, to a comfortable level, or the current interaction strategy is adjusted to improve the user's comfort during the human-computer interaction process.

[0070] See Figure 3 It illustrates a user status management system based on a brain-computer interface human-computer interaction system, including a data acquisition module, a processing module, and a control module; among which, The acquisition module is used to collect user status information during human-computer interaction. The user status information includes user EEG data and user facial image data. The user EEG signal data includes at least the user's forehead EEG signal data. The processing module is used to construct a user state sequence based on the acquired user state information according to time features. The user state sequence includes a frontal EEG signal sequence and an eye image sequence. It calculates a state arousal index based on the obtained user state sequence. Based on the obtained state arousal index, when the state arousal index meets the arousal feature conditions, it performs feature enhancement processing on the eye image sequence to obtain an enhanced eye image sequence. It extracts user state features based on the EEG fluctuation sequence and the enhanced eye image sequence, including EEG features, facial expression features, and confidence features. Based on the obtained user state features, it uses a trained state analysis model to analyze the state features and obtain the user state recognition result. The control module is used to control the human-computer interaction system according to the user status recognition results in order to adjust the user status.

[0071] It should be noted that the functional units / modules in the various embodiments of the present invention can be integrated into one processing unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated into one unit / module. The integrated unit / module described above can be implemented in hardware or in the form of software functional units / modules.

[0072] From the above description of the embodiments, those skilled in the art should understand that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor can be implemented in one or more of the following units: application-specific integrated circuit (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), field-programmable gate array (FPGA), processor, controller, microcontroller, microprocessor, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments can be implemented by a computer program instructing the associated hardware. During implementation, the program described above can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium that is accessible to a computer. Computer-readable media can include, but is not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should be able to analyze that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A user state management method based on a brain-computer interface human-computer interaction system, characterized in that, The method comprises the following steps: S1: collecting user state information of a user in a human-computer interaction process, wherein the user state information comprises user electroencephalogram (EEG) data and user facial image data, and the user EEG signal data at least comprises user forehead EEG signal data; S2: constructing a user state sequence according to the obtained user state information according to time characteristics, wherein the user state sequence comprises an EEG signal sequence and an eye image sequence; S3: calculating a state excitation index according to the obtained user state sequence; S4: when the state excitation index meets excitation characteristic conditions, performing feature enhancement processing on the eye image sequence according to the obtained state excitation index to obtain an enhanced eye image sequence; S5: extracting user state features according to the EEG signal sequence and the enhanced eye image sequence, wherein the user state features comprise EEG features, expression features and confidence features; S6: analyzing the state features by using a trained state analysis model according to the obtained user state features to obtain a user state recognition result; S7: controlling the human-computer interaction system according to the user state recognition result to adjust the user state.

2. The user state management method of a brain-computer interface based human-computer interaction system according to claim 1, characterized in that, Step S1 specifically comprises: collecting multi-channel EEG signal data of the user in the human-computer interaction process based on an EEG device, wherein the collected multi-channel EEG signal data comprises user forehead EEG signal data, and the collected multi-channel EEG signal data is a continuous EEG signal sequence; collecting facial image data of the user in the human-computer interaction process based on a camera, wherein the collected facial image data is a continuous video image sequence.

3. The user state management method of a brain-computer interface based human-computer interaction system according to claim 2, characterized in that, Step S2 specifically comprises: locking the user eye position according to the obtained user facial image data, and segmenting an eye image from the user facial image according to the locked eye position to obtain an eye image sequence; extracting a forehead EEG signal sequence from the multi-channel EEG signal data; aligning the EEG signal sequence and the eye image sequence to the same time axis to construct the user state sequence.

4. The user state management method of a brain-computer interface based human-computer interaction system according to claim 3, characterized in that, Step S3 specifically comprises: performing data window frame processing on the obtained user state sequence to obtain user state sequence segments corresponding to different time points; performing frequency band analysis on the EEG signal sequence segments corresponding to each time point, extracting the power of the Beta frequency band and the Gamma frequency band of the EEG signal sequence segment, and summing and normalizing the obtained Beta frequency band power and Gamma frequency band power to obtain an EEG fluctuation feature value corresponding to each time point, and composing an EEG fluctuation sequence according to the EEG fluctuation feature values of each time point; For each time corresponding eye image sequence segment, the optical flow method is used to track the pixel motion in the eye image, the average pixel motion amplitude value in the sequence segment is obtained, and the eye change feature value corresponding to the time is obtained after normalization processing , and the eye change feature sequence is composed according to the eye change feature value of each time. Based on the current time , according to the set reference time length , the corresponding electroencephalogram fluctuation sequence segment and the eye change feature sequence segment are respectively intercepted from the electroencephalogram fluctuation sequence and the eye change feature sequence. According to the obtained brain wave fluctuation sequence segment and eye change feature sequence segment Calculate the calculation state excitation index of the current moment, wherein the state excitation index calculation function adopted is: wherein, the state excitation index of the current time point, the state excitation index of the current time point, the eye change feature value of the time point, the average eye change feature value of each time point in the eye image sequence segment, the average eye change feature value of each time point in the eye image sequence segment, the brain wave fluctuation feature value of the time point t, the time point variable, whose range is to , the current time point, the set reference time length.

5. The user state management method of a brain-computer interface based human-computer interaction system according to claim 4, characterized in that, Step S4 comprises: According to the obtained state excitation index When , the eye image corresponding to the current time in the eye image sequence is subjected to feature enhancement processing to obtain an enhanced eye image sequence, wherein represents a set state excitation threshold.

6. The user state management method of a brain-computer interface based human-computer interaction system according to claim 5, characterized in that, In step S4, the feature enhancement processing on the eye image at the current time point specifically comprises: Extracting a target ocular image pixel values of the pixels ; According to the target eye image A copy operation is performed to obtain an eye image copy ; An eye image copy is obtained Gaussian blur processing is performed to obtain a Gaussian blur processed eye image copy ; performing feature enhancement processing on the target eye image, wherein the feature enhancement function used is: wherein, represents the pixel value at the pixel point after the feature enhancement processing; represents the pixel value at the pixel point in the target eye image, represents the pixel value at the pixel point in the copy of the eye image after the Gaussian blur processing, represents the set enhancement coefficient, wherein , represents the state excitation index at the current moment . obtaining an enhanced eye image corresponding to the current time point according to the pixel values of each pixel point after the feature enhancement processing; and replacing the image data of the eye image sequence at the corresponding time point with the obtained enhanced eye image to obtain an enhanced eye image sequence.

7. The user state management method of a brain-computer interface based human-computer interaction system according to claim 5, characterized in that, Step S5 comprises: According to the obtained electroencephalogram signal sequence, feature extraction is performed to obtain electroencephalogram features ; wherein the electroencephalogram features include at least one of frequency domain features, time domain features and entropy features of the electroencephalogram wave sequence According to the obtained enhanced eye image sequence, feature extraction is performed to obtain expression features ; wherein the expression features include at least one of eye opening degree features, blinking features, and periorbital movement features. According to the resulting state excitation index a confidence feature is derived ; According to the obtained electroencephalogram features, expression features and confidence features, a user state feature is constructed .

8. The user state management method of a brain-computer interface based human-computer interaction system according to claim 7, characterized in that, Step S6 comprises: inputting the state feature sequence into the trained state analysis model according to the obtained user state features, and analyzing the user state by the state analysis model to obtain a user state recognition result; The state analysis model comprises an input layer, a branch layer, a feature fusion layer, a decision layer and an output layer; the input layer is used to acquire user state features, and electroencephalogram features, expression features and confidence features are respectively output to corresponding branch layers; the branch layer comprises electroencephalogram branches, expression branches and confidence branches designed in parallel; the three branches respectively extract corresponding features through a multi-layer full connection layer structure and output to the feature fusion layer; the feature fusion layer fuses high-level feature vectors of the three branches and further outputs the fused feature vectors to the decision layer; the decision layer further extracts the fused feature vectors to obtain final feature vectors, and uses a classifier to classify and identify the extracted final feature vectors to obtain a probability distribution corresponding to each user state; and the output layer outputs a user state recognition result according to the probability distribution of the user state.

9. The user state management method of a brain-computer interface based human- computer interaction system according to claim 1, characterized in that, Step S7 comprises: According to the user state recognition result, when the user state recognition result is abnormal, the human-computer interaction system is intervened, including pausing, stopping, adjusting or environmental parameter control, to adjust the user state.

10. A user state management system for a brain-computer interface human- computer interaction system, characterized in that, The system comprises a collection module, a processing module and a control module; wherein The collection module is used to collect user state information of a user in a human-computer interaction process, wherein the user state information comprises user electroencephalogram data and user facial image data, and the user electroencephalogram signal data at least comprises user forehead electroencephalogram signal data; The processing module is used to construct a user state sequence according to the acquired user state information in terms of time characteristics, wherein the user state sequence comprises an electroencephalogram signal sequence and an eye image sequence; to calculate a state excitation index according to the obtained user state sequence; to perform feature enhancement processing on the eye image sequence to obtain an enhanced eye image sequence when the state excitation index meets excitation feature conditions; to extract user state features including electroencephalogram features, expression features and confidence features according to the electroencephalogram signal sequence and the enhanced eye image sequence; and to obtain a user state recognition result by analyzing the state features using a trained state analysis model; The control module is used to control the human-computer interaction system according to the user state recognition result to adjust the user state.

Images