Unsupervised continuous emotion electroencephalogram analysis method and device based on deep reinforcement learning

Abstract

The invention provides an unsupervised continuous emotion electroencephalogram analysis method and device based on deep reinforcement learning, and the method comprises the steps: in a prototype learning stage, obtaining sample features of time sequence signals corresponding to sample objects, carrying out the clustering processing of the sample features based on a K-Means algorithm, obtaining the global distribution information of emotion information in the time sequence signals, and carrying out the clustering processing of the global distribution information; in the reinforcement learning stage, a distribution-prototype reward function is determined according to the global distribution information, an Actor-Critic framework is adopted, a near-end strategy optimization algorithm PPO is used, an Actor executes a current action for a given time sequence signal based on a current strategy, Critic determines a current score of the current action according to a distribution-prototype reward fed back by the environment, and the current score of the current action is determined according to the distribution-prototype reward fed back by the environment. And the Actor performs strategy learning based on the current score until a preset iteration termination condition is reached, and a model used for determining key signals in the target time sequence signals according to the target time sequence signals is obtained. Therefore, the accuracy and reliability of the finally obtained model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to an unsupervised continuous emotional EEG analysis method and device based on deep reinforcement learning. Background Art

[0002] Human emotion is a continuous dynamic process characterized by complex interactions between the human body and the external environment. How to identify task-related emotional fragments from continuous temporal signals is a major challenge.

[0003] A major shortcoming of existing research is that it relies on static labeling methods, that is, assigning a single fixed label to the entire time series signal. This method cannot capture the dynamic characteristics of human emotions, because emotional states are dynamic in nature and constantly change with internal cognitive processes and external stimuli. In addition, continuous time series signals often contain states that are irrelevant to the specific task being studied. These irrelevant states introduce noise and confounding factors, which undermine the accuracy and reliability of emotion recognition models.

[0004] When task-irrelevant temporal signals are included in the training data, they introduce irrelevant information that may degrade the model performance. As a result, the model may incorrectly associate these irrelevant patterns with emotional states, thereby diverting attention away from the true emotional dynamics that are relevant to the task, resulting in reduced accuracy in emotion recognition.

[0005] Therefore, developing an AI approach that can dynamically adapt to the dynamics of human emotions and accurately identify key task-relevant signals is crucial to improving the accuracy and effectiveness of emotion recognition models.

[0006] The content of the background technology section is only the information known to the inventor personally, and does not mean that the above information has entered the public domain before the application date of this disclosure, nor does it mean that it can become the prior art of the present disclosure. Summary of the invention

[0007] The present disclosure provides an unsupervised continuous emotional EEG analysis method and device based on deep reinforcement learning, which is used to solve at least one of the above-mentioned technical problems.

[0008] In a first aspect, the present disclosure provides an unsupervised continuous emotional EEG analysis method based on deep reinforcement learning, the method comprising a prototype learning stage based on a heuristic algorithm and a reinforcement learning stage based on a distribution-prototype reward function; In the prototype learning stage: obtaining sample features of the time series signal corresponding to each sample object, and clustering the sample features based on the K-Means algorithm to obtain global distribution information of the emotional information in the time series signal; In the reinforcement learning stage: a distribution-prototype reward function is determined according to the global distribution information, and the Actor-Critic framework is used to use the proximal policy optimization algorithm PPO, so that the Actor performs a current action based on the current strategy for a given timing signal, the Critic determines a current score of the current action based on the distribution-prototype reward of environmental feedback, and the Actor performs strategy learning based on the current score until a preset termination iteration condition is reached, thereby obtaining a model for determining a key signal in the target timing signal according to the target timing signal, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

[0009] In some embodiments, clustering the sample features based on the K-Means algorithm to obtain global distribution information of the emotion information in the time series signal includes: Inputting the sample features into the K-Means algorithm to obtain an emotion prototype set, wherein the emotion prototype set represents multiple emotion categories and a prototype feature vector of each emotion category, and the prototype feature vector represents a cluster center; Calculate the within-cluster variance for each emotion category; The global distribution information of the emotion information in the time series signal is determined based on the prototype feature vector and the intra-cluster variance of each emotion category.

[0010] In some embodiments, the distribution-prototype reward function includes a Center reward function and an Inter-Intra reward function; Among them, the Center reward function is used to characterize the intensity of the emotional information contained in the given time series signal; the Inter-Intra reward function is used to characterize the relative confidence that the given time series signal belongs to a specific emotion category calculated in the prototype learning stage.

[0011] In some embodiments, the probability ratio between the strategy of the current iteration and the strategy of the previous iteration is less than a preset probability ratio threshold, so that the trajectory is reused multiple times.

[0012] In some embodiments, the method further comprises: Evaluating the pros and cons of the current strategy based on the temporal difference error; An advantage function is determined based on the generalized advantage estimate and the temporal difference error, so that the Actor updates the current policy based on the advantage function to perform policy learning.

[0013] In some embodiments, the update range of the Actor's strategy probability in each iteration is determined based on a preset upper limit and a lower limit of the ratio of the new strategy to the old strategy, and is represented by an objective function with clipping.

[0014] In some embodiments, the policy learning is constrained based on a regularization term, wherein the regularization term includes a desired proportion of the selected key signal.

[0015] In a second aspect, the present disclosure provides an unsupervised continuous emotional EEG analysis device based on deep reinforcement learning, comprising: A prototype learning unit, used to obtain sample features of the time series signal corresponding to each sample object in the prototype learning stage, and cluster the sample features based on the K-Means algorithm to obtain global distribution information of the emotional information in the time series signal; A reinforcement learning unit is used to determine the distribution-prototype reward function according to the global distribution information during the reinforcement learning stage based on the distribution-prototype reward function, and adopt the Actor-Critic framework using the proximal policy optimization algorithm PPO, so that the Actor performs a current action for a given timing signal based on the current strategy, the Critic determines the current score of the current action based on the environmental feedback distribution-prototype reward, and the Actor performs strategy learning based on the current score until a preset termination iteration condition is reached, thereby obtaining a model for determining the key signal in the target timing signal according to the target timing signal, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

[0016] In some embodiments, the prototype learning unit is specifically used to: Inputting the sample features into the K-Means algorithm to obtain an emotion prototype set, wherein the emotion prototype set represents multiple emotion categories and a prototype feature vector of each emotion category, and the prototype feature vector represents a cluster center; Calculate the within-cluster variance for each emotion category; The global distribution information of the emotion information in the time series signal is determined based on the prototype feature vector and the intra-cluster variance of each emotion category.

[0017] In some embodiments, the distribution-prototype reward function includes a Center reward function and an Inter-Intra reward function; Among them, the Center reward function is used to characterize the intensity of the emotional information contained in the given time series signal; the Inter-Intra reward function is used to characterize the relative confidence that the given time series signal belongs to a specific emotion category calculated in the prototype learning stage.

[0018] In some embodiments, the probability ratio between the strategy of the current iteration and the strategy of the previous iteration is less than a preset probability ratio threshold, so that the trajectory is reused multiple times.

[0019] In some embodiments, the reinforcement learning unit is further configured to: Evaluating the pros and cons of the current strategy based on the temporal difference error; An advantage function is determined based on the generalized advantage estimate and the temporal difference error, so that the Actor updates the current policy based on the advantage function to perform policy learning.

[0020] In some embodiments, the update range of the Actor's strategy probability in each iteration is determined based on a preset upper limit and a lower limit of the ratio of the new strategy to the old strategy, and is represented by an objective function with clipping.

[0021] In some embodiments, the policy learning is constrained based on a regularization term, wherein the regularization term includes a desired proportion of the selected key signal.

[0022] In a third aspect, the present disclosure provides a processor-readable storage medium, wherein the processor-readable storage medium stores a computer program, wherein the computer program is used to enable the processor to execute the method described in the first aspect above.

[0023] In a fourth aspect, the present disclosure provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0024] The present disclosure provides an unsupervised continuous emotional EEG analysis method and device based on deep reinforcement learning, wherein the analysis method includes a prototype learning stage based on a heuristic algorithm and a reinforcement learning stage based on a distribution-prototype reward function. In the prototype learning stage: the sample features of the time series signals corresponding to each sample object are obtained, and the sample features are clustered based on the K-Means algorithm to obtain the global distribution information of the emotional information in the time series signal. In the reinforcement learning stage: the distribution-prototype reward function is determined according to the global distribution information, and the Actor-Critic framework is used to use the proximal policy optimization algorithm PPO, so that the Actor performs the current action for a given time series signal based on the current strategy, and the Critic determines the current score of the current action based on the distribution-prototype reward of the environmental feedback, and the Actor performs strategy learning based on the current score until the preset termination iteration condition is reached, and a model for determining the key signal in the target time series signal according to the target time series signal is obtained, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function. In this embodiment, through the two-stage analysis, the accuracy and reliability of the finally obtained model can be improved, so as to effectively and reliably identify emotions based on the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0026] Figure 1 A flowchart of an unsupervised continuous emotional EEG analysis method based on deep reinforcement learning provided by the present disclosure; Figure 2 A schematic diagram of the principle of the unsupervised continuous emotional EEG analysis method based on deep reinforcement learning provided by the present disclosure; Figure 3 A schematic diagram of comparison results provided for one embodiment of the present disclosure; Figure 4 A schematic diagram of comparison results provided by another embodiment of the present disclosure; Figure 5 A schematic diagram of comparison results provided by another embodiment of the present disclosure; Figure 6 A schematic diagram of the structure of an unsupervised continuous emotional EEG analysis device based on deep reinforcement learning provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0027] The above drawings show clear embodiments of the present disclosure, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0028] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0029] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of the present disclosure are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0030] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0031] The term "plurality" in the embodiments of the present disclosure refers to two or more than two, and other quantifiers are similar thereto.

[0032] The terms "first", "second", "third", etc. in the present disclosure are used to distinguish similar or similar objects or entities, and are not necessarily meant to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of the present disclosure.

[0033] The term "unit / module" as used in this disclosure refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or a combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0034] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0035] According to one aspect of the present disclosure, the present disclosure provides an unsupervised continuous emotional EEG analysis method based on deep reinforcement learning.

[0036] See also Figure 1 , Figure 1 The following is a flow chart of the unsupervised continuous emotional EEG analysis method based on deep reinforcement learning (hereinafter referred to as the analysis method) provided by the present disclosure. Figure 1 As shown, the method includes two stages: a prototype learning stage based on a heuristic algorithm and a reinforcement learning stage based on a distribution-prototype reward function. Specifically, the method includes: S101: In the prototype learning stage: obtain the sample features of the time series signal corresponding to each sample object, and cluster the sample features based on the K-Means algorithm to obtain the global distribution information of the emotional information in the time series signal.

[0037] The execution subject of this embodiment may be an unsupervised continuous emotion EEG analysis device based on deep reinforcement learning (hereinafter referred to as the analysis device). The analysis device may be a server, a processor, a chip, etc., which is not limited in this embodiment. The analysis device is introduced elsewhere in this specification.

[0038] In this step, the analysis device uses the K-Means algorithm as a heuristic method to obtain a global perspective of the distribution of emotional information in sample features, that is, to obtain global distribution information of emotional information in time series signals.

[0039] Combination Figure 2 It can be seen that the analysis method disclosed in the present invention includes a prototype learning stage framed by a dotted line on the left and a reinforcement learning stage framed by a dotted line on the right. In this specification, "left" and "right" are based on the viewing angle when browsing the attached drawings.

[0040] like Figure 2 As shown in Figure 1, in the prototype learning stage, the sample features are mainly clustered to obtain an emotional space that includes different categories of emotions. The emotional space represents global distribution information.

[0041] In some embodiments, “clustering the sample features based on the K-Means algorithm to obtain global distribution information of the emotion information in the time series signal” in S101 may include the following steps: Step 11: Input the sample features into the K-Means algorithm to obtain an emotion prototype set, wherein the emotion prototype set represents multiple emotion categories and a prototype feature vector of each emotion category, and the prototype feature vector represents a cluster center.

[0042] Exemplarily, the analysis device inputs the sample features into the K-Means algorithm to obtain the emotion prototype set . Among them, each represents a cluster center corresponding to the emotion category, Represents the number of categories.

[0043] That is to say, by clustering the sample features, the emotion prototype (i.e., cluster center) that serves as the representative point of the emotion space can be determined to capture the global distribution of emotions, that is, to obtain the global distribution information of the emotion information.

[0044] The prototype feature vector of a certain emotion category can be calculated by taking the average of all sample features belonging to this category. For example, the prototype feature vector It can be expressed by formula 1: It can be expressed by formula 2: in, Indicates that it belongs to the emotional category The sample set, is the number of samples in this category.

[0045] Step 12: Calculate the intra-cluster variance for each emotion category.

[0046] For example, to quantify the variance within each cluster, the analysis device may use the average of the sum of squared errors within the cluster, which reflects the distribution of sample features within the cluster. The variance of the distribution within the cluster indicates the individual differences of each emotion. For example, the variance within the cluster is It can be expressed by formula 3: in, represents the number of sample features in the cluster, represents a single sample feature, Indicates the corresponding cluster center.

[0047] Step 13: Determine the global distribution information of the emotion information in the time series signal based on the prototype feature vector and the intra-cluster variance of each emotion category.

[0048] In some embodiments, in order to better represent the emotional distribution in the sample features, the prototype learning process uses the cluster center as the average of the data distribution and uses the average of the sum of squared errors within the cluster to describe the variance. This combination of prototype learning and K-Means clustering provides a solid foundation for the reinforcement learning process, enabling the model to effectively understand the complex emotional information present in the learning sample features.

[0049] Combined with the above analysis, it can be seen that in this embodiment, not only the intrinsic data distribution characteristics of local emotions should be considered, but also the broader context provided by the global emotional space, which encapsulates the distribution of various emotion categories. In order to more effectively represent the emotional distribution, by introducing prototype learning, the analysis device can model each emotion category as a prototype, thereby capturing the globally distributed emotional information with a better representation. Prototype learning enables the analysis device to integrate global emotional information into the subsequent reward structure, ensuring that the reinforcement learning process is based on a comprehensive understanding of the emotional landscape represented in the sample features.

[0050] S102: In the reinforcement learning stage: determine the distribution-prototype reward function according to the global distribution information, and use the proximal policy optimization algorithm PPO in the Actor-Critic framework, so that the Actor performs the current action based on the current strategy for a given timing signal, and the Critic determines the current score of the current action based on the distribution-prototype reward of the environment feedback, and the Actor performs strategy learning based on the current score until the preset termination iteration condition is reached, and a model for determining the key signal in the target timing signal according to the target timing signal is obtained, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

[0051] Exemplary, combined Figure 2 From the above analysis, we can see that: In the prototype learning stage, the analysis device conducts a global search for sample features based on the heuristic algorithm K-Means, and uses the searched cluster center, i.e., the mean, as the learned emotion prototype, and calculates the intra-cluster variance of each cluster center. The mean and variance can well describe the emotion prototype and characterize individual differences, and pass such global distribution information to the reinforcement learning stage.

[0052] In the reinforcement learning stage, we carefully designed the distribution-prototype reward function using the global distribution information obtained by the heuristic algorithm, and adopted the policy-based Actor-Critic framework using the Proximal Policy Optimization (PPO) algorithm for policy learning. The actor interacts with the emotional space in a trial-and-error manner, and learns the interactive relationship with the emotional prototype in the local video-level sample feature distribution. At the end of each round of interaction, the Critic scores the actor's decision in this round to find the improvement point of the strategy. Through continuous iteration, the actor finally learns the optimal strategy for identifying key signals.

[0053] like Figure 2 As shown in the figure, Actor includes a long short-term memory network (LSTM) and a fully connected layer (FC) to predict the action probability through LSTM and FC and perform the corresponding action.

[0054] In some embodiments, the distribution-prototype reward function includes a Center reward function and an Inter-Intra reward function.

[0055] Among them, the Center reward function is used to characterize the intensity of the emotional information contained in a given time series signal; the Inter-Intra reward function is used to characterize the relative confidence that a given time series signal belongs to a specific emotion category calculated in the prototype learning stage.

[0056] For example, a specific emotion category is the emotion category of the prototype feature to which a given time series signal is closest.

[0057] Combined with the above analysis, it can be seen that the analysis device obtains the global distribution information of emotional information in an unsupervised manner through the prototype learning stage, and adds this global distribution information to the reward function of the reinforcement learning stage. Relatively speaking, the cluster centers obtained by heuristic search represent the prototypes of each emotional category. These prototypes effectively capture the distribution of sample features in the entire emotional category, while other sample features belonging to the same category are clustered around these prototypes.

[0058] From a probabilistic perspective, the emotion prototype can be understood as the mean of sample features, and the variance of this distribution stems from the inherent variability of emotions and the non-stationarity of time series characteristics. The analysis device uses the mean and variance of each emotion cluster to reflect the individual differences in the distribution of sample features of each emotion category.

[0059] In this embodiment, the goal of the actor is to maximize the expected reward by selecting key signals related to emotions. To achieve this goal, the analysis device proposes two reward functions based on the global distribution information obtained from the prototype learning phase: the Center reward function and the Inter-Intra reward function.

[0060] Among them, the Center reward function can be used to measure the strength of the emotional information contained in a given time series signal. The given time series signal is any time series signal among the time series signals corresponding to each sample object. In other words, the Center reward function can be used to measure the strength of the emotional information contained in the sample features of any time series signal.

[0061] In some embodiments, the analysis device may evaluate the intensity by calculating the Euclidean distance between the sample feature of any time series signal and its cluster center. It can be expressed by formula 4: The analysis device can calculate the sample characteristics of any time series signal through equation 5: With its cluster center The Euclidean distance between , Formula 5: In this embodiment, the analysis device uses the distance measurement between the sample features of the given time series signal and the cluster center in the emotional space for two purposes. Purpose 1. The proximity between the sample features of the given time series signal and the cluster center reflects the intensity of the emotions they represent. 2. It helps to alleviate the influence of interference information caused by the non-stationary characteristics of the given time series signal.

[0062] The Inter-Intra reward function represents the relative confidence that the sample feature of a given time series signal belongs to a specific emotion category. The analysis device can use inverse variance weighting to calculate the distance between the sample feature of a given time series signal and the cluster center of other categories. This weighting method has a weighted average with the minimum variance, and the analysis device can use it to estimate the confidence that the sample feature of a given time series signal is far away from the cluster center of other emotion categories.

[0063] In some embodiments, the Inter-Intra Reward Function It can be expressed by formula 6: in, It can be expressed by formula 7: It can be expressed by formula 8: in, Represents the distance between the sample feature of a given time series signal and the cluster center of the category to which it belongs; Represents the weighted distance between the sample features of a given time series signal and the clustering centers of other categories, Represents variance.

[0064] In some embodiments, the probability ratio between the strategy of the current iteration and the strategy of the previous iteration is less than a preset probability ratio threshold, so that the trajectory is reused multiple times.

[0065] Combined with the above analysis, it can be seen that in this embodiment, the analysis device uses PPO to train a model for determining the key signal in the target time series signal according to the target time series signal. This model can be called an emotion-agent model. In the whole process, the key signals related to emotions are searched at the trial level, and the Emotion-Agent model learns the strategy of identifying key signals in a trial-and-error manner in a discrete action space.

[0066] To stabilize the training process, the analysis device uses PPO-Clip to limit the ratio between the new and old policies, reducing oscillations and accelerating convergence. PPO is a policy-based Actor-Critic method. To improve sample efficiency during training, importance sampling is introduced, allowing the model to reuse trajectories multiple times.

[0067] Among them, the probability ratio between the strategy of the current iteration and the strategy of the previous iteration is It can be expressed by formula 9: in, Indicates that the strategy of the current iteration is in state Next select action The probability of Indicates that the strategy of the previous iteration is in state Next select action The probability of represents the time step, Represents the parameters of the strategy for the current iteration, Represents the parameters of the strategy of the previous iteration.

[0068] In some embodiments, the analysis device can evaluate the pros and cons of the current strategy based on the temporal difference error, and determine the advantage function based on the generalized advantage estimate (GAE) and the temporal difference error, so that the Actor updates the current strategy based on the advantage function to perform strategy learning.

[0069] Exemplarily, in order to better estimate the cumulative benefit, the analysis device uses a generalized advantage estimate, which provides a more accurate advantage function. The generalized advantage estimate can be expressed by Formula 10, Formula 10: in, represents the discount factor, The generalized advantage estimator hyperparameter that represents the trade-off between controlling bias and variance, Represents the time step The advantage function estimate of .

[0070] In some embodiments, the update range of the Actor's strategy probability in each iteration is determined based on a preset upper limit and lower limit of the ratio of the new and old strategies, and is represented by an objective function with clipping.

[0071] Exemplarily, in order to further stabilize the training process, the analysis device applies PPO-Clip, which limits the change between the new and old strategies. The objective function with clipping can be expressed by Equation 11: in, represents the loss function of the strategy, Represents the time step The expected value of Represents the time step The advantage function estimate of Represents the probability ratio The clipping operation ensures that its value is within and between the old and new strategies, Represents a hyperparameter that is used to control the range of change between the new and old strategies (i.e., the upper and lower limits of the ratio of the new and old strategies), thereby stabilizing the training process.

[0072] In some embodiments, policy learning is constrained based on a regularization term that includes a desired proportion of a critical signal.

[0073] For example, in order to prevent the Actor from selecting too many key signals, the analysis device introduces a regularization term to limit the action probability of the learned policy function. The regularization term can be expressed by Formula 12: in, represents the loss function of the regularization term; is a scalar representing the desired proportion of the selected key signal; Represents the time step The action probability of Represents the total number of time steps.

[0074] On this basis, the analysis device can use the Adam optimizer to update the parameters of the policy function For example, to update the parameter It can be expressed by formula 13: in, Represents the preset learning rate, which is used to control the step size of parameter update; It is a preset regularization coefficient used to balance the strategy loss and regularization term; Represents the loss function of the policy.

[0075] In some embodiments, the Critic is responsible for estimating during the decision-making process as a prediction of the actual discounted cumulative reward throughout the process. This estimate guides the Actor to converge to the optimal policy. The Critic's error function (loss function) is the mean square error (MSE) between the estimated value and the actual discounted cumulative reward, which can be expressed by Equation 14: in, Represents the time step The estimated value of the value function, Represents the time step Discount accumulation rewards.

[0076] Combining the above analysis and Figure 2 It can be seen that during the training process, the Emotion-Agent model can be divided into two parts: Actor and Critic. The Actor constantly interacts with the emotion space during the decision-making process, and iteratively searches for the optimal strategy based on the reward mechanism fed back by the emotion space. The Critic guides this exploration process by estimating the cumulative expected return of each state, thereby assisting the Actor in finding the optimal strategy.

[0077] In some embodiments, the analysis method provided by the present disclosure further includes a Markov decision process (MDP). MDP can be understood as the analysis device identifying the most emotion-related segments (ie, key signals) from continuous time series signals and modeling them as a sequential decision process.

[0078] The analysis device is defined as a tuple based on MDP ,in, represents the state space, represents the action space, Corresponding to the possible actions that the agent may take. Transition probability function Describes the probability of moving from one state to another given a specific action. Assign numerical rewards based on state-action-state transitions to provide feedback for the agent's decisions.

[0079] In some embodiments, the time series signal may be an electroencephalogram (EEG) signal, an audio signal, or a video signal, etc., which is not limited in this embodiment. If the time series signal is an EEG signal, the sample feature may be a differential entropy (DE) feature.

[0080] It is understandable that the above example mainly describes the analysis method from the training stage. After the Emotion-Agent model is obtained based on the above analysis method, the time series signal to be identified can be input into the Emotion-Agent model to obtain the key signal, thereby identifying the emotion corresponding to the time series signal to be identified.

[0081] In order to verify the technical effect of the technical solution provided by the present invention, experiments can be conducted on the SEED database. The analysis method disclosed in the present invention is to use the entire video-level EEG signal of each subject as training data, encourage the agent to extract the key EEG segments related to emotions in an unsupervised manner through a reward function, use KNN as a test classifier, and input the extracted key EEG segments into the classifier for classification. In the experimental part, accuracy can be used as an evaluation indicator, and a classification confusion matrix can be drawn to analyze the analysis method. Among them, the calculation of Accuracy can be expressed by Formula 15, Formula 15: in, and represents the number of correctly classified samples, and Indicates the number of misclassified samples.

[0082] according to Figure 3The results shown show that in the cross-subject EEG emotion task on the SEED database, compared with the traditional unsupervised learning method, the Emotion-Agent model proposed in the present disclosure has achieved the highest performance in classification accuracy on the three-category task. In the three-category task, the Emotion-Agent model is 9.32% higher than TAS-Net and 20.27% higher than the unsupervised method EEGFuseNet. To a certain extent, the Emotion-Agent model proposed in the present disclosure can extract key EEG segments that are more relevant to emotions in an unsupervised manner, thereby improving the classification accuracy on subsequent emotion recognition tasks. The experimental results also prove that the Emotion-Agent model proposed in the present disclosure has a higher emotion recognition accuracy on the cross-subject EEG emotion recognition task.

[0083] In order to further verify the effectiveness of the analysis method proposed in this disclosure, SVM, KNN and MLP were used as classifiers for emotion recognition without (w / o) and with (w) the proposed Emotion-Agent, and a cross-subject experiment was conducted on the SEED dataset. Accuracy and F1-Scores are used as evaluation indicators here. The specific calculation of F1-Scores can be expressed by Formula 16: in, Indicates the accuracy, Represents the recall rate.

[0084] like Figure 4 The results shown show that when SVM is used as a classifier, the accuracy of the analysis method proposed in the present invention is improved by 21.93% and the F1-Scores are improved by 27.1% compared with not using it; when KNN is used as a classifier, the accuracy of the analysis method proposed in the present invention is improved by 19.53% and the F1-Scores are improved by 26.07% compared with not using it; when SVM is used as a classifier, the accuracy of the analysis method proposed in the present invention is improved by 12.02% and the F1-Scores are improved by 13.21% compared with not using it.

[0085] like Figure 5 As shown, Figure 5 The classification results of the confusion matrix obtained by direct classification and classification after sampling with the Emotion-Agent model are compared. The SEED DE features are used as input and the classification confusion matrix is ​​obtained by using SVM as the classifier. Figure 5As shown in the figure, the experimental results show that compared with direct classification, the Emotion-Agent model achieves higher classification accuracy in all three emotion categories.

[0086] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as EEG signals) involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0087] According to an embodiment of the present disclosure, the present disclosure also provides an unsupervised continuous emotional EEG analysis device based on deep reinforcement learning.

[0088] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of an unsupervised continuous emotion EEG analysis device based on deep reinforcement learning provided by an embodiment of the present disclosure. Figure 6 As shown, the analysis device 600 includes: The prototype learning unit 601 is used to obtain the sample features of the time series signal corresponding to each sample object in the prototype learning stage, and perform clustering processing on the sample features based on the K-Means algorithm to obtain the global distribution information of the emotion information in the time series signal; The reinforcement learning unit 602 is used to determine the distribution-prototype reward function according to the global distribution information in the reinforcement learning stage based on the distribution-prototype reward function, and adopt the Actor-Critic framework using the proximal policy optimization algorithm PPO, so that the Actor performs the current action for a given timing signal based on the current strategy, the Critic determines the current score of the current action based on the distribution-prototype reward of the environment feedback, and the Actor performs strategy learning based on the current score until the preset termination iteration condition is reached, and a model for determining the key signal in the target timing signal according to the target timing signal is obtained, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

[0089] In some embodiments, the prototype learning unit 601 is specifically used for: Inputting the sample features into the K-Means algorithm to obtain an emotion prototype set, wherein the emotion prototype set represents multiple emotion categories and a prototype feature vector of each emotion category, and the prototype feature vector represents a cluster center; Calculate the prototype feature vector and within-cluster variance for each emotion category; The global distribution information of the emotion information in the time series signal is determined based on the prototype feature vector and the intra-cluster variance of each emotion category.

[0090] In some embodiments, the distribution-prototype reward function includes a Center reward function and an Inter-Intra reward function; Among them, the Center reward function is used to characterize the intensity of the emotional information contained in the given time series signal; the Inter-Intra reward function is used to characterize the relative confidence that the given time series signal belongs to a specific emotion category calculated in the prototype learning stage.

[0091] In some embodiments, the probability ratio between the strategy of the current iteration and the strategy of the previous iteration is less than a preset probability ratio threshold, so that the trajectory is reused multiple times.

[0092] In some embodiments, the reinforcement learning unit 602 is further configured to: Evaluating the pros and cons of the current strategy based on the temporal difference error; An advantage function is determined based on the generalized advantage estimate and the temporal difference error, so that the Actor updates the current policy based on the advantage function to perform policy learning.

[0093] In some embodiments, the update range of the Actor's strategy probability in each iteration is determined based on a preset upper limit and a lower limit of the ratio of the new strategy to the old strategy, and is represented by an objective function with clipping.

[0094] In some embodiments, the policy learning is constrained based on a regularization term, wherein the regularization term includes a desired proportion of the selected key signal.

[0095] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0096] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.

[0097] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0098] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0100] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the analysis method. For example, in some embodiments, the analysis method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the analysis method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the analysis method in any other appropriate manner (e.g., by means of firmware).

[0101] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0103] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0104] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0105] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0106] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0107] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program codes.

[0108] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer executable instructions. These computer executable instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These processor executable instructions may also be stored in a processor readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the processor readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These processor-executable instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. An unsupervised continuous emotional EEG analysis method based on deep reinforcement learning, characterized in that: The method includes a prototype learning phase based on a heuristic algorithm and a reinforcement learning phase based on a distribution-prototype reward function; In the prototype learning stage: obtaining sample features of the time series signal corresponding to each sample object, and clustering the sample features based on the K-Means algorithm to obtain global distribution information of the emotional information in the time series signal; In the reinforcement learning stage: a distribution-prototype reward function is determined according to the global distribution information, and the Actor-Critic framework is used to use the proximal policy optimization algorithm PPO, so that the Actor performs a current action based on the current strategy for a given timing signal, the Critic determines a current score of the current action based on the distribution-prototype reward of environmental feedback, and the Actor performs strategy learning based on the current score until a preset termination iteration condition is reached, thereby obtaining a model for determining a key signal in the target timing signal according to the target timing signal, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

2. The method according to claim 1, characterized in that The clustering process of the sample features based on the K-Means algorithm to obtain the global distribution information of the emotion information in the time series signal includes: Inputting the sample features into the K-Means algorithm to obtain an emotion prototype set, wherein the emotion prototype set represents multiple emotion categories and a prototype feature vector of each emotion category, and the prototype feature vector represents a cluster center; Calculate the within-cluster variance for each emotion category; The global distribution information of the emotion information in the time series signal is determined based on the prototype feature vector and the intra-cluster variance of each emotion category.

3. The method according to claim 1, characterized in that The distribution-prototype reward function includes a Center reward function and an Inter-Intra reward function; Among them, the Center reward function is used to characterize the intensity of the emotional information contained in the given time series signal; the Inter-Intra reward function is used to characterize the relative confidence that the given time series signal belongs to a specific emotion category calculated in the prototype learning stage.

4. The method according to claim 1, characterized in that: The probability ratio between the strategy of the current iteration and the strategy of the previous iteration is less than a preset probability ratio threshold, so that the trajectory is reused multiple times.

5. The method according to claim 1, characterized in that The method further comprises: Evaluating the pros and cons of the current strategy based on the temporal difference error; An advantage function is determined based on the generalized advantage estimate and the temporal difference error, so that the Actor updates the current policy based on the advantage function to perform policy learning.

6. The method according to claim 1, characterized in that The update range of the Actor's strategy probability in each iteration is determined based on the upper and lower limits of the preset new and old strategy ratios, and is represented by an objective function with clipping.

7. The method according to claim 1, characterized in that The policy learning is constrained based on a regularization term that includes a desired proportion of the selected key signal.

8. An unsupervised continuous emotional EEG analysis device based on deep reinforcement learning, characterized in that: include: A prototype learning unit, used to obtain sample features of the time series signal corresponding to each sample object in the prototype learning stage, and cluster the sample features based on the K-Means algorithm to obtain global distribution information of the emotional information in the time series signal; A reinforcement learning unit is used to determine the distribution-prototype reward function according to the global distribution information in the reinforcement learning stage based on the distribution-prototype reward function, and adopt the Actor-Critic framework to use the proximal policy optimization algorithm PPO, so that the Actor performs a current action for a given timing signal based on the current strategy, the Critic determines the current score of the current action based on the distribution-prototype reward of environmental feedback, and the Actor performs strategy learning based on the current score until a preset termination iteration condition is reached, thereby obtaining a model for determining the key signal in the target timing signal according to the target timing signal, wherein the distribution-prototype reward is calculated based on the distribution-prototype reward function.

9. A processor-readable storage medium, characterized in that: The processor-readable storage medium stores a computer program, and the computer program is used to cause the processor to execute the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images