Emotion recognition method and device, nonvolatile storage medium, and electronic device

Abstract

The application discloses an emotion recognition method and device, a nonvolatile storage medium and an electronic device. The method comprises the following steps: acquiring original data of a plurality of participating devices respectively, and extracting feature data of the original data; training a local model of the participating device according to an initialization parameter and the feature data, obtaining a target parameter, and uploading the target parameter to a central model; determining a first loss value according to the target parameter, and determining a second loss value according to the first loss value and a weight coefficient of the first loss value; in the case that a minimum value of the second loss value at a current time is less than or equal to an average value of the first loss value at a target time, determining emotion recognition results corresponding to different activation functions according to the different activation functions. The application solves the technical problem of low aggregation efficiency caused by the fact that the central server and the participating devices based on federated learning of emotion recognition use the same aggregation model parameters and the same loss function for aggregation.

Description

Technical Field

[0001] This application relates to the field of intelligent service technology, and more specifically, to an emotion recognition method and apparatus, a non-volatile storage medium, and an electronic device. Background Technology

[0002] Whether it's an emotion recognition system or an intelligent product recommendation system, timeliness is the decisive factor in measuring system effectiveness. Real-time identification of consumer emotions and rapid adjustment of recommendation strategies based on these emotions can effectively improve shopping efficiency and enhance the shopping experience. User data is distributed across different locations, and transmitting and integrating this data can easily lead to data security issues. Federated learning, as a distributed computing paradigm, can effectively improve model learning speed while addressing data security concerns.

[0003] However, federated learning in related technologies uses common model parameters and a common loss function for aggregation, which is not suitable for personalized systems. This can lead to the loss of dynamic information, thereby affecting system performance.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an emotion recognition method and apparatus, a non-volatile storage medium, and an electronic device to at least solve the technical problem of low aggregation efficiency caused by the central server and participating devices using the same aggregation model parameters and the same loss function for aggregation in emotion recognition-based federated learning.

[0006] According to one aspect of the embodiments of this application, an emotion recognition method is provided, comprising: acquiring raw data from multiple participating devices and extracting feature data from the raw data, wherein the participating devices are participants in a federated learning based on emotion recognition; training local models of the participating devices according to initialization parameters and feature data to obtain target parameters, and uploading the target parameters to a central model, wherein the initialization parameters and the central model are parameters and models received locally by the participating devices from a central server, respectively, and the central server is a server for federated learning based on emotion recognition; determining a first loss value according to the target parameters, and determining a second loss value according to the first loss value and the weight coefficient of the first loss value, wherein the first loss value is the loss value of the target participating device in a target time period, and the second loss value is the loss value of the central server in the target time period, and the first loss value and the second loss value are used to characterize the accuracy of the model training result; and determining the emotion recognition result corresponding to different activation functions according to different activation functions when the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time, wherein the target time is the previous time adjacent to the current time.

[0007] Optionally, before determining the first loss value based on the target parameters and the second loss value based on the first loss value and its weighting coefficient, the method further includes: determining the computing resource utilization rate of each participating device based on the CPU utilization rate, memory utilization rate, and hard disk utilization rate of each participating device; determining the data consumption of each participating device when running its local model; determining the participating device as the first participating device when the computing resource utilization rate is greater than the data consumption rate; and determining the first participating device as the target participating device when the network bandwidth between the central server and the first participating device is the target bandwidth.

[0008] Optionally, extracting feature data from the original data includes: preprocessing the physiological data in the original data to obtain target physiological data, wherein the preprocessing includes at least: data cleaning and normalization; obtaining a first feature of the target physiological data, wherein the first feature includes at least one of the following: statistical features, frequency domain features, and nonlinear entropy domain features; inputting the target physiological data into a first deep learning model to obtain a second feature of the target physiological data, wherein the second feature is a multi-dimensional feature; fusing the first feature and the second feature to obtain a target matrix, wherein the target matrix is ​​a three-dimensional feature matrix; inputting the target matrix into the first deep learning model to obtain time-frequency features, and determining the time-frequency features as feature data.

[0009] Optionally, the emotion recognition result includes: the target object's emotion, interests, and intentions, wherein the emotion includes at least one of the following: pleasure, positivity, negativity, and anger; the interests are used to indicate the type of goods the target object intends to buy, wherein the interests include at least one of the following: daily necessities and decorations; the intentions are used to indicate the type of purchasing behavior of the target object, wherein the intentions include at least one of the following: regular purchases, sports purchases, simple purchases, surfing purchases, access purchases, and bargaining purchases.

[0010] Optionally, the emotion recognition result corresponding to different activation functions is determined according to different activation functions, including: the central model determines the emotion in the emotion recognition result according to the first activation function; the central model determines the interest point in the emotion recognition result according to the second activation function; and the central model determines the intention in the emotion recognition result according to the third activation function.

[0011] Optionally, after determining the sentiment recognition results corresponding to different activation functions, the method further includes: obtaining the sentiment vector in the sentiment recognition results and the product description vector of the product to be recommended, wherein the sentiment vector includes emotion, interest points and willingness; determining the correlation coefficient between the sentiment recognition results and the product to be recommended based at least on the sentiment vector and the product description vector; determining the target recommended product based on the correlation coefficient, and displaying the target recommended product.

[0012] Optionally, the above method further includes: displaying the emotion of the target object in the emotion recognition result; inputting the emotion adjusted by the target object into the central model and training the central model; and determining the target emotion recognition result based on the trained central model.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the storage medium including a stored program, wherein the program controls the device where the storage medium is located to execute the above-mentioned emotion recognition method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes the above-described emotion recognition method during runtime.

[0015] In this embodiment, raw data from multiple participating devices is acquired, and feature data is extracted from the raw data. The participating devices are participants in a federated learning model based on emotion recognition. Local models on the participating devices are trained using initialization parameters and feature data to obtain target parameters, which are then uploaded to a central model. The initialization parameters and the central model are the parameters and model received locally by the participating devices from the central server, which is the server for the federated learning model based on emotion recognition. A first loss value is determined based on the target parameters, and a second loss value is determined based on the first loss value and its weight coefficients. The first loss value is the loss value of the target participating device within a target time period, and the second loss value is the loss value of the central server within the target time period. The loss values, first loss value, and second loss value are used to characterize the accuracy of the model training results. When the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time, the emotion recognition result corresponding to different activation functions is determined according to different activation functions. The target time is the previous time adjacent to the current time. By determining the loss value of the central server based on the loss values ​​of the participating devices, the purpose of optimizing federated learning based on emotion recognition is achieved. This improves the aggregation efficiency of federated learning based on emotion recognition and solves the problem of low aggregation efficiency caused by using the same aggregation model parameters and the same loss function for aggregation between the central server and participating devices in federated learning based on emotion recognition. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0017] Figure 1This is a flowchart of an emotion recognition method according to an embodiment of this application;

[0018] Figure 2-1 This is a flowchart illustrating a product recommendation method based on the present emotion recognition method according to an embodiment of this application;

[0019] Figure 2-2 This is a flowchart illustrating another product recommendation method based on the present emotion recognition method according to an embodiment of this application;

[0020] Figure 2-3 This is a flowchart illustrating another product recommendation method based on the present emotion recognition method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an interactive data acquisition method according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of a raw data processing method according to an embodiment of this application;

[0023] Figure 5 This is a flowchart of a GlobalTrans-DNN algorithm according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of an optimized federated learning model according to an embodiment of this application;

[0025] Figure 7 This is a flowchart of a federated learning method according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of an emotion recognition result according to an embodiment of this application;

[0027] Figure 9 This is a structural diagram of an emotion recognition device according to an embodiment of this application;

[0028] Figure 10 This is a structural diagram of another emotion recognition device according to an embodiment of this application. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] According to an embodiment of this application, an embodiment of an emotion recognition method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] Emotions have always been a powerful weapon for marketers, with consumers' inner experiences such as joy and satisfaction playing a decisive role in purchasing decisions. Current recommendation algorithms, often based on historical data, assume that consumer preferences are static. However, people's preferences change in real time based on their emotions; therefore, recommendation systems should consider dynamic factors such as emotions. Emotions are expressed through physiological signals and facial expressions, and the difficulty in faking these signals allows for more accurate emotion identification. Intelligent recommendations based on consumers' real-time emotions can achieve more precise recommendations. Therefore, advanced systems are needed to extract emotions from massive amounts of data and make real-time recommendations based on multimodal data, including emotion, user data, and interaction data.

[0033] Whether it's emotion recognition or intelligent recommendation systems, timeliness is a decisive factor in measuring system effectiveness. Real-time identification of consumer emotions and rapid adjustment of recommendation strategies based on these emotions will improve shopping efficiency and enhance the shopping experience. User data is distributed across different locations, and transmitting and integrating it can easily lead to data security issues. Federated learning, as a distributed computing paradigm, can effectively improve model learning speed while addressing data security concerns.

[0034] However, most federated learning techniques use common model parameters and loss functions for aggregation, which is unsuitable for personalized systems, leading to the loss of dynamic information and impacting system performance. Furthermore, in the participant selection phase of federated learning, a slow participant will affect the overall runtime.

[0035] To address the aforementioned problems in related technologies, this application, based on time-domain optimized federated learning, processes multimodal data generated by users' online shopping in real time, including physiological data, interaction data, historical data, and hotspot data. This allows for the prediction of users' shopping sentiment, shopping type (intention), and points of interest, among other emotional information. By combining these factors, a sentiment-based personalized real-time recommendation system is achieved. The entire recommendation process considers emotional factors and the timeliness of recommendations, introducing federated learning and optimizing the model and computational framework to achieve rapid data processing and accurate identification of user states; ultimately, more efficient personalized recommendations are realized. This proposes a new model for combining federated learning and sentiment-based recommendation systems.

[0036] Figure 1 This is a flowchart of an emotion recognition method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0037] Step S102: Obtain the raw data of multiple participating devices and extract the feature data of the raw data. The participating devices are the participants in the federated learning based on emotion recognition.

[0038] According to an optional embodiment of this application, participants (participating devices) include, but are not limited to, mobile terminals and personal computers.

[0039] As an optional embodiment of this application, the raw data includes at least physiological data, interaction data, historical data, and hotspot data, wherein the physiological data includes at least electroencephalogram (EEG) signals and electrocardiogram (ECG) signals. When a user (target object) browses the shopping system, they will generate slight emotional reactions and interaction data when faced with different types of products. The user's physiological data can be collected through various smart wearable devices, the Emotiv EPOC Neuroheadset can be used to collect and record the user's EEG signals, and the Shimmer 2R4 platform can be used to collect and record the user's ECG signals. Interaction data consists of log data generated from the user's interaction with the system, including browser history, which can serve as a data source for determining the user's consumption category, user preferences, etc. Log data includes the categories, quantities, and frequencies of products clicked by the user, favorited store items, and shopping cart records. Historical data includes data on products purchased by the user and products recommended by the system. Hotspot data consists of data on popular products on the internet and can also serve as a database of user emotions.

[0040] According to another optional embodiment of this application, the feature extraction process for log data is as follows: A keyword set is set, keyword retrieval technology is used to extract sentences containing the keywords, and jieba word segmentation technology is used to split the entire sentence into words to obtain corresponding phrases. Word2Vector technology is applied to the word vectors to transform them from phrases into computer-understandable word vectors, facilitating network processing. Similarly, feature fusion is performed on the word vectors to form multi-dimensional features, which are then input into the Transformer attention mechanism to obtain the features of the log data, referred to as time-frequency spatial features.

[0041] In some optional embodiments of this application, the feature extraction process of historical data is as follows: extract the descriptive tags related to the user from the historical shopping data to obtain information such as the user's shopping type and user feature representation, and then use OneHot encoding to vectorize the tags to obtain the features of the historical data, which are called comprehensive features.

[0042] As an optional embodiment of this application, the feature extraction process of hot data is as follows: hot information from various online platforms is obtained using big data technology, popular keywords and phrases are extracted, and they are matched one-to-one with products. Selected products are vectorized, i.e., product vectors are described [tableware, convenience, travel, internet celebrity, etc.]. The vectors are quantized, and multi-dimensional features of the corresponding items are extracted to obtain the features of hot data, which are called sentiment features.

[0043] Step S104: Train the local model of the participating device according to the initialization parameters and feature data to obtain the target parameters, and upload the target parameters to the central model. The initialization parameters and the central model are the parameters and model that the participating device receives locally from the central server. The central server is a server for federated learning based on emotion recognition.

[0044] According to another optional embodiment of this application, the participating device receives the initialization model and initialization parameters published by the central server, downloads the initialization model to its local machine, trains the local model using local feature data, and obtains the trained parameters, i.e., the target parameters in step S104. After obtaining the target parameters, the target parameters are uploaded to the central model of the central server.

[0045] Step S106: Determine the first loss value based on the target parameters, and determine the second loss value based on the first loss value and the weight coefficient of the first loss value. The first loss value is the loss value of the target participating device in the target time period, and the second loss value is the loss value of the central server in the target time period. The first loss value and the second loss value are used to characterize the accuracy of the model training results.

[0046] In some optional embodiments of this application, the central server sets up the FL-GlobalTrans-DNN model and initializes the model parameters to ω(0). After receiving the global model state, the participants perform self-training, and the participants set the loss function:

[0047]

[0048] Where x kl This is the training data for the kth participant, y kl It's a real label.

[0049] The central server determines the loss function based on the weighting coefficients of the participants:

[0050]

[0051] D k is the weight coefficient for the k-th participant.

[0052] The above steps achieve the technical effect of accelerating the convergence of the central model.

[0053] Step S108: If the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time, determine the emotion recognition result corresponding to different activation functions according to different activation functions, wherein the target time is the previous time adjacent to the current time.

[0054] As an optional embodiment of this application, the central server compares the loss values ​​of the participating devices and the central server to determine whether to stop training the central model. Training stops when the condition is met, indicating successful model training. F(ω) is the current loss value of the central server, and F′ is the loss value of the model. k (ω) represents the loss value of the k-th participant in the previous training round.

[0055] Based on the above steps, by determining the loss value of the central server according to the loss values ​​of the participating devices, the goal of optimizing federated learning based on emotion recognition is achieved. This results in improving the aggregation efficiency of federated learning based on emotion recognition, and solves the technical problem of low aggregation efficiency caused by the central server and participating devices using the same aggregation model parameters and the same loss function for aggregation.

[0056] According to an optional embodiment of this application, before determining the first loss value based on the target parameters and the second loss value based on the first loss value and its weighting coefficient, the following method can be used: determining the computing resource utilization rate of each participating device based on the CPU utilization rate, memory utilization rate, and hard disk utilization rate of each participating device; determining the data consumption of each participating device when running its local model; determining the participating device as the first participating device when the computing resource utilization rate is greater than the data consumption rate; and determining the first participating device as the target participating device when the network bandwidth between the central server and the first participating device is the target bandwidth.

[0057] According to another optional embodiment of this application, the central server sends a request to the sub-servers to obtain the computing resource utilization of each participating device using the formula γ = (CPU utilization + memory utilization + disk utilization / 3 * 100%). The computational load required for each participating device to run a local model is determined. If the resources provided by a participating device, i.e., its computing resource utilization, are less than or equal to the required computational load, that device is not selected as a participant. Among the participants whose computing resource utilization meets the requirements, the top k devices with the best network connections are selected as participants based on the network link conditions between the central server and them. The network link conditions between the central server and the participating devices can be obtained using the Wondershaper tool.

[0058] In some optional embodiments of this application, extracting feature data from the original data includes the following steps: preprocessing the physiological data in the original data to obtain target physiological data, wherein the preprocessing includes at least: data cleaning and normalization; obtaining a first feature of the target physiological data, wherein the first feature includes at least one of the following: statistical features, frequency domain features, and nonlinear entropy domain features; inputting the target physiological data into a first deep learning model to obtain a second feature of the target physiological data, wherein the second feature is a multi-dimensional feature; fusing the first feature and the second feature to obtain a target matrix, wherein the target matrix is ​​a three-dimensional feature matrix; inputting the target matrix into the first deep learning model to obtain time-frequency features, and determining the time-frequency features as feature data.

[0059] As an optional embodiment of this application, the physiological data includes electroencephalogram (EEG) signals and electrocardiogram (ECG) signals. The physiological signals have 14 channels, thus providing 14-dimensional data. Discrete wavelet transform is used to remove high- and low-frequency noise from the original physiological data. A Butterworth filter is used for noise reduction preprocessing of the physiological signals. Feature extraction is then performed on the preprocessed data. Since physiological signals are time-frequency signals, statistical features such as difference values ​​and medians are extracted first, followed by frequency domain features and nonlinear entropy domain features. Because manual feature extraction relies on experience and may result in missing features, a deep learning model (CNN) is used to automatically extract high-dimensional features.

[0060] Each feature derived above has a shape of 1×dim×C, where dim is the feature dimension and C is the number of channels. The hand-crafted features and deep features are then fused pairwise to form a global spatiotemporal 2×1×dim×C multimodal 3D feature matrix. The combined features are then input into a Transformer to extract features from the physiological data, termed important temporal-frequency features.

[0061] Through the above steps, after extracting the manual and deep features of the sequence, the two features are complemented and spliced ​​together in the spatiotemporal dimension to form a global feature. This feature has a higher dimension and more comprehensive time-frequency information. Considering the correlation of feature data extracted by each participating device and the data differences brought about by the individualization of each participating device, the Transformer mechanism is introduced, which uses autoencoder and decoder to better understand the relevant feature information between the beginning and end of the sequence.

[0062] In some optional embodiments of this application, the emotion recognition result includes: the target object's emotion, points of interest, and intention, wherein the emotion includes at least one of the following: pleasure, positivity, negativity, and anger; points of interest are used to indicate the type of goods the target object intends to purchase, wherein points of interest include at least one of the following: daily necessities and decorations; intention is used to indicate the type of purchasing behavior of the target object, wherein intention includes at least one of the following: regular purchase, sports purchase, simple purchase, surfing purchase, access purchase, and bargaining purchase.

[0063] According to an optional embodiment of this application, determining the emotion recognition result corresponding to different activation functions based on different activation functions includes: the central model determining the emotion in the emotion recognition result based on a first activation function; the central model determining the interest point in the emotion recognition result based on a second activation function; and the central model determining the intention in the emotion recognition result based on a third activation function.

[0064] According to another optional embodiment of this application, feature data is input into the participant's local model, which is a fully connected neural network model. This fully connected neural network model can be divided into three groups, each group having different classification dimensions and different numbers of layers. Therefore, depending on the classification type, the fully connected neural network can automatically adjust its structure, achieving the technical effect of using a single fully connected neural network model to complete three classifications, and utilizing different activation functions to activate corresponding neurons and obtain corresponding classification results based on the classification scenario.

[0065] In some optional embodiments of this application, after determining the emotion recognition results corresponding to different activation functions, the following method is used: obtaining the emotion vector in the emotion recognition results and the product description vector of the product to be recommended, wherein the emotion vector includes emotion, interest points and willingness; determining the correlation coefficient between the emotion recognition results and the product to be recommended based at least on the emotion vector and the product description vector; determining the target recommended product based on the correlation coefficient and displaying the target recommended product.

[0066] As an optional embodiment of this application, after obtaining the user's (target object's) emotional state (emotion), intention (interest), and shopping type (willingness), the user's current emotional vector is derived by combining the above values. A correlation coefficient is calculated between this vector and the product vector to obtain a score for each product item. Based on the score, products are selectively recommended. The emotional vector can be represented as:

[0067]

[0068] Where E represents emotion, I represents interest, and L represents willingness.

[0069] The correlation coefficient between the sentiment vector and the product description matrix P is calculated, and products with higher correlation coefficients are recommended. Users can see the recommended information on the homepage, and can view detailed product information by clicking on them. Users can also set their preferred products.

[0070] In some optional embodiments of this application, the emotion of the target object in the emotion recognition result is displayed; the emotion adjusted by the target object is input into the central model and the central model is trained; the target emotion recognition result is determined based on the trained central model.

[0071] As another optional embodiment of this application, the user's (target object's) emotions are displayed on an emotion recognition display interface. In this interface, the user can change their own recognition result. The recognition network inputs the modified emotion into a central model to train the model, improving its accuracy. The recommendation system then adjusts its recommendations based on the latest output value of the central model.

[0072] Figure 2-1 This is a flowchart illustrating a product recommendation method based on the present emotion recognition method according to an embodiment of this application. Figure 2-2 This is a flowchart of another product recommendation based on the emotion recognition method according to an embodiment of this application. Figure 2-3 This is a flowchart of another product recommendation based on the emotion recognition method according to an embodiment of this application, such as... Figure 2-1 , Figure 2-2 and Figure 2-3 As shown, it includes:

[0073] In step S202, the user (target object) logs into the shopping system. Personal sensors collect the user's EEG and ECG signals, and the big data backend automatically collects the user's behavior logs and online sentiment data. The HDFS file management system stores the multimodal data in the database.

[0074] Step S204: Perform data cleaning, noise reduction, artifact removal, and other preprocessing on the collected data to obtain well-organized data.

[0075] Step S206: Using the data encoding Word2Vector and OneHot technologies, the text word sequence is converted into a computer-recognizable numerical sequence representation.

[0076] Step S208: Manual feature extraction is performed on the preprocessed digital sequences, that is, the time domain, frequency domain and nonlinear domain related features of the sequence signal are extracted.

[0077] Step S210: Since manual features are limited by experience, a convolutional neural network is used on the original numerical sequence. The input network automatically extracts high-dimensional features. The features extracted by the two methods are then integrated, that is, the discrete low-dimensional features are combined into multi-dimensional features, making them global features. Global features contain comprehensive information, but because of their larger dimensionality, the processing is more complicated.

[0078] Therefore, by inputting global features into the attention mechanism of the Transformer network (FL-GlobalTrans-DNN), the data dimensionality is reduced, and the relevant feature information between the beginning and end of the sequence is better understood, which is suitable for task scenarios with high temporal correlation, such as emotion recognition.

[0079] Step S212: After obtaining suitable features, it is necessary to perform calculations on the features and input them into the federated learning framework. Before inputting the features into the federated learning framework, it is necessary to estimate the computing power resources of each participating device.

[0080] Step S212-2: Determine whether each slave server (participating device) has the computing power to run the model.

[0081] Step S212-4: Identify participating devices with computing power that have a workable model as participants.

[0082] Step S212-6: Determine participating devices that do not possess computing power for a runnable model as non-participants.

[0083] In step S214, the central server obtains the following three values: the computing power resource usage of each participating device, the amount of resources required to run the model, and the network link status.

[0084] Step S214-2: If the computing resources available on a device are less than or equal to the resources required to run the algorithm model, the device is directly eliminated and not considered as a participant. The top k devices with good network link bandwidth are selected as participants.

[0085] Step S216: The central server publishes the global model status.

[0086] Step S218: Participants input their collected data into the algorithm network to train their respective models and perform gradient updates.

[0087]

[0088] Where, ω k b i Let η be the parameter to be trained, and η be the initial learning rate.

[0089] The loss functions for participants and the central server are set differently. The loss function of the central server's model changes according to the changes in the participants' loss values, as follows:

[0090]

[0091] Step S218-2: Determine whether the participant's local model has converged. If the participant's local model has not converged, return to step S214.

[0092] Step S220: When the local model of each participant converges, assign weight coefficients to different participants and different ω parameters based on the recognition loss value of each participant.

[0093]

[0094] Where, r k The weighting coefficients for the participants.

[0095] Step S222: Update the parameters of the central server model.

[0096] Step S224: If the minimum loss value of the central server is less than or equal to the average loss value of the participants, the model recognition ends and the corresponding emotion recognition result can be obtained. Otherwise, the above training process needs to be repeated until the central server model converges.

[0097] Step S226: After obtaining the user's emotional state, shopping type, and intention (interest), the user's current emotional vector is derived by combining the above values. The correlation coefficient is calculated with the product vector to obtain the product score. Based on the score, products are selectively recommended.

[0098] Step S228: Determine whether the user's emotions are positive.

[0099] Step S228-2: If the user's mood is positive, continue to push similar product items.

[0100] In steps S228-4 and S228-5, if the user's emotion is not positive, the knowledge base adjusts the push notifications based on the user's emotion, selecting products that are more relevant to the emotion.

[0101] Users can view recommended products and product details on the product recommendation homepage in the online shopping interaction in step S202. Users can also view their own sentiment assessment results in real time and judge the recognition effect themselves. If the recognition is inaccurate, users can modify the sentiment results themselves through step S228-6, which can effectively improve the accuracy of the background algorithm model and enable precise recommendations. The system continues to evaluate user sentiment in real time based on the new recommendations.

[0102] Figure 3 This is a schematic diagram of an interactive data acquisition according to an embodiment of this application, such as... Figure 3 As shown, when a user (the target audience) logs into the shopping system and browses products, sensors record the physiological signals generated, including electroencephalogram (EEG) and electrooculogram (EOG) signals. The background big data system, Flume, records real-time user interaction logs, noting clicks, browsing of product information, and whether the user viewed detailed product information. The database stores user descriptions, shopping records, and is continuously updated with statistics on purchase frequency, product categories, and query information. Another database also records online sentiment data, specifically current trending sentiment categories and related trending product information across various online platforms.

[0103] Figure 4 This is a schematic diagram of a raw data processing method according to an embodiment of this application, such as... Figure 4 As shown, different processing flows are provided for different types of data, but they ultimately form the same representation, i.e., ordered vectors.

[0104] For physiological data, due to its mutability, filtering and noise reduction are necessary. For EEG signals, discrete wavelet transform is used to remove high and low frequency noise from the original physiological data. For ECG signals, Butterworth filters are used for noise reduction preprocessing. After data cleaning, feature extraction is performed, extracting time-domain signals such as mean and variance, and then frequency signals and nonlinear characteristics such as multi-scale dispersive entropy are extracted in the frequency domain. Features are then normalized. For interaction log data, keyword-based segmentation extracts user-related keyword phrases, which are then transformed using Word2Vector to convert textual descriptions into numerical vector descriptions. For user historical data, the data is summarized, user tags are extracted, and the tags are filtered and OneHot encoded. For trending information, it is mapped to products to form trending product description vectors.

[0105] Figure 5This is a flowchart of a GlobalTrans-DNN algorithm according to an embodiment of this application, such as... Figure 5 As shown, feature fusion forms global features, thereby reducing the complexity of feature classification and extraction. The introduction of the Transformer module not only extracts complete time-frequency information but also reduces feature complexity, achieving a filtering effect.

[0106] Handcrafted features and deep features are fused in the spatial dimension to form high-dimensional global spatiotemporal features. Furthermore, the problem of long-distance limitations in sequences is addressed through parallel processing of the sequences. These features are then input into a Transformer encoding layer to extract deep features. The encoded sequence is then fed into an interactive attention layer to calculate the similarity features of each global feature, achieving temporal alignment of features across different modalities. The Transformer-optimized features are then input into a unique deep adaptive neural network. The network selectively activates neurons based on the classification task, thereby reducing model complexity and accelerating recognition efficiency.

[0107] Figure 6 This is a schematic diagram of an optimized federated learning model according to an embodiment of this application, such as... Figure 6 As shown, the computational load required to run the algorithm model is determined based on the computing power evaluation network, the remaining computing resources of each participating device, and the network status, thereby selecting participating devices. The top k devices are selected as participants, and an initial model is distributed to each participant. Each participant trains its model locally. Once the local model converges, weight coefficients are assigned to the participants based on the loss value, and the model parameters are sent to the central server for parameter aggregation. The central server updates the global parameters based on the weight coefficients and the average model parameters, and then proceeds to the next round of model training. When the global loss function based on the weights converges, the training ends, and the algorithm model is obtained.

[0108] Figure 7 This is a flowchart of a federated learning method according to an embodiment of this application. Compared with existing federated learning methods, this method has made corresponding innovations in participant selection and global fusion parameter updates, enabling the model to converge faster under the federated learning computation framework. While ensuring model accuracy, it consumes less computation time. This method can achieve a speedup of 2-3 orders of magnitude in model convergence.

[0109] Based on the computing power prediction network, the computing power required to complete the algorithm model is estimated. Devices that cannot provide sufficient computing power are eliminated in the first step. Then, based on the remaining computing resources and network conditions, k participants are selected, and the model from the central server is distributed to the local machines of these k participants. Participants train locally, and upon convergence, the weighted aggregation parameter ω is calculated based on the recognition accuracy of each participant. k This value is used to update the global model parameters. If the central server model converges, training stops; otherwise, the next round of training begins.

[0110] Figure 8 This is a schematic diagram of an emotion recognition result according to an embodiment of this application, such as... Figure 8 As shown, based on the model's output, four user emotions are identified using Arousal, Valence, and Dominance to obtain the user's current emotional state; user interests are obtained based on the user's (target audience's) interest description; and the user's shopping intention is determined based on the user's shopping type. By analyzing the user's current emotional state, interests, and shopping type, the recommendation strategy and products are modified to encourage positive changes in the user's behavior.

[0111] Figure 9 This is a structural diagram of an emotion recognition device according to an embodiment of this application, such as... Figure 9 As shown, the device includes:

[0112] The multimodal data acquisition module 902 is used to collect data generated by users while browsing and shopping through sensors, big data, and other technologies. The data types are primarily related to emotions: physiological signals reflect user emotions, interaction data reflects user interests, historical data creates user profiles, and emotion collection displays current public trends. The multimodal data is then stored completely.

[0113] The multimodal data processing module 904 is used for serialization of multimodal data. It utilizes natural language processing techniques (Word2Vector) and machine learning statistical methods to convert textual information describing people and products into sequence vectors. It employs both manual feature extraction and automatic feature extraction via deep networks. Time-frequency and non-linear features are extracted manually, while high-dimensional features are automatically extracted using convolutional neural networks. Multiple feature classes are locally integrated to form global features, facilitating unified processing and saving processing time. Features are also integrated spatially to form higher-dimensional features.

[0114] The temporally optimized federated learning module 906 performs the following five steps:

[0115] S1. Selection of Participant Devices: The central server obtains the computing resource usage of each device, uses computing power to estimate the network, assesses the computing power required to complete the algorithm model, selects devices with matching computing power resources, and then selects k candidates based on the network link bandwidth.

[0116] S2. Model Broadcast: After initializing appropriate model parameters, the central server broadcasts the current model parameters to the participants;

[0117] S3. Local Training: Participants train locally on the data of the current device based on the shared model, set the loss function, and the loss function of the participants is the same. The loss function of the central server is determined according to the weight coefficient of the participants' loss function.

[0118] S4. Parameter Aggregation: After local training is complete and the local loss value converges, the weight values ​​of each participant can be calculated based on the magnitude of their loss function and their accuracy. An average weight value is then calculated based on the different weight values ​​of different participants. Value, where r k The weight coefficient for the k-th participant is determined by updating the global weight value ω of the central server. (t+1) =ω (t) -ω (t-1) ;

[0119] S5. Model Convergence: Repeat the above training until the model converges, that is, the minimum loss value of the central server is less than the average loss value of all participants.

[0120] The temporal-optimized federated learning module 906 prioritizes efficiency, innovating in model deployment and parameter fusion iteration. Models are deployed to participating devices with good network conditions and strong computing power, thus accelerating feedback. The parameter fusion iteration takes into account the differences in personalized data, assigning different weights to participants with good recognition performance, thereby speeding up model training.

[0121] The user sentiment profile comprehensive analysis module 908 is used to input global features into the Transformer model. Using a self-attention mechanism, it acquires the best features through self-encoding and decoding to better understand the relevant feature information between sequences. The Transformer outputs features with lower dimensionality. These processed features are then input into a deep neural network. Since there are three recognition tasks, using three separate models would be too time-consuming. Therefore, a single network is used for discriminative analysis. The neurons in the deep neural network are selectively activated based on the classification task. For example, a 6-layer deep neural network is used for the four-class classification of sentiment, while a 12-layer network is needed for the five-class classification of user type.

[0122] The recommended interaction module 910 is used to analyze the sentiment results, which are ultimately represented as vectors. The correlation coefficient between the product description matrix P and the product description matrix is ​​calculated, and products with higher correlation coefficients are recommended. Users can see the recommended information on the homepage and view detailed product information by clicking on them. Users can also set their recommended products. Finally, there is an emotion recognition display interface where users can change their recognition results, such as emotion, interests, and user profile. The recognition network improves the model accuracy based on user changes, and the recommendation system adjusts its recommendations based on the changed information.

[0123] Figure 10 This is a structural diagram of another emotion recognition device according to an embodiment of this application, such as... Figure 10 As shown, the device includes:

[0124] The extraction module 1002 is used to acquire the raw data of multiple participating devices and extract the feature data of the raw data, wherein the participating devices are participants in the federated learning based on emotion recognition;

[0125] The training module 1004 is used to train the local model of the participating device based on the initialization parameters and feature data, obtain the target parameters, and upload the target parameters to the central model. The initialization parameters and the central model are the parameters and model that the participating device receives locally from the central server. The central server is a server for federated learning based on emotion recognition.

[0126] The first determining module 1006 is used to determine a first loss value based on the target parameters, and to determine a second loss value based on the first loss value and the weight coefficient of the first loss value. The first loss value is the loss value of the target participating device in the target time period, and the second loss value is the loss value of the central server in the target time period. The first loss value and the second loss value are used to characterize the accuracy of the model training results.

[0127] The second determining module 1008 is used to determine the emotion recognition result corresponding to different activation functions based on different activation functions when the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time. The target time is the previous time adjacent to the current time.

[0128] It should be noted that, Figure 10 Preferred embodiments of the shown examples can be found in [reference needed]. Figure 1 The relevant descriptions of the embodiments shown will not be repeated here.

[0129] This application also provides a non-volatile storage medium, which includes a stored program, wherein the program, when running, controls the device where the storage medium is located to execute the above-mentioned emotion recognition method.

[0130] A non-volatile storage medium performs the following functions: It acquires raw data from multiple participating devices and extracts feature data from the raw data, where the participating devices are participants in the federated learning based on emotion recognition; it trains the local models of the participating devices based on initialization parameters and feature data to obtain target parameters, and uploads the target parameters to a central model, where the initialization parameters and the central model are the parameters and model received locally by the participating devices from the central server, respectively, and the central server is the server for the federated learning based on emotion recognition; it determines a first loss value based on the target parameters, and a second loss value based on the first loss value and its weight coefficients, where the first loss value is the loss value of the target participating device within the target time period, and the second loss value is the loss value of the central server within the target time period; the first and second loss values ​​are used to characterize the accuracy of the model training results; when the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time, it determines the emotion recognition result corresponding to different activation functions based on different activation functions, where the target time is the previous time adjacent to the current time.

[0131] This application also provides an electronic device, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described emotion recognition method when it runs.

[0132] The processor runs a program that performs the following functions: acquiring raw data from multiple participating devices and extracting feature data from the raw data, wherein the participating devices are participants in the federated learning based on emotion recognition; training the local models of the participating devices based on initialization parameters and feature data to obtain target parameters, and uploading the target parameters to the central model, wherein the initialization parameters and the central model are the parameters and model received locally by the participating devices from the central server, and the central server is the server for the federated learning based on emotion recognition; determining a first loss value based on the target parameters, and determining a second loss value based on the first loss value and its weight coefficient, wherein the first loss value is the loss value of the target participating device within the target time period, and the second loss value is the loss value of the central server within the target time period, and the first and second loss values ​​are used to characterize the accuracy of the model training results; and determining the emotion recognition results corresponding to different activation functions based on different activation functions when the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time, wherein the target time is the previous time adjacent to the current time.

[0133] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0134] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0135] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0138] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0139] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An emotion recognition method, characterized in that, include: The raw data from multiple participating devices are acquired and feature data from the raw data is extracted. The participating devices are participants in the federated learning based on emotion recognition, and the raw data includes physiological data, interaction data, historical data, and hotspot data. The local model of the participating device is trained according to the initialization parameters and the feature data to obtain the target parameters, and the target parameters are uploaded to the central model. The initialization parameters and the central model are respectively the parameters and model that the participating device receives locally from the central server. The central server is the server of the federated learning based on emotion recognition. A first loss value is determined based on the target parameters, and a second loss value is determined based on the first loss value and the weight coefficient of the first loss value. The first loss value is the loss value of the target participating device in the target time period, and the second loss value is the loss value of the central server in the target time period. The first loss value and the second loss value are used to characterize the accuracy of the model training results. If the minimum value of the second loss at the current time is less than or equal to the average value of the first loss at the target time, the emotion recognition result corresponding to the different activation functions is determined according to different activation functions. The target time is the previous time adjacent to the current time. The emotion recognition result includes: the emotion, interest points, and intentions of the target object. The central model determines the emotion in the emotion recognition result according to the first activation function; the central model determines the interest points in the emotion recognition result according to the second activation function; and the central model determines the intentions in the emotion recognition result according to the third activation function.

2. The method according to claim 1, characterized in that, Before determining a first loss value based on the target parameters, and before determining a second loss value based on the first loss value and its weighting coefficients, the method further includes: The computing resource utilization rate of each participating device is determined based on the CPU utilization rate, memory utilization rate, and hard disk utilization rate of each participating device. Determine the data consumption of each participating device when it runs its respective local model; When the computing resource utilization rate is greater than the data consumption, the participating device is determined to be the first participating device; When the network bandwidth between the central server and the first participating device is the target bandwidth, the first participating device is determined to be the target participating device.

3. The method according to claim 1, characterized in that, Extracting feature data from the original data includes: The physiological data in the original data are preprocessed to obtain the target physiological data, wherein the preprocessing includes at least: data cleaning and normalization. The first feature of the target physiological data is obtained, wherein the first feature includes at least one of the following: statistical features, frequency domain features, and nonlinear entropy domain features; The target physiological data is input into a first deep learning model to obtain a second feature of the target physiological data, wherein the second feature is a multi-dimensional feature; The first feature and the second feature are fused to obtain a target matrix, wherein the target matrix is ​​a three-dimensional feature matrix; The target matrix is ​​input into the second deep learning model to obtain time-frequency features, and the time-frequency features are determined as the feature data.

4. The method according to claim 1, characterized in that, The emotions mentioned include at least one of the following: pleasure, positivity, negativity, and anger; The points of interest are used to indicate the type of goods that the target object intends to buy, wherein the points of interest include at least one of the following: daily necessities and decorative items; The intention is used to indicate the type of purchasing behavior of the target object, wherein the intention includes at least one of the following: regular, sporty, simple, surfing, access, and bargaining.

5. The method according to claim 1, characterized in that, After determining the emotion recognition results corresponding to the different activation functions, the method further includes: Obtain the emotion vector from the emotion recognition result and the product description vector of the product to be recommended, wherein the emotion vector includes the emotion, the interest point, and the intention; Based at least on the sentiment vector and the product description vector, determine the correlation coefficient between the sentiment recognition result and the product to be recommended; The target recommended products are determined based on the correlation coefficient, and then the target recommended products are displayed.

6. The method according to claim 5, characterized in that, The method further includes: The emotion of the target object is displayed in the emotion recognition results; The emotion adjusted by the target object is input into the central model to train the central model; The target emotion recognition result is determined based on the trained central model.

7. An emotion recognition device, characterized in that, include: An extraction module is used to acquire raw data from multiple participating devices and extract feature data from the raw data. The participating devices are participants in federated learning based on emotion recognition, and the raw data includes physiological data, interaction data, historical data, and hotspot data. The training module is used to train the local model of the participating device according to the initialization parameters and the feature data to obtain the target parameters, and upload the target parameters to the central model. The initialization parameters and the central model are respectively the parameters and model that the participating device receives locally from the central server. The central server is the server of the federated learning based on emotion recognition. The first determining module is used to determine a first loss value based on the target parameters, and to determine a second loss value based on the first loss value and the weight coefficient of the first loss value, wherein the first loss value is the loss value of the target participating device in the target time period, the second loss value is the loss value of the central server in the target time period, and the first loss value and the second loss value are used to characterize the accuracy of the model training results. The second determining module is used to determine the emotion recognition result corresponding to different activation functions based on different activation functions when the minimum value of the second loss value at the current time is less than or equal to the average value of the first loss value at the target time. The target time is the previous time adjacent to the current time. The emotion recognition result includes the emotion, interest points, and intentions of the target object. The central model determines the emotion in the emotion recognition result based on the first activation function; the central model determines the interest points in the emotion recognition result based on the second activation function; and the central model determines the intentions in the emotion recognition result based on the third activation function.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the emotion recognition method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the emotion recognition method according to any one of claims 1 to 6.

Images