A method, apparatus, equipment, and storage medium for training a customer emotion recognition model for business halls.

Abstract

This invention discloses a method, apparatus, device, and storage medium for training a customer emotion recognition model in a business hall. The method includes: acquiring a multimodal dataset of customers in the business hall, where each sample in the dataset includes at least a standardized facial image sequence, acoustic feature map, environmental features, and an emotion label; extracting features through a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; inputting the high-dimensional emotion feature vector into a meta-learning-based emotion recognition model to obtain the emotion confidence distribution for each sample; inputting the environmental features and the emotion confidence distribution into an adaptive threshold generation network to obtain an emotion judgment threshold; updating each network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with a total loss function; and constructing a customer emotion recognition model based on the updated multimodal feature extraction network, the emotion recognition model, and the adaptive threshold generation network. This method can obtain a highly adaptable customer emotion recognition model.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of power marketing technology, and in particular to a method, apparatus, equipment and storage medium for training a customer emotion recognition model for business halls. Background Technology

[0002] With the rapid development of smart power services, power service halls, as crucial customer service front-ends, directly impact a company's social image and customer satisfaction through their service quality. Timely and accurate identification of customers' emotional states during transactions is vital for preventing service disputes, identifying potential complaint risks, and providing personalized care services. Therefore, emotion recognition technology based on audio and video analysis is gradually being introduced into the intelligent monitoring and service quality assessment systems of power service halls.

[0003] Existing methods typically apply general emotion recognition models directly to power service halls. However, in this specific vertical scenario, general emotion recognition models cannot accurately identify customer emotions. Therefore, there is a need for a method that can accurately identify customer emotions in power service halls. Summary of the Invention

[0004] This invention provides a method, apparatus, equipment, and storage medium for training a customer emotion recognition model in a business hall, in order to solve the problem that existing technologies cannot accurately identify customer emotions in power business halls.

[0005] According to one aspect of the present invention, a method for training a customer emotion recognition model for a business hall is provided, the method comprising: Obtain a multimodal dataset of customers in the business hall, wherein each sample in the multimodal dataset includes at least a standardized facial image sequence, acoustic feature map, environmental features, and emotion label; A high-dimensional emotion feature vector is obtained by extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network. The high-dimensional emotion feature vector is input into the meta-learning-based emotion recognition model to obtain the emotion confidence distribution of each sample. The environmental features and the emotion confidence distribution are input into an adaptive threshold generation network to obtain the emotion determination threshold. Based on the emotion confidence distribution, emotion judgment threshold, and emotion label, the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network are updated using the total loss function. A customer emotion recognition model is then constructed based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

[0006] According to another aspect of the present invention, a training device for a customer emotion recognition model in a business hall is provided, the device comprising: The acquisition module is used to acquire a multimodal dataset of customers in the business hall. Each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label. The extraction module is used to extract facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; The recognition module is used to input the high-dimensional emotion feature vector into the emotion recognition basic model based on meta-learning to obtain the emotion confidence distribution of each sample; The determination module is used to input the environmental features and the emotion confidence distribution into the adaptive threshold generation network to obtain the emotion determination threshold. The update module is used to update the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function, and to construct a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: at least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the customer emotion recognition model training method for business halls as described in any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the customer emotion recognition model training method for business halls as described in any embodiment of the present invention.

[0009] This invention discloses a method, apparatus, device, and storage medium for training a customer emotion recognition model in a business hall. The method includes: acquiring a multimodal dataset of customers in the business hall, wherein each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label; extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map using a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; inputting the high-dimensional emotion feature vector into a meta-learning-based emotion recognition basic model to obtain an emotion confidence distribution for each sample; inputting the environmental features and the emotion confidence distribution into an adaptive threshold generation network to obtain an emotion judgment threshold; updating the multimodal feature extraction network, the emotion judgment threshold, and the emotion label using a total loss function, and constructing a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition basic model, and adaptive threshold generation network. This method can train a customer emotion recognition model that can accurately identify customer emotions and is highly adaptable, solving the problem that existing technologies cannot accurately identify customer emotions in power service halls.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating a customer emotion recognition model training method for a business hall, as provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of a multimodal feature extraction network structure provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating the training process of a meta-learning-based emotion recognition foundation model provided in an embodiment of the present invention; Figure 4 A flowchart illustrating a customer emotion recognition model training method for a business hall, provided as an embodiment of the present invention; Figure 5This is a schematic diagram of the structure of a customer emotion recognition model training device for a business hall provided in Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of the electronic device used in the customer emotion recognition model training method for business halls according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention. It should be understood that the various steps described in the method embodiments of the present invention can be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0014] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, any variations of the terms "comprising" and "having," etc., are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0017] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0018] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0019] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0022] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0023] Applying a general emotion recognition model directly to the specific vertical scenario of a power service hall presents the following challenges: (1) The business hall is noisy and the business handling broadcasts are constantly being broadcast. The lighting conditions in the hall also change with the weather and time, resulting in poor robustness of traditional emotion recognition models based on a single modality (pure video or pure audio).

[0024] (2) In actual business, most customers are in a calm state, and samples of negative emotions such as strong anger or anxiety are relatively scarce (long-tail distribution). Traditional deep learning models such as Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) usually require massive amounts of balanced data for training. In the absence of negative samples, they are prone to overfitting, resulting in underreporting of sudden negative emotions.

[0025] (3) The layout of business halls, dialects and customer characteristics vary greatly in different regions. The existing model has weak generalization ability. When a model trained in one business hall is directly deployed to another business hall, the accuracy often drops significantly due to domain offset, resulting in a serious cold start problem for the model in the new scenario.

[0026] (4) Existing technologies mostly use a fixed confidence threshold of the normalized exponential function (Softmax) to determine the emotion category. However, in a noisy environment, the confidence distribution of the model output tends to be flat (high entropy value), and a fixed threshold can easily lead to a large number of false alarms (e.g., misjudging loud talking as anger), which increases the burden of ineffective intervention for staff.

[0027] Therefore, there is an urgent need for a new method that can collaboratively optimize model initialization and decision boundaries while improving model generalization ability and scene adaptation accuracy, so as to effectively meet the needs of power business halls for a highly robust and accurate emotion recognition system.

[0028] Example 1 Figure 1 This is a flowchart illustrating a customer emotion recognition model training method for a business hall, as provided in Embodiment 1 of the present invention. This method is applicable to recognizing customer emotions in a business hall. The method can be executed by a customer emotion recognition model training device for a business hall, wherein the device can be implemented by software and / or hardware and is generally integrated on an electronic device. In this embodiment, the electronic device includes, but is not limited to, devices such as computers.

[0029] like Figure 1 As shown, Embodiment 1 of the present invention provides a method for training a customer emotion recognition model for business halls, comprising the following steps: S110. Obtain a multimodal dataset of customers in the business hall, wherein each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label.

[0030] In this dataset, the service hall can be a power company service hall, and the customer can be a user within that service hall. The multimodal dataset can be a collection of data of different types, and each sample in the multimodal dataset must include at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label. The standardized facial image sequence can refer to a sequence of facial images of the customer; the acoustic feature map can be a feature map of audio related to the customer; the environmental features can be environmental data related to the customer's service hall; and the emotion label can be a manually labeled tag representing the customer's emotions. It is understood that the customer's facial and voice-related data is obtained with the customer's consent, or is used for emotion recognition only with the customer's consent.

[0031] In this embodiment, customer data from the business hall can be acquired and preprocessed to construct a multimodal dataset. For example, facial videos of customers can be acquired using a high-definition camera, and audio recordings of customer conversations can be acquired using a directional microphone. Face detection, key point localization, and alignment cropping are performed on the facial videos to remove background interference; noise reduction, silence removal, and frame-by-frame windowing are performed on the conversation audio. Simultaneously, timestamp alignment technology is used to achieve spatiotemporal synchronization of the audio and video streams, constructing multimodal sample pairs to obtain the multimodal dataset.

[0032] In one embodiment, obtaining the multimodal dataset of customers in the business hall includes: collecting customer data from each business hall and constructing structured metadata; each sample in the structured metadata includes customer facial video, dialogue audio, environmental metadata, and emotion tags; preprocessing the customer facial video and the dialogue audio respectively to obtain a standardized facial image sequence and an acoustic feature map; performing spatiotemporal alignment of the standardized facial image sequence and the acoustic feature map based on timestamp alignment technology to obtain an aligned standardized facial image sequence and a corresponding acoustic feature map; and constructing a multimodal dataset based on the aligned standardized facial image sequence, the corresponding acoustic feature map, environmental metadata, and emotion tags.

[0033] Among these, facial video refers to facial data captured by a camera, and conversation audio refers to the customer's voice data. Emotional labels can be manually labeled, such as satisfaction, calmness, anger, and anxiety. Standardized facial image sequences refer to standardized sequences of facial images of uniform size. Acoustic feature maps refer to the spectral characteristics of speech over time. Timestamp alignment technology refers to a data synchronization processing technique that uses the timestamps inherent in the data as a benchmark to perform time-series correction, matching, and normalization on multiple sets of time-series data from different sources, enabling each data point to correspond at specific moments and be synchronized in time on a unified timeline.

[0034] In this embodiment, customer data from each business hall can be collected to construct structured metadata. Each sample in the structured metadata includes customer facial video, dialogue audio, environmental metadata, and emotion tags. The customer facial video and dialogue audio are preprocessed to obtain standardized facial image sequences and acoustic feature maps. The standardized facial image sequences and acoustic feature maps are spatiotemporally aligned based on timestamp alignment technology to obtain aligned standardized facial image sequences and corresponding acoustic feature maps. Based on the aligned standardized facial image sequences, corresponding acoustic feature maps, environmental metadata, and emotion tags, a multimodal dataset is constructed.

[0035] In one embodiment, the environmental metadata includes at least one or more of the following: lighting conditions, noise level, customer type, business scenario, timestamp, and environmental sensor data.

[0036] In this embodiment, lighting conditions can include natural light, artificial light, and mixed light, while noise levels can be categorized as quiet, noisy, and broadcast interference. Customer types can include characteristics such as age, gender, and facial expression. Business scenarios can include consultation, payment, and complaint scenarios, and timestamps can refer to the time of data collection. Environmental sensor data can refer to the noise level and light intensity of the business hall.

[0037] The data collection process must cover diverse service hall environments and customer states. For example, high-definition visible light cameras and high-sensitivity directional microphone arrays can be deployed in multiple typical power service halls, such as those in city center halls, suburban halls, and peak-hour halls, to simultaneously collect facial video and audio of customers in front of the service windows. The video capture frame rate should be no less than 25fps, with a resolution of at least 640×480 to ensure clear facial details; the audio sampling rate should be set to 16kHz, with a bit depth of 16bit, and directional sound pickup technology should be used to reduce environmental noise interference. During the collection process, strict adherence to data privacy and ethical guidelines is required. Customer faces should be blurred in real-time until authorized for use, and all data should be stored on a local encrypted server. The collected data should cover different lighting conditions, noise levels, customer types, and business scenarios to ensure data diversity and representativeness. Each sample must be recorded with a corresponding timestamp, environmental sensor data, and manually labeled emotion tags to form structured metadata.

[0038] The collected raw audio and video data undergoes standardized preprocessing to improve data quality and adapt to model input requirements. For facial videos, deep learning-based face detection algorithms, such as Multi-task Cascaded Convolutional Networks (MTCNN) or You Only Look Once Face (YOLO-Face), can be used to locate the customer's facial region in each frame. Face alignment and normalization are then performed, cropping a standardized 224×224 pixel facial image sequence. Histogram equalization and adaptive illumination correction techniques are employed to mitigate the effects of uneven illumination. For dialogue audio, Wiener filtering and spectral subtraction are used to remove steady-state current noise and air conditioning noise from the background. A speech activity detection algorithm is used to remove silent segments, retaining only the effective human voice. Simultaneously, utilizing the unified timestamps of the acquisition devices and taking video frames of standardized facial image sequences as a benchmark, the dialogue audio is segmented into short segments corresponding to the video frames. Mel-Frequency Cepstral Coefficients (MFCC) features and prosodic features are extracted and transformed into visualized acoustic feature maps, thereby achieving alignment with the visual images in the data dimension. Subsequently, the processed standardized facial image sequences, corresponding acoustic feature maps, and environmental features corresponding to environmental metadata are packaged into standardized multimodal sample pairs in the format of "video segment-audio segment-environmental feature-emotion tag".

[0039] This embodiment can collect high-quality, well-aligned multimodal data from the real-world scenario of a power service hall, and after a series of standardization processes, form a standardized dataset that can be used for model training and testing.

[0040] S120. Facial visual features and speech acoustic features are extracted from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network to obtain a high-dimensional emotion feature vector.

[0041] Among these, multimodal feature extraction networks refer to networks capable of processing both video and audio data simultaneously. Facial visual features refer to features that characterize facial states and emotional information. Speech acoustic features refer to features that reflect a customer's timbre, intonation, speech rate, and energy distribution. High-dimensional emotion feature vectors refer to high-dimensional vectors that fuse facial visual features and speech acoustic features.

[0042] In this embodiment, a high-dimensional emotion feature vector can be obtained by extracting facial visual features and speech acoustic features from a standardized facial image sequence and acoustic feature map using a multimodal feature extraction network.

[0043] In one embodiment, the step of extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map using a multimodal feature extraction network to obtain a high-dimensional emotion feature vector includes: inputting the standardized facial image sequence into the visual branch of the multimodal feature extraction network to extract facial visual features to obtain a visual feature vector; inputting the acoustic feature map into the speech branch of the multimodal feature extraction network to extract speech acoustic features to obtain a speech feature vector; and using the multimodal cross-attention mechanism of the multimodal feature extraction network, dynamically adjusting the fusion coefficient based on the information entropy of the current sample face and speech, and fusing the visual feature vector and the speech feature vector to obtain a high-dimensional emotion feature vector.

[0044] Among them, visual feature vectors refer to vectors used to represent facial features of a customer, while speech feature vectors refer to vectors used to represent voice features of a customer. Information entropy refers to a quantitative indicator representing the degree of disorder or uncertainty of features; the higher the entropy value, the more dispersed the feature information distribution and the stronger the uncertainty; the lower the entropy value, the more concentrated the feature distribution and the higher the certainty of the information. The fusion coefficient refers to an adjustable parameter used to regulate the respective contribution weights of visual and speech features during the fusion process, and can dynamically allocate weights according to the contribution of different modalities in emotional expression.

[0045] In this embodiment, the multimodal feature extraction network may include a visual branch, a speech branch, and a multimodal cross-attention mechanism. A standardized facial image sequence can be input into the visual branch to extract facial visual features to obtain a visual feature vector. An acoustic feature map can be input into the speech branch to extract speech acoustic features to obtain a speech feature vector. Then, through the multimodal cross-attention mechanism, the fusion coefficient is dynamically adjusted according to the information entropy of the current sample face and speech to fuse the visual feature vector and the speech feature vector to obtain a high-dimensional emotion feature vector.

[0046] For example, Figure 2 This is a schematic diagram of a multimodal feature extraction network structure provided in an embodiment of the present invention, as shown below. Figure 2As shown, to achieve a comprehensive and robust representation of customer emotions, a dual-branch attention fusion multimodal feature extraction network is designed, constructing visual and speech branches respectively. Considering the computing power limitations of edge devices in the business hall, the visual branch adopts a lightweight and efficient mobile lightweight convolutional network (MobileNet) or an 18-layer residual network (ResNet-18) as the backbone network. The input is a preprocessed sequence of consecutive multi-frame facial images, and the visual branch extracts the spatiotemporal features of facial expressions through 3D convolutional layers. The speech branch can use a time-delay neural network (TDNN) or a transform encoder, with the input being the Mel-spectrum map of the audio (i.e., the acoustic feature map). Through multi-layer temporal convolution, the long-term dependence of acoustic features is captured, and the MFCC and emotional prosodic features of the speech are extracted. To fuse information from two modalities, a multimodal cross-attention mechanism is introduced. First, visual and speech feature vectors are projected onto a common subspace. Then, intermodal attention weights are calculated, and the fusion coefficients are dynamically adjusted based on the information entropy of facial and speech data in the current sample. The visual and speech feature vectors are then fused to obtain a high-dimensional emotion feature vector containing rich semantic information. For example, speech features are prominent when expressing "anger," while facial expressions are prominent when expressing "helplessness." Alternatively, when a customer is detected wearing a mask, causing facial features to be blurred, the network automatically assigns higher weights to the speech feature vector. Conversely, when the environment is extremely noisy, resulting in a low speech signal-to-noise ratio, the focus is on the visual feature vector. During the initial training phase, the multimodal feature extraction network can be pre-trained using the publicly available Facial Expression Recognition 2013 Dataset (FER2013) and the Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) to accelerate convergence and improve feature expressiveness.

[0047] S130. Input the high-dimensional emotion feature vector into the emotion recognition basic model based on meta-learning to obtain the emotion confidence distribution of each sample.

[0048] Meta-learning refers to a machine learning paradigm that allows models to learn general prior knowledge and initial parameters from a large number of different tasks, quickly grasp learning patterns, and adapt to new tasks and scenarios with only a small number of new samples. Emotion confidence distribution refers to the distribution of the probability that a sample belongs to different emotion categories.

[0049] In this embodiment, a meta-learning-based emotion recognition basic model can be constructed. The high-dimensional emotion feature vector is input into the meta-learning-based emotion recognition basic model to obtain the emotion confidence distribution of each sample.

[0050] In one embodiment, the meta-learning-based emotion recognition foundation model includes an inner loop and an outer loop. The step of inputting the high-dimensional emotion feature vector into the meta-learning-based emotion recognition foundation model to obtain the emotion confidence distribution of each sample includes: constructing a meta-training set based on a multimodal dataset, and dividing the meta-training set into multiple meta-tasks and sub-training sets corresponding to each meta-task; each sub-training set includes a support set and a query set; for each meta-task, in the inner loop, based on the high-dimensional emotion feature vectors corresponding to each sample in the support set of the meta-task, outputting the emotion confidence distribution of the support set, and calculating the gradient for local fine-tuning based on the original initialization parameters to obtain the inner-layer model parameters; in the outer loop, using the inner-layer model parameters, based on the high-dimensional emotion feature vectors corresponding to the query sets of each meta-task, outputting the emotion confidence distribution of the query set, calculating the loss based on the emotion confidence distribution to obtain the loss of the meta-task; and updating the gradient of the original initialization parameters based on the loss of each meta-task to obtain the global initialization parameters.

[0051] In this context, the inner loop refers to the training loop within each meta-task, with each meta-task having its own inner loop. The outer loop refers to the training loop for all meta-tasks. Meta-learning refers to independent basic learning task units; different meta-tasks simulate different scenarios in a business hall, such as facial expression recognition or speech emotion recognition in a specific scene. The support set refers to the dataset used for training within each meta-task, enabling rapid model learning and fine-tuning. The query set refers to the dataset used to test the model's training performance after training within each meta-task. Original initialization parameters refer to the model's parameters before training, while global initialization parameters refer to the parameters after training. Inner model parameters refer to the model parameters temporarily used by the inner loop.

[0052] In this embodiment, the multimodal dataset can be divided into a meta-training set, a meta-validation set, and a test set to support the subsequent meta-learning training process. For each meta-task, in the inner loop, the emotion confidence distribution of the support set can be output based on the high-dimensional emotion feature vectors corresponding to each sample in the support set of the meta-task, and the gradient can be calculated and locally fine-tuned based on the original initialization parameters to obtain the inner-layer model parameters. In the outer loop, the inner-layer model parameters can be used to output the emotion confidence distribution of the query set based on the high-dimensional emotion feature vectors corresponding to the query set of each meta-task, and the loss can be calculated based on the emotion confidence distribution to obtain the loss of the task. Based on the loss of each meta-task, the original initialization parameters are updated by gradient to obtain the global initialization parameters.

[0053] For example, Figure 3 This is a schematic diagram illustrating the training process of a meta-learning-based emotion recognition foundation model provided in an embodiment of the present invention, as shown below. Figure 3 As shown, to address the weak cross-scene generalization ability of the model and overcome the cold start problem caused by data scarcity and scene differences, a Model-Agnostic Meta-Learning (MAML) framework can be adopted. Based on historical data collected from different business halls, time periods (peak and off-peak), lighting conditions (weak and strong), and background noise levels (high and low), the meta-training set is further divided into multiple meta-tasks, and a sub-training set corresponding to each meta-task is constructed. The support set is used for rapid model adaptation, and the query set is used to evaluate generalization ability. Each meta-task simulates a few-sample learning scenario. The model is pre-trained using the meta-training set, and a set of globally initialized parameters with strong generalization and rapid adaptation capabilities is obtained through a two-layer iterative optimization strategy (the inner layer updates specific task parameters, and the outer layer updates initialization parameters).

[0054] For example, the preprocessed meta-training set is organized as Individual Tasks Each task simulates a new branch office or a new time slot, including a support set and a query set. The training process employs a two-layer loop optimization: in the inner loop, for each sampled meta-task... The model uses the support set of the task to calculate gradients and temporarily updates the model parameters. get In the outer loop, the model uses the updated parameters. Calculate the loss on the query set, and sum the query set losses for all tasks, applying the original initialization parameters. Perform true gradient updates. Through thousands of such meta-iterations, the model does not learn specific sentiment classification rules, but rather a set of initialization parameters that can quickly adapt to any new service hall environment. This allows the model to quickly reach convergence when faced with a completely new and unseen power service hall scenario, requiring only a small number of samples for fine-tuning, significantly improving cross-domain generalization performance and few-sample learning efficiency.

[0055] S140. Input the environmental features and the emotion confidence distribution into the adaptive threshold generation network to obtain the emotion judgment threshold.

[0056] Among them, the adaptive threshold generation network refers to a network that can adaptively adjust the threshold according to the input content. The emotion judgment threshold refers to the threshold used to determine whether a customer's certain emotion is valid. If the confidence level is higher than the emotion judgment threshold, the customer is judged to be in the corresponding emotion; if it is lower, no judgment is made. Each category of emotion has a corresponding emotion judgment threshold.

[0057] In this embodiment, environmental features and emotional confidence distribution can be input into an adaptive threshold generation network to obtain an emotional judgment threshold.

[0058] Traditional classifiers typically use a fixed threshold (e.g., 0.5) to determine the category. However, in open environments like power service halls, noise interference can cause significant fluctuations in the model's prediction confidence for negative emotions, easily leading to false positives, such as misinterpreting a customer's loud inquiry as anger. This invention addresses the high false positive rate of fixed thresholds in noisy environments by designing an Adaptive Threshold Generation Network (ATGN) based on environmental perception. ATGN primarily consists of a lightweight multilayer perception mechanism. Its inputs are an environmental feature vector obtained by concatenating various environmental features, and an emotion confidence distribution. Environmental features include at least the ambient noise level (decibels), ambient light intensity, and background crowd density. The emotion confidence distribution includes the maximum probability value output by the Softmax function and its entropy value. The entropy value reflects the model's uncertainty regarding the current prediction; the higher the entropy, the higher the uncertainty.

[0059] ATGN consists of two hidden layers, using a Rectified Linear Unit (ReLU) activation function, and an output layer using a sigmoid function, outputting a value proportional to the number of sentiment categories. Equal emotion judgment thresholds ATGN learns the nonlinear mapping between environmental states and optimal decision boundaries using a multilayer perceptron, dynamically outputting judgment thresholds for various emotions (such as satisfaction, calmness, anger, and anxiety) for the current sample. When environmental noise is high or model prediction uncertainty is high, the judgment threshold for sensitive emotions (such as anger) is automatically increased to suppress false alarms. For example, the training objective guides ATGN to automatically increase the judgment threshold for the "anger" category (e.g., from 0.5 to 0.8) when high environmental noise (>75dB) is detected and the model prediction entropy is high, thereby suppressing false positive predictions caused by noise and ensuring that alarms are triggered only when the features are extremely significant.

[0060] S150. Based on the emotion confidence distribution, emotion judgment threshold, and emotion label, update the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network using the total loss function, and construct a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

[0061] The total loss function can refer to the function that calculates the loss of the entire network (i.e., the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network). The total loss function can be determined based on the respective loss functions of the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network.

[0062] In this embodiment, the loss value can be calculated based on the emotion confidence distribution, emotion judgment threshold, and emotion label, combined with the total loss function, thereby updating the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network, and constructing a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

[0063] This invention provides a method for training a customer emotion recognition model for power service halls, comprising: acquiring a multimodal dataset of customers in the service hall, wherein each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label; extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map using a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; inputting the high-dimensional emotion feature vector into a meta-learning-based emotion recognition model to obtain an emotion confidence distribution for each sample; inputting the environmental features and the emotion confidence distribution into an adaptive threshold generation network to obtain an emotion judgment threshold; updating the multimodal feature extraction network, the emotion judgment threshold, and the emotion label using a total loss function, and constructing a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition model, and adaptive threshold generation network. This method can train a customer emotion recognition model that can accurately identify customer emotions and is highly adaptable, solving the problem of inaccurate identification of customer emotions in power service halls in existing technologies.

[0064] Based on the above embodiments, modified embodiments of the above embodiments are proposed. It should be noted that, in order to keep the description brief, only the differences from the above embodiments are described in the modified embodiments.

[0065] In one embodiment, updating the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function, includes: calculating the overall total loss of the network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function; updating the parameters of the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the gradient backpropagated from the total loss, to obtain the updated multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network; when the network converges as a whole, using the updated multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network as the trained multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network; otherwise, continuing to iteratively update the overall network.

[0066] In this embodiment, the total loss of the entire network can be calculated using the total loss function based on the emotion confidence distribution, emotion judgment threshold, and emotion label. The gradient is then backpropagated based on the total loss to update the parameters of the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network, resulting in the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network. When the updated network converges, the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network can be used as the trained multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network; otherwise, the overall network continues to be iteratively updated.

[0067] For example, to simultaneously improve the model's feature extraction and threshold decision-making capabilities, and to achieve deep integration between the emotion recognition base model and the adaptive threshold generation network, a collaboratively optimized total loss function was designed to jointly train the aforementioned network model. The total loss function consists of three parts: ; in, The loss is for emotion classification, usually cross-entropy loss, used to supervise the accuracy of the model's emotion classification. For the threshold boundary constraint loss, a variant of hinge loss is employed to penalize predicted probability values ​​falling below the adaptive sentiment judgment threshold. The nearby blurred samples prompt the model to generate more discriminative features; It is a generalization loss unique to meta-learning. It calculates the model's performance on the query set after inner layer updates and is used to measure the model's generalization performance on the query set, forcing the model to learn initial parameters with strong transfer capabilities. For hyperparameter weights, Weights for the emotion classification loss, The weights are the threshold boundary constraint losses. The weights for the generalization loss, unique to meta-learning, can be adjusted through hyperparameter weights to balance the various loss weights. During training, the parameters of the two networks are updated synchronously: each batch samples a batch of tasks from the meta-training set, and performs inner layer adaptation, threshold generation, loss calculation, and backpropagation sequentially. An end-to-end backpropagation is performed using either the Adaptive Moment Estimation (Adam) optimizer or the Stochastic Gradient Descent (SGD) optimizer, synchronously updating the weights of the feature extraction network, the meta-learning initialization parameters, and the adaptive threshold generation network until the model converges.

[0068] This invention enables the model to possess strong cross-scene generalization ability and dynamic environment adaptability by constructing a two-layer collaborative optimization model that integrates meta-learning and adaptive threshold generation mechanisms, and by performing end-to-end offline training using historical data.

[0069] In one embodiment, after obtaining the customer emotion recognition model, the method further includes: acquiring labeled samples collected in the business hall to be deployed; using the labeled samples as a support set to fine-tune the customer emotion recognition model to obtain an adjusted customer emotion recognition model; and deploying the adjusted customer emotion recognition model to the business hall to be deployed.

[0070] Among them, the business hall to be deployed can be the business hall that needs to identify customer emotions, and the labeled samples can be samples collected in the business hall to be deployed that have been manually labeled and preprocessed.

[0071] In this embodiment, labeled samples collected in the target service hall can be obtained. These labeled samples are used as a support set to fine-tune the customer emotion recognition model, resulting in an adjusted model. The adjusted model is then deployed to the service hall. During the actual deployment phase, the model loads the initialization parameters obtained through meta-learning. A small number of labeled samples collected from the target service hall are used as a support set to rapidly fine-tune the model, adapting it to the specific distribution of the current scenario. During inference, the model combines real-time generated adaptive thresholds to perform the final emotion determination on the input audio and video streams, outputting the recognition result. Deploying the offline-trained model to the actual power service hall environment enables real-time, accurate, and adaptive customer emotion recognition, providing service personnel with timely emotional state feedback and early warnings.

[0072] For example, existing surveillance equipment can be installed or reused in the target business hall first, ensuring that the coverage and acquisition quality of the cameras and microphones meet the requirements. Next, an emotion recognition base model and an adaptive threshold generation network model can be deployed on edge computing devices or local servers in the business hall, and the pre-trained network model parameters can be loaded. To adapt to the specificities of the target scene (such as a specific dialect or typical lighting patterns), a small number of recently collected labeled samples from the business hall can be used to construct a support set for the current scene, allowing for one or more steps of gradient fine-tuning of the base model. After fine-tuning, the model possesses high adaptability to the current scene while retaining cross-scene generalization capabilities. After system startup, real-time audio and video streams undergo face detection, audio noise reduction, and spatiotemporal alignment through a preprocessing module, forming a standardized input data stream.

[0073] During online inference, the system operates in real-time or near real-time mode. For each time window, the preprocessed video frame sequence and audio segment are input into a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; the high-dimensional emotion feature vector is then input into the emotion recognition base model to obtain a preliminary emotion softmax confidence distribution. Simultaneously, current environmental sensor data (noise and illumination) and confidence features (entropy and maximum probability) are input into an adaptive threshold generation network, which outputs a dynamic emotion judgment threshold. Comparison of decision-making modules The probability of each category and its corresponding sentiment judgment threshold If it exists and If the confidence score is the maximum among all emotion categories, then output that confidence score. Corresponding emotion tags; if all If all values ​​are below the threshold or multiple values ​​exceed the threshold, an uncertain state is output, taking into account contextual information (such as time-series smoothing). The recognition results, along with the confidence score and environmental parameters, can be pushed to the service personnel's terminal. If a strong negative emotion (such as anger or anxiety) is identified, the system automatically triggers an alert, reminding the duty manager to intervene promptly. The system also supports periodic (e.g., weekly) incremental fine-tuning of the model using newly accumulated labeled data to continuously adapt to environmental changes and evolving user behavior, achieving long-term stable and high-precision emotion perception.

[0074] This invention integrates meta-learning and adaptive thresholding to collaboratively optimize a customer emotion recognition model for power service halls. This method addresses the problems of low accuracy and high false alarm rates in traditional emotion recognition models in power service scenarios, caused by significant environmental noise interference, substantial individual customer differences, and a scarcity of negative emotion samples. By introducing model-independent meta-learning into the multimodal emotion recognition field and constructing task distributions for different service hall scenarios, the model can quickly adapt to new environments or user characteristics with limited samples, thus solving the cold start problem. Combined with an environment-aware adaptive threshold generation network, the emotion classification boundary is dynamically adjusted based on real-time environmental noise levels and model prediction confidence, addressing misjudgment under dynamic noise. Finally, end-to-end joint optimization using a collaborative loss function enables rapid deployment and high-precision recognition of the model in different power service hall scenarios.

[0075] Based on the technical solutions of the above embodiments, this invention provides several specific implementation methods.

[0076] As one specific implementation method of this embodiment. Figure 4 This is a flowchart illustrating a customer emotion recognition model training method for business halls, as provided in an embodiment of the present invention. Figure 4 As shown, this method consists of two main stages: offline training and online inference. In the offline stage, historical multimodal data is used for meta-task construction and co-training to obtain initial model parameters and threshold generation network parameters with strong generalization capabilities. In the online stage, the model is deployed in the target business hall, and a small number of real-time samples are used to quickly fine-tune the model. Dynamic emotion recognition is then performed using real-time generated adaptive thresholds. This method effectively improves the model's cold-start performance in new business halls with scarce data, enhances the model's ability to perceive customer emotions in complex and dynamic environments, reduces false alarm rates, and provides technical support for improving the quality of power services and providing early warning of sudden disputes.

[0077] Example 2 Figure 5 This is a schematic diagram of a customer emotion recognition model training device for a business hall provided in Embodiment 2 of the present invention. The device is applicable to recognizing the emotions of customers in a business hall. The device can be implemented by software and / or hardware and is generally integrated into an electronic device.

[0078] like Figure 5 As shown, the device includes: The acquisition module 210 is used to acquire a multimodal dataset of customers in the business hall. Each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label. Extraction module 220 is used to extract facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; The recognition module 230 is used to input the high-dimensional emotion feature vector into the emotion recognition basic model based on meta-learning to obtain the emotion confidence distribution of each sample; The determination module 240 is used to input the environmental features and the emotion confidence distribution into the adaptive threshold generation network to obtain the emotion determination threshold. The update module 250 is used to update the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function, and to construct a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

[0079] This embodiment provides a training device for a customer emotion recognition model in a business hall. The device includes an acquisition module for acquiring a multimodal dataset of customers in the business hall, where each sample includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label. An extraction module extracts facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map using a multimodal feature extraction network to obtain a high-dimensional emotion feature vector. A recognition module inputs the high-dimensional emotion feature vector into a meta-learning-based emotion recognition model to obtain the emotion confidence distribution for each sample. A judgment module inputs the environmental features and the emotion confidence distribution into an adaptive threshold generation network to obtain an emotion judgment threshold. An update module updates the multimodal feature extraction network, the emotion recognition model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, using a total loss function. The updated multimodal feature extraction network, emotion recognition model, and adaptive threshold generation network are then used to construct a customer emotion recognition model. This device trains a customer emotion recognition model that can accurately identify customer emotions and is highly adaptable, solving the problem that existing technologies cannot accurately identify customer emotions in power service halls.

[0080] Furthermore, module 210 is specifically used for: Data from customers in each business hall is collected to construct structured metadata; each sample in the structured metadata includes customer facial video, conversation audio, environmental metadata, and emotion tags; The customer's facial video and the dialogue audio are preprocessed to obtain a standardized facial image sequence and acoustic feature map; The standardized facial image sequence and acoustic feature map are spatiotemporally aligned based on timestamp alignment technology to obtain the aligned standardized facial image sequence and corresponding acoustic feature map; A multimodal dataset is constructed based on aligned standardized facial image sequences, corresponding acoustic feature maps, environmental metadata, and emotion labels.

[0081] Furthermore, the environmental metadata includes at least one or more of the following: lighting conditions, noise level, customer type, business scenario, timestamp, and environmental sensor data.

[0082] Furthermore, the extraction module 220 is specifically used for: The standardized facial image sequence is input into the visual branch of the multimodal feature extraction network to extract facial visual features and obtain a visual feature vector. The acoustic feature map is input into the speech branch of the multimodal feature extraction network to extract speech acoustic features and obtain a speech feature vector; By employing the multimodal cross-attention mechanism of a multimodal feature extraction network, the fusion coefficient is dynamically adjusted based on the information entropy of the face and speech in the current sample, and the visual feature vector and the speech feature vector are fused to obtain a high-dimensional emotion feature vector.

[0083] Furthermore, the meta-learning-based emotion recognition model includes an inner loop and an outer loop, with a recognition module 230, specifically used for: A meta-training set is constructed based on a multimodal dataset, and the meta-training set is divided into multiple meta-tasks and sub-training sets corresponding to each meta-task; each sub-training set includes a support set and a query set. For each meta-task, in the inner loop, the emotion confidence distribution of the support set is output based on the high-dimensional emotion feature vector corresponding to each sample in the support set of the meta-task, and the gradient is calculated and locally fine-tuned based on the original initialization parameters to obtain the inner model parameters. In the outer loop, the inner model parameters are used to output the sentiment confidence distribution of the query set based on the high-dimensional sentiment feature vector corresponding to the query set of each meta-task. The loss is calculated based on the sentiment confidence distribution to obtain the loss of the meta-task. Based on the loss of each meta-task, the original initialization parameters are updated using gradients to obtain the global initialization parameters.

[0084] Furthermore, update module 250, specifically for: Based on the emotion confidence distribution, emotion judgment threshold, and emotion label, the total loss of the network as a whole is calculated using the total loss function. Based on the backpropagation gradient of the total loss, the parameters of the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network are updated to obtain the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network. When the network converges as a whole, the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network are used as the trained multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network; otherwise, the overall network continues to be iteratively updated.

[0085] Furthermore, after obtaining the customer emotion recognition model, the device is also used for: Obtain labeled samples collected in the business halls where deployment is pending; The labeled samples are used as a support set to fine-tune the customer emotion recognition model, resulting in the adjusted customer emotion recognition model. The adjusted customer emotion recognition model will be deployed to the branch offices where it will be deployed. The aforementioned customer emotion recognition model training device for business halls can execute the customer emotion recognition model training method for business halls provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0086] Example 3 Figure 6 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0087] like Figure 6As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0088] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0089] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the customer emotion recognition model training method for a business hall.

[0090] In some embodiments, the customer emotion recognition model training method for a business hall can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the customer emotion recognition model training method for a business hall described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the customer emotion recognition model training method for a business hall by any other suitable means (e.g., by means of firmware).

[0091] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0092] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0093] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0094] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user, such as a cathode ray tube (CRT) or liquid crystal display (LCD) monitor; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0095] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0096] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0097] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0098] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training a customer emotion recognition model for business halls, characterized in that, The method includes: Obtain a multimodal dataset of customers in the business hall, wherein each sample in the multimodal dataset includes at least a standardized facial image sequence, acoustic feature map, environmental features, and emotion label; A high-dimensional emotion feature vector is obtained by extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network. The high-dimensional emotion feature vector is input into the meta-learning-based emotion recognition model to obtain the emotion confidence distribution of each sample. The environmental features and the emotion confidence distribution are input into an adaptive threshold generation network to obtain the emotion determination threshold. Based on the emotion confidence distribution, emotion judgment threshold, and emotion label, the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network are updated using the total loss function. A customer emotion recognition model is then constructed based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

2. The method according to claim 1, characterized in that, The acquisition of the multimodal dataset of customers in the business hall includes: Data from customers in each business hall is collected to construct structured metadata; each sample in the structured metadata includes customer facial video, conversation audio, environmental metadata, and emotion tags; The customer's facial video and the dialogue audio are preprocessed to obtain a standardized facial image sequence and acoustic feature map; The standardized facial image sequence and acoustic feature map are spatiotemporally aligned based on timestamp alignment technology to obtain the aligned standardized facial image sequence and corresponding acoustic feature map; A multimodal dataset is constructed based on aligned standardized facial image sequences, corresponding acoustic feature maps, environmental metadata, and emotion labels.

3. The method according to claim 2, characterized in that, The environmental metadata includes at least one or more of the following: lighting conditions, noise level, customer type, business scenario, timestamp, and environmental sensor data.

4. The method according to claim 1, characterized in that, The process involves extracting facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map using a multimodal feature extraction network to obtain a high-dimensional emotion feature vector, including: The standardized facial image sequence is input into the visual branch of the multimodal feature extraction network to extract facial visual features and obtain a visual feature vector. The acoustic feature map is input into the speech branch of the multimodal feature extraction network to extract speech acoustic features and obtain a speech feature vector; By employing the multimodal cross-attention mechanism of a multimodal feature extraction network, the fusion coefficient is dynamically adjusted based on the information entropy of the face and speech in the current sample, and the visual feature vector and the speech feature vector are fused to obtain a high-dimensional emotion feature vector.

5. The method according to claim 1, characterized in that, The meta-learning-based emotion recognition model includes an inner loop and an outer loop. The high-dimensional emotion feature vector is input into the meta-learning-based emotion recognition model to obtain the emotion confidence distribution of each sample, including: A meta-training set is constructed based on a multimodal dataset, and the meta-training set is divided into multiple meta-tasks and sub-training sets corresponding to each meta-task; each sub-training set includes a support set and a query set. For each meta-task, in the inner loop, the emotion confidence distribution of the support set is output based on the high-dimensional emotion feature vector corresponding to each sample in the support set of the meta-task, and the gradient is calculated and locally fine-tuned based on the original initialization parameters to obtain the inner model parameters. In the outer loop, the inner model parameters are used to output the sentiment confidence distribution of the query set based on the high-dimensional sentiment feature vector corresponding to the query set of each meta-task. The loss is calculated based on the sentiment confidence distribution to obtain the loss of the meta-task. Based on the loss of each meta-task, the original initialization parameters are updated using gradients to obtain the global initialization parameters.

6. The method according to claim 1, characterized in that, The step of updating the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function, includes: Based on the emotion confidence distribution, emotion judgment threshold, and emotion label, the total loss of the network as a whole is calculated using the total loss function. Based on the backpropagation gradient of the total loss, the parameters of the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network are updated to obtain the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network. When the network converges as a whole, the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network are used as the trained multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network; otherwise, the overall network continues to be iteratively updated.

7. The method according to claim 1, characterized in that, After obtaining the customer emotion recognition model, the method further includes: Obtain labeled samples collected in the business halls where deployment is pending; The labeled samples are used as a support set to fine-tune the customer emotion recognition model, resulting in the adjusted customer emotion recognition model. The adjusted customer emotion recognition model is deployed to the branch office to be deployed.

8. A training device for a customer emotion recognition model in a business hall, characterized in that, The device includes: The acquisition module is used to acquire a multimodal dataset of customers in the business hall. Each sample in the multimodal dataset includes at least a standardized facial image sequence, an acoustic feature map, environmental features, and an emotion label. The extraction module is used to extract facial visual features and speech acoustic features from the standardized facial image sequence and acoustic feature map through a multimodal feature extraction network to obtain a high-dimensional emotion feature vector; The recognition module is used to input the high-dimensional emotion feature vector into the emotion recognition basic model based on meta-learning to obtain the emotion confidence distribution of each sample; The determination module is used to input the environmental features and the emotion confidence distribution into the adaptive threshold generation network to obtain the emotion determination threshold. The update module is used to update the multimodal feature extraction network, the emotion recognition base model, and the adaptive threshold generation network based on the emotion confidence distribution, the emotion judgment threshold, and the emotion label, combined with the total loss function, and to construct a customer emotion recognition model based on the updated multimodal feature extraction network, emotion recognition base model, and adaptive threshold generation network.

9. An electronic device, characterized in that, The device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the customer emotion recognition model training method for a business hall as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the customer emotion recognition model training method for a business hall as described in any one of claims 1-7.

Images