Response strategy generation and execution method and device, equipment and medium

Abstract

The invention relates to the field of artificial intelligence and the field of financial science and technology, and discloses a response strategy generation and execution method, which comprises the following steps of: constructing a multi-task learning structure based on an enhanced spatio-temporal feature extraction model by collecting voice signals of a user and extracting emotional features; sentiment classification, sentiment intensity regression, phoneme recognition and gender recognition are used as joint tasks for training, and a multi-task learning model is obtained. After emotion features are input into the model, emotion category labels and emotion intensity are generated, a strategy table is inquired in combination with a current application scene, and a corresponding response strategy is determined and executed. Through the construction of the multi-task learning model and the joint task training, the emotion category and emotion intensity of the user are accurately identified, and the response strategy is dynamically adjusted in combination with the application scene, so that the emotion perception ability and the personalized service effect of the smart home system are improved, and the user experience and the interaction effect are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and financial technology, and in particular to a response strategy generation and execution method, device, equipment and storage medium. Background Art

[0002] With the development of science and the advancement of technology, smart devices are gradually being integrated into people's daily lives. Artificial intelligence has appeared in every corner of family life, from mobile phones to TVs, from speakers to refrigerators, from sweeping robots to service robots, and people's lives have undergone earth-shaking changes. As one of the most convenient ways for human-computer interaction, voice has been widely used, especially in the field of smart homes, where AI voice technology has gradually become a common way of interaction.

[0003] Although AI voice technology has made significant progress in the application of smart homes, there are still significant deficiencies in emotion recognition and response strategies, which are specifically manifested in the following aspects:

[0004] Existing intelligent voice systems mainly focus on recognizing voice commands and user identities, but are unable to effectively perceive and understand the user's emotional state. When users express emotions such as anxiety, fatigue, or excitement, the system cannot make corresponding adjustments or provide services that match the user's emotions, limiting the improvement of user experience.

[0005] As the existing smart home systems cannot recognize the user's emotional state, their response strategies are simple and passive. For example, when a user orders to turn off the lights in an angry tone, the system will mechanically execute the command without providing additional care or comfort. This lack of emotional perception mechanism fails to fully tap the potential of AI technology in improving the human-computer interaction experience.

[0006] In the field of financial technology, emotion recognition is also of great value. Modern financial institutions' customer service systems have widely used voice interaction technology, but they mainly focus on identity authentication, command recognition and process operation, and cannot accurately capture the emotional state of customers when they consult, complain or make suggestions. For example, when handling customer complaints or confusion, if the bank's customer service system can identify the customer's emotional changes, it will help to adopt more appropriate communication methods and strategies, avoid further escalation of customer dissatisfaction, and improve customer service experience. At the same time, emotion recognition can also be used in market research, customer emotion trend analysis, etc. to provide support for financial decision-making. However, the existing voice interaction system of financial technology lacks the function of emotion recognition, making it difficult to achieve personalized and emotional customer interaction, and failing to make full use of customer emotion data to optimize service quality. Summary of the invention

[0007] The main purpose of the present invention is to provide a response strategy generation and execution method, device, equipment and storage medium, aiming to solve the technical problem that the existing technology cannot effectively identify and understand the user's emotional state, thereby providing targeted and intelligent response strategies.

[0008] To achieve the above object, the present invention provides a response strategy generation and execution method, comprising:

[0009] Collecting a user's voice signal and extracting emotional features from the voice signal;

[0010] Constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0011] Training the multi-task learning structure to obtain a multi-task learning model;

[0012] Inputting the emotion feature into the multi-task learning model, and obtaining the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model;

[0013] The spatiotemporal features are processed by the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and the sentiment intensity is obtained by the sentiment intensity regression module of the multi-task learning model;

[0014] According to the emotion category label, emotion intensity and current application scenario, query the preset strategy table to determine the corresponding response strategy;

[0015] The response strategy is executed.

[0016] Furthermore, to achieve the above-mentioned purpose, the present invention provides a response strategy generation and execution device, comprising:

[0017] A voice collection module is used to collect the user's voice signal and extract emotional features from the voice signal;

[0018] A multi-task learning structure building module, used to build a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0019] A multi-task learning model training module, used to train the multi-task learning structure to obtain a multi-task learning model;

[0020] A feature extraction module, used to input the emotional features into the multi-task learning model, and obtain spatiotemporal features through a shared spatiotemporal feature extraction module of the multi-task learning model;

[0021] A sentiment analysis module, used to process the spatiotemporal features through the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and obtain sentiment intensity through the sentiment intensity regression module of the multi-task learning model;

[0022] A strategy matching module is used to query a preset strategy table and determine a corresponding response strategy according to the emotion category label, emotion intensity and current application scenario;

[0023] A policy execution module is used to execute the response policy.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a response strategy generation and execution program stored in the memory and executable on the processor, and the response strategy generation and execution program, when executed by the processor, implements the steps of the response strategy generation and execution method as described above.

[0025] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium, on which a response strategy generation and execution program is stored, and when the response strategy generation and execution program is executed by a processor, the steps of the response strategy generation and execution method as described above are implemented.

[0026] Beneficial effects: The present invention relates to the fields of artificial intelligence and financial technology, and discloses a response strategy generation and execution method. By collecting the user's voice signal and extracting emotional features, a multi-task learning structure based on an enhanced spatiotemporal feature extraction model is constructed, and emotion classification, emotion intensity regression, phoneme recognition, and gender recognition are trained as joint tasks to obtain a multi-task learning model. After the emotional features are input into the model, the emotion category label and emotion intensity are generated, and the strategy table is queried in combination with the current application scenario to determine and execute the corresponding response strategy. Through the construction of a multi-task learning model and joint task training, the present invention can accurately identify the user's emotion category and emotion intensity, and dynamically adjust the response strategy in combination with the application scenario, thereby improving the emotional perception ability and personalized service effect of the smart home system, and significantly improving the user experience and interactive effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0028] Figure 1 A schematic diagram of an application environment of a response strategy generation and execution method in an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of a flow chart of an embodiment of a method for generating and executing a response strategy of the present invention;

[0030] Figure 3 A schematic diagram of functional modules of a preferred embodiment of the response strategy generation and execution device of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0032] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0034] The response strategy generation and execution method provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can collect the user's voice signal and extract emotional features through the user terminal, construct a multi-task learning structure based on the enhanced spatiotemporal feature extraction model, and train emotion classification, emotion intensity regression, phoneme recognition and gender recognition as joint tasks to obtain a multi-task learning model. After the emotional features are input into the model, the emotion category label and emotion intensity are generated, and the strategy table is queried in combination with the current application scenario to determine and execute the corresponding response strategy. The present invention realizes accurate identification of the user's emotion category and emotion intensity through the construction of a multi-task learning model and joint task training, and dynamically adjusts the response strategy in combination with the application scenario, thereby improving the emotional perception ability and personalized service effect of the smart home system, and significantly improving the user experience and interactive effect. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0035] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a response strategy generation and execution method provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0036] like Figure 2 As shown, the response strategy generation and execution method proposed by the present invention includes the following steps:

[0037] S10, collecting a voice signal of a user, and extracting emotional features from the voice signal;

[0038] In this embodiment, the user's voice signal is collected in real time through the audio acquisition module (such as microphone array) in the smart home device. The audio acquisition module needs to have high sensitivity and noise reduction functions to ensure the quality and accuracy of the signal. With the use of advanced microphone array technology, spatial filtering can be achieved through multi-channel microphones to enhance the target voice signal while suppressing environmental noise. In addition, the audio acquisition module should have an automatic gain control function to cope with volume changes in different environments and ensure that the voice signal is clearly captured.

[0039] The collected speech signals are preprocessed, including noise reduction, silence detection and signal normalization, to improve the accuracy of subsequent feature extraction. Noise reduction can reduce the impact of background noise on emotional feature extraction, silence detection can remove useless silence segments, and signal normalization can balance the dynamic range of the signal. Classic digital signal processing techniques, such as frequency domain filtering and time domain noise reduction algorithms, can be used to remove background noise. Silence detection uses a short-time energy detection method to identify silence segments and remove them. Signal normalization adjusts the signal amplitude to keep its amplitude within a preset range to reduce the impact of amplitude fluctuations.

[0040] The preprocessed speech signal is divided into several short time frames, and a window function (such as Hamming window or Hamming window) is applied to each frame signal to reduce the impact of boundary effects on spectrum analysis. Frame processing is because the speech signal has short-term stability, and window processing is used to smooth the frame boundaries. Usually, the speech signal is divided into short time frames of 20ms to 40ms, and overlapping technology (such as 10ms overlap between frames) is applied to improve the time-frequency resolution. A window function is applied to each frame signal to reduce spectrum leakage and ensure that the frequency domain analysis of the signal is more accurate.

[0041] After framing and windowing, emotional features are extracted from each frame of speech signal. These features include Mel Frequency Cepstral Coefficients (MFCC) and Mel Energy Cepstral Coefficients (MEDC), which can reflect the timbre, pitch and dynamic characteristics of speech signals and are key features for speech emotion recognition. Each frame of speech signal is transformed into the frequency domain by Fast Fourier Transform (FFT), and the spectral information is mapped to the Mel frequency domain using a Mel filter bank to extract MFCC features. At the same time, the logarithm of the spectral energy of each frame and its dynamic changes are calculated to extract MEDC features. The results of feature extraction can be used as input for emotion recognition models.

[0042] Example description: In the field of smart home, the user's life experience can be significantly improved. For example, when a user enters the home and issues a command to the smart speaker through voice, the system can not only understand the user's specific command, but also provide more humane services by analyzing the user's voice emotional state.

[0043] When a user returns home from work, the voice speaker detects that the user's voice carries a sense of fatigue or depression. Based on the extracted emotional features and the identified "tired" emotion, the system can automatically adjust the lights in the home to warm and soft tones and play soothing background music to create a relaxing atmosphere for the user and help relieve fatigue.

[0044] When the user's voice shows joy or excitement, the system can adjust the lights to bright and lively colors and play cheerful music to create a positive and joyful home environment. At the same time, the system can also recommend suitable entertainment activities or information based on the user's emotional state, such as playing the user's favorite entertainment programs or providing interesting interactive functions.

[0045] If the user issues a command with anger or anxiety, the system will not only execute the user's command, but also take appropriate measures based on their emotional state, such as prompting the user to take a deep breath or playing soothing music. Smart home devices can help users relax and provide care and comfort through appropriate emotional response mechanisms.

[0046] Example description: In the intelligent customer service system in the financial field, it can be used to improve the customer service experience. For example, when a user calls a voice customer service and expresses doubts about the loan policy, the system can detect the user's anxiety through emotional feature extraction, and intelligently adjust the tone and speed of the customer service voice, while providing a more detailed explanation or guiding to manual customer service. This emotional perception function can significantly improve customer satisfaction and trust in financial services and reduce customer anxiety caused by unclear information.

[0047] By extracting emotional features and applying multi-task learning models, we can more accurately understand the user's emotional state and provide corresponding response strategies based on the current application scenarios.

[0048] S20, constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0049] In this embodiment, the enhanced spatiotemporal feature extraction model is used to capture the spatiotemporal features in the speech signal, and the enhanced structure can more accurately extract complex information in the speech signal, such as pitch, tone, speaking speed, etc. The spatiotemporal feature extraction model usually includes a multi-layer deep neural network structure, such as a convolutional neural network (CNN) and a long short-term memory network (LSTM), which are used to extract spatial features and temporal features, respectively. The spatial features of the speech signal are extracted using a convolutional neural network (CNN) to identify the local feature structure in the speech; then, the LSTM network is used to capture the time-dependent features of the speech signal to form a comprehensive modeling of the spatiotemporal features of the speech signal. The enhanced structure can improve the expressiveness and training stability of the model by adding residual connections or gated recurrent units (GRU).

[0050] The purpose of using sentiment classification, sentiment intensity regression, phoneme recognition, and gender recognition as joint tasks is to optimize multiple objectives simultaneously in a multi-task learning framework. Sentiment classification is used to identify the type of speech emotion, sentiment intensity regression is used to predict the intensity of emotion, phoneme recognition is used to analyze the basic speech units of speech, and gender recognition is used to distinguish the gender of the speaker. The multi-task learning architecture is designed so that the models share the feature extraction part of the first few layers and then perform independent optimization in the task-specific branches. For example, after sharing the spatiotemporal feature extraction part, different fully connected layers and activation functions are used to handle each task, so as to achieve the recognition of multiple speech features in one model.

[0051] By constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, it is possible to efficiently complete sentiment classification, sentiment intensity regression, phoneme recognition, and gender recognition tasks in the same model, achieve sharing and collaborative optimization of feature extraction, and improve the training efficiency and generalization performance of the model.

[0052] S30, training the multi-task learning structure to obtain a multi-task learning model;

[0053] In this embodiment, when training the multi-task learning structure, a joint loss function needs to be used to balance the optimization effect of each task. The training data set needs to contain labeled emotion, phoneme, and gender information so that the model can learn the corresponding features in each task. A mixed loss function is used, for example, a cross entropy loss function is used for the emotion classification task, a mean square error loss function is used for the emotion intensity regression task, a connection temporal classification (CTC) loss function is used for phoneme recognition, and a cross entropy loss function is used for the gender recognition task. The entire multi-task learning model is optimized by back propagation and gradient descent.

[0054] By constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, sentiment classification, sentiment intensity regression, phoneme recognition, and gender recognition are trained as joint tasks. This not only improves the recognition accuracy of each task, but also effectively shares and utilizes spatiotemporal features to improve the overall performance and efficiency of the model.

[0055] S40, inputting the emotion feature into the multi-task learning model, and acquiring the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model;

[0056] In this embodiment, the multi-dimensional emotional features (e.g., Mel frequency cepstral coefficients MFCC, Mel energy cepstral coefficients MEDC and spectral features) extracted from the user's voice signal are input into the multi-task learning model. These features represent the pitch, timbre, energy changes, etc. of the voice, capturing the dynamic information related to emotion in the voice.

[0057] The shared spatiotemporal feature extraction module of the multi-task learning model is used to process emotional features and generate spatiotemporal features. This module combines the time dimension (such as the change of speech over time) and the frequency dimension (such as the frequency of change of pitch) to transform emotional features into a set of high-level features containing both temporal and frequency domain information.

[0058] Time dimension: Captures short-term and long-term changes in speech signals, such as speech speed and emotional changes. This information is particularly important for judging the intensity of emotions.

[0059] Frequency dimension: Extracts information such as timbre and pitch to help distinguish the frequency domain characteristics of different emotions. For example, an increase in high-frequency components may be related to anger.

[0060] As the common layer of the model, the shared spatiotemporal feature extraction module shares the extracted spatiotemporal features with the sentiment classification module and the sentiment intensity regression module. This sharing mechanism enables the classification and regression tasks of the model to be based on the same spatiotemporal information, thereby improving the overall sentiment recognition accuracy and stability. The role of the shared spatiotemporal feature extraction module is to reduce redundancy, avoid repeated feature extraction, and reduce model complexity; and ensure that the sentiment classification and sentiment intensity regression modules are based on consistent features, thereby obtaining coordinated classification and regression results.

[0061] Example: When a customer representative at a financial service call center interacts with a customer, the system extracts emotional features (such as MFCC, MEDC, etc.) by analyzing the customer's voice signal in real time. The spatiotemporal feature extraction module of the multi-task learning model converts these emotional features into spatiotemporal features, and identifies the customer's emotional state (such as anger, frustration, satisfaction, etc.) and emotional intensity based on these features. Assuming that a customer is angry due to an account problem, the system will identify the customer's high-intensity anger through the spatiotemporal features obtained by the spatiotemporal feature extraction module, triggering a "quick resolution" strategy. For example, the system can automatically assign priorities and remind customer service staff to take more proactive measures. Through spatiotemporal feature extraction, the system can not only identify customer emotions in a single interaction, but also accumulate emotional data over a long period of time and analyze customer emotional trends. For example, the system may detect that certain topics or products trigger more negative emotions among customers, thereby helping financial institutions optimize products or services.

[0062] By sharing the spatiotemporal feature extraction module to generate rich time series and frequency domain information, the multi-task learning model can accurately identify the category and intensity of emotions, thereby improving the accuracy and efficiency of emotion recognition and simplifying the complexity of the model. At the same time, the system can intelligently adjust service strategies based on user emotions and provide personalized responses. This emotion recognition mechanism is suitable for scenarios such as smart homes and financial services. It not only improves user experience, but also provides data support for emotion trend analysis, which helps optimize products and services.

[0063] S50, processing the spatiotemporal features through the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and obtaining sentiment intensity through the sentiment intensity regression module of the multi-task learning model;

[0064] In this embodiment, the multi-task learning model uses a shared spatiotemporal feature extraction part and processes it in the sentiment classification branch and sentiment intensity regression branch respectively. The classification branch uses the Softmax activation function to output the sentiment category label, and the regression branch uses the linear activation function to output the sentiment intensity value. The output layer of the model generates classification and regression results respectively according to the task requirements.

[0065] "Spatiotemporal features" refers to the combination of the time series characteristics of the speech signal and the frequency domain information when the model extracts the user's emotional features. For example, the distribution of changes in pitch, timbre, and energy in the user's voice in the time dimension constitutes the spatiotemporal features. Through the shared spatiotemporal feature extraction module, this information is converted into a set of features that contain time series and frequency relationships.

[0066] Spatiotemporal features are the input basis for the two tasks of sentiment classification and sentiment intensity regression. The model uses these spatiotemporal features to identify the emotional patterns in speech:

[0067] Sentiment classification branch: Use these spatiotemporal features to identify specific emotion patterns, such as "anger" will show the characteristics of increased volume and faster speech speed, while "sadness" may show low volume and slow speech speed. With this spatiotemporal information, the sentiment classification branch can more accurately determine the emotion category.

[0068] Emotion intensity regression branch: Based on the dynamic changes of spatiotemporal features (such as the amplitude of energy changes and the frequency of timbre changes), the regression branch can determine the intensity of emotions. For example, larger energy fluctuations and faster speech speeds correspond to stronger emotions.

[0069] The multi-task learning model outputs emotion category labels and emotion intensity. These output results are used for subsequent decision-making and response strategy selection of the smart home system. The emotion category label represents the user's current emotional state, while the emotion intensity reflects the intensity of the emotion.

[0070] The emotion category label is output through the Softmax function, and the category with the highest probability is selected as the user's emotion state. The emotion intensity is output as a continuous value through the regression model, indicating the strength of the emotion. The output emotion category label and emotion intensity can be further used for personalized services or environmental adjustment.

[0071] By inputting emotional features into the multi-task learning model and outputting the emotional category label and emotional intensity, the user's emotional state and its intensity can be accurately identified, providing a basis for emotional perception for the smart home system. Not only does it improve the accuracy of emotion recognition, it can also help the smart home system adjust the environment settings or provide personalized services according to the user's emotional state, thereby significantly improving the user experience and comfort.

[0072] S60, querying a preset strategy table according to the emotion category label, emotion intensity and current application scenario, and determining a corresponding response strategy;

[0073] In this embodiment, the stored strategy table is accessed using the information of the emotion category label, emotion intensity, and the current application scenario, and the strategy table defines the response strategy combinations under different emotions and scenarios. The strategy table contains predefined response rules so that the appropriate response strategy can be quickly selected based on the input parameters. The strategy table can be implemented using a database or memory data structure (such as a hash table, key-value storage) to improve query efficiency. When the system receives the emotion category label and emotion intensity, it combines the current application scenario as a query condition and quickly locates the corresponding strategy through an index or matching mechanism.

[0074] The emotion category label is used as one of the core inputs to filter out all candidate response strategies related to the emotion in the strategy table. Different emotion categories correspond to different types of strategy responses. For example, "happy" may trigger lively background music, while "sad" may adjust the lights to a warm and soft tone. The strategy table can be classified and indexed according to the emotion category label. When matching, the system retrieves the strategy set related to the identified emotion category from the strategy table and uses it as a candidate strategy for the next step of analysis.

[0075] Emotional intensity is used to further refine the selected response strategies. For example, when the emotional intensity is high, the strategy may be more obvious or positive. For example, in the case of "happy" emotion, high intensity may trigger brighter lights and more dynamic music, while low-intensity emotions will trigger a milder response. According to the emotional intensity standard (such as divided into three levels: low, medium, and high), secondary screening or adjustment is performed in the candidate strategy set to ensure the adaptability and rationality of the response strategy. This strategy adjustment can be achieved through preset rules or weight mechanisms.

[0076] The current application scenario information (such as user location, ambient lighting, background noise, etc.) is used to adjust and optimize the final response strategy. For example, in the bedroom scenario, the strategy is adjusted to ensure comfort and quietness, while in the living room scenario, the response strategy may be more active and open. The current scene data is obtained through the scene recognition module, and it is matched with the scene template in the strategy table to determine the appropriate strategy adjustment parameters. The strategy table can preset response options for different scenarios to ensure the environmental adaptability of the strategy.

[0077] Based on the candidate strategies and their adjustment parameters, the final response strategy is selected and determined, and prepared for execution. The response strategy may include adjusting the settings of smart home devices (such as lighting, temperature, music, etc.) to provide users with a personalized service experience. Use a strategy optimization algorithm or weight evaluation mechanism to select the best solution from the candidate strategies and generate specific execution instructions. For example, based on user historical preferences or strategy weight calculation, the best response solution is selected.

[0078] By querying the preset strategy table and determining the response strategy based on the emotion category label, emotion intensity and current application scenario, it can provide an intelligent, emotional and context-aware home service experience. Compared with the traditional single voice response system, it can dynamically adjust the settings of home appliances according to user emotions and scenarios, provide more intimate and personalized responses, and significantly improve user satisfaction and living comfort.

[0079] S70, executing the response strategy.

[0080] In this embodiment, the response policy obtained from the policy table is parsed to identify specific operation instructions and execution order. The parsing process includes converting high-level policy descriptions into specific executable instructions, such as parameter adjustment for controlling home devices. A rule engine or script parser can be used to parse the descriptive language in the policy table into device control commands. For example, "set the light to warm mode" is parsed into specific instructions for controlling light brightness and color temperature.

[0081] The control module establishes a communication connection with smart home devices (such as lighting systems, audio systems, temperature control devices, etc.) and checks whether the connection is successful. The connection check ensures that the device is ready to receive instructions and perform corresponding operations. Wireless communication protocols (such as Wi-Fi, Bluetooth, Zigbee, etc.) can be used for connection. The control module will send a heartbeat signal or a handshake request to confirm that the device is online and can execute instructions. If the connection fails, the system records the error and prepares to execute the backup strategy.

[0082] If the connection with the device is not successfully established, the backup strategy is executed or a failure notification is sent to the user. The backup strategy may include switching to a backup device or notifying the user to take manual intervention. The system sets up an error handling mechanism. If a connection failure is detected, the backup strategy is automatically activated, such as switching to a backup light or sending a mobile phone notification to remind the user that the device is not responding. The notification can be pushed by the APP or voice reminder.

[0083] After the connection is successfully established, the parsed specific operation instructions are sent to the execution module so that the device can complete the specified task according to the instructions. For example, adjust the brightness of the light, play specific music, set the air conditioning temperature, etc. Use the device control API or protocol to encode the instructions and send them to the target device. After receiving the instructions, the device decodes them and performs the corresponding actions.

[0084] The execution module controls the relevant smart home devices to complete the required operations according to the specific operation instructions received. The system can monitor the completion of the operation and record the status or provide feedback after the execution. The device adjusts its own parameters according to the instructions, such as the lighting system adjusts the color temperature, the audio system plays music, and the air conditioning system adjusts the temperature. If the device supports status feedback, the system can further confirm whether the instruction execution is successful and record the log.

[0085] Example: A bank's customer service center receives a large number of customer calls every day. Customers' needs include account inquiries, transaction records, problem feedback, etc. In these interactions, customers' emotional states may include anger, anxiety, impatience, or joy. However, traditional voice interaction systems can only recognize and process customers' voice commands, but cannot recognize customers' emotional states. This leads to inadequate customer service experience, especially when customers express dissatisfaction or doubts, the system cannot give an appropriate response, which aggravates customer dissatisfaction and affects service quality and customer loyalty.

[0086] Therefore, by introducing emotion recognition technology based on a multi-task learning model, the customer service system can analyze the customer's emotional state during the call in real time through the emotion classification and emotion intensity regression modules, and automatically adjust the response strategy based on the emotion category and emotion intensity.

[0087] During a customer call, the system collects the customer's voice signal in real time and extracts emotional features from it, including Mel Frequency Cepstral Coefficients (MFCC), Mel Energy Cepstral Coefficients (MEDC), etc. These emotional features are used as inputs to the multi-task learning model, and are processed by the spatiotemporal feature extraction module to generate spatiotemporal features, capturing the temporal dynamics and frequency change information of the voice signal, providing a basis for emotion recognition.

[0088] The system inputs the generated spatiotemporal features into the multi-task learning model. The sentiment classification module predicts the customer's sentiment category (such as anger, anxiety, confusion) based on these features, and the sentiment intensity regression module outputs the sentiment intensity (such as mild anger, severe anger). These two modules work together to accurately judge the customer's emotional state and its intensity.

[0089] Based on the identified emotion category and intensity, the system automatically queries and matches the appropriate response strategy from the strategy library. For example:

[0090] Mild dissatisfaction: When the system identifies that the customer's emotion is mildly angry or anxious, the system can choose a softer and more patient tone to respond and give prompts to encourage the customer to continue communicating.

[0091] Severe dissatisfaction: If it is detected that the customer's emotional intensity is high (such as the anger emotional intensity value is higher than the preset threshold), the system will automatically assign it to an experienced customer service representative first, or send a soothing message to remind the customer service staff to pay attention to the customer's emotions.

[0092] Doubt or confusion: When a customer expresses confusion, the system can choose to explain the relevant terms of service in detail, proactively provide further help or relevant links, and guide the customer to complete the operation.

[0093] The emotional data of each customer call is automatically stored in the system to form a customer emotional profile. This data helps analyze the trend of customer emotional changes. For example, the system can analyze that certain service processes or product terms often cause customer dissatisfaction. The bank can optimize these services or terms in a targeted manner to reduce the occurrence of negative customer emotions.

[0094] After daily accumulation of emotion recognition data, the system can analyze the trend of different customers' emotional preferences, thereby providing customers with more personalized services. For example, for customers with large emotional fluctuations, the system can provide customers with more direct service support in advance or proactively remind them of the services they need, thereby enhancing the customer's service experience.

[0095] Emotion recognition enables the customer service system to identify and adapt to the emotional state of customers, providing responses with appropriate tone, rhythm and content, thereby reducing customer dissatisfaction caused by emotional issues and significantly improving customer experience. The system can identify trends in customer emotions and help financial institutions identify service links that are prone to cause customer emotional fluctuations, thereby optimizing these processes, reducing customer emotional fluctuations at the source and improving service efficiency. Based on the analysis of emotional data, the system can provide personalized service recommendations and customized emotional response strategies for different types of customers, creating a smarter service system that better meets customer needs for financial institutions.

[0096] By analyzing and executing response strategies, it is possible to achieve automated and personalized control of smart home devices, providing a more natural and comfortable user experience. Compared with the traditional single-device control mode, it can achieve collaborative control between devices, and has error handling and user reminder functions, which enhances the stability of the system and the consistency of the user experience. When the user's emotions change or the environment changes, the system can adjust in real time and provide home services that meet the user's needs, greatly improving the intelligence level of smart homes and user satisfaction.

[0097] The present invention relates to the field of artificial intelligence technology, and discloses a response strategy generation and execution method. By collecting the user's voice signal and extracting emotional features, a multi-task learning structure based on an enhanced spatiotemporal feature extraction model is constructed, and emotion classification, emotion intensity regression, phoneme recognition, and gender recognition are trained as joint tasks to obtain a multi-task learning model. After the emotional features are input into the model, the emotion category label and emotion intensity are generated, and the strategy table is queried in combination with the current application scenario to determine and execute the corresponding response strategy. Through the construction of a multi-task learning model and joint task training, the present invention can accurately identify the user's emotion category and emotion intensity, and dynamically adjust the response strategy in combination with the application scenario, thereby improving the emotional perception ability and personalized service effect of the smart home system, and significantly improving the user experience and interactive effect.

[0098] In one embodiment, the above S30 includes:

[0099] S301, using a unified sentiment metric standard, dividing the sentiment categories in the original training data into multiple categories, and generating corresponding sentiment category labels;

[0100] S302, numerically representing the emotion intensity in the original training data to generate a corresponding emotion intensity value;

[0101] S303, matching the emotion category label and the emotion intensity value with the spatiotemporal features of the original training data to generate processed training data;

[0102] S304, using the processed training data as input of the multi-task learning structure to train the multi-task learning structure;

[0103] S305, in each training iteration, respectively determining the loss values ​​of the emotion intensity regression task, the emotion classification task, the gender recognition task, and the phoneme recognition task, wherein the emotion intensity regression task adopts a consistency correlation coefficient loss function, the emotion classification task and the gender recognition task both adopt a focus loss function, and the phoneme recognition task adopts a connection temporal classification loss function;

[0104] S306, according to the preset weights, weighted sum of all loss values ​​to generate a total loss value;

[0105] S307, based on the total loss value, using a gradient descent algorithm to optimize and update the parameters of the multi-task learning structure;

[0106] S308, continue iterative training until a preset stop condition is met, and the training is completed and the multi-task learning model is obtained.

[0107] In this embodiment, the emotion categories in the speech data are divided according to a predetermined standard, and corresponding labels are generated as the target of the emotion classification task. These labels are used to guide the classification module of the model to learn the characteristics of different emotion categories.

[0108] Classification criteria: Based on the commonly used classification criteria in emotional psychology, such as happiness, sadness, anger, calmness, etc., ensure that the emotional classification has theoretical basis and consistency.

[0109] Label encoding: One-hot encoding (One-Hot Encoding) or integer encoding (Label Encoding) is performed on the sentiment category label to adapt to the input format of the model classification task.

[0110] Data consistency: Correct sentiment category data, handle noisy labels or inconsistent data annotations, and ensure high quality of model training data.

[0111] The emotion intensity in the original speech data is mapped into numerical form as the target of the emotion intensity regression task. These numerical values ​​represent the intensity of the emotion (such as mild, moderate, and strong), providing learning targets for the regression module. The emotion intensity is divided into different numerical ranges according to the intensity of the speech expression. For example: mild (0.2); moderate (0.5); strong (0.8). The emotion intensity value is normalized so that it falls within the interval [0,1] to facilitate the regression calculation of the model. Samples that do not meet the standards are screened out through manual verification or automated rules to ensure the accuracy of the training data.

[0112] The input features (speech signal’s spatiotemporal features) are matched one by one with the target labels (emotional category labels and emotion intensity values) to construct a complete training dataset as the model input. Multidimensional emotion features (such as Mel-frequency cepstral coefficients MFCC, Mel-energy cepstral coefficients MEDC, and spectral features) are extracted from the speech signal. The spatiotemporal features are normalized (such as mean normalization or standard deviation normalization) to ensure that all features have a consistent distribution range. The feature matrix is ​​aligned with the target label to generate training sample pairs (feature-target) and stored in batches to adapt to the training process.

[0113] Use the processed training data as input to provide training samples for classification and regression tasks for the multi-task learning structure. Organize the training data into input tensors (feature matrices) and output tensors (target matrices) to ensure that they match the input and output interfaces of the multi-task learning structure. Load the training data batch by batch through the data loader, support randomization and data augmentation operations, and improve the generalization performance of training. Pass the input tensor directly to the shared spatiotemporal feature extraction module of the model to provide consistent input features for subsequent tasks.

[0114] The loss value of each task is calculated to measure the difference between the model's predicted value and the true target value, providing guidance for model optimization. The sentiment intensity regression task uses the consistency correlation coefficient loss function to minimize the correlation difference between the predicted value and the target value. The sentiment classification task uses the focus loss function to focus on optimizing the loss value of difficult classification samples. The gender recognition task and the sentiment classification task share the focus loss function to improve the robustness of the classification task. The phoneme recognition task uses the connection temporal classification (CTC) loss function to solve the problem of inconsistent input and output sequence lengths. The loss value of each task is calculated independently to measure its model performance separately.

[0115] The loss values ​​of each task are weighted and summed to generate the total loss value used to optimize the model. Task weights are set according to the importance of the task and the training priority. For example:

[0116] The weight of the sentiment classification task is 0.4; the weight of the sentiment intensity regression task is 0.3; the weight of other tasks is 0.15 (equally divided).

[0117] The total loss value is generated by weighted summation:

[0118] Total loss value = w1·L1+w2·L2+w3·L3+w4·L4;

[0119] Among them, wi is the weight and Li is the task loss value.

[0120] Use the gradient descent algorithm (or its variants) to optimize the model parameters, gradually reduce the total loss value, and improve the model performance. Select an optimization algorithm suitable for the task (such as Adam, RMSprop), and dynamically adjust the learning rate to accelerate convergence. Calculate the gradient of the total loss value to the model parameters through the back propagation algorithm. Adjust the parameter value based on the gradient until the total loss value converges.

[0121] Through multiple rounds of iterative optimization of model parameters, a multi-task learning model with stable performance is finally obtained.

[0122] The training stop condition can be set as: convergence of the total loss value (such as less than a preset threshold); or stable performance of the validation set (such as stable accuracy and regression error).

[0123] Save the converged model weights and structure for subsequent reasoning.

[0124] This embodiment constructs a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, and trains emotion classification, emotion intensity regression, phoneme recognition, and gender recognition as joint tasks, which not only improves the accuracy of emotion recognition, but also effectively improves the generalization ability and adaptability of the model. The system can recognize and respond to the user's emotional state in real time, providing a more natural and humanized interactive experience for smart homes.

[0125] In one embodiment, in the above S10, extracting the emotion feature from the speech signal includes:

[0126] S101, performing frame division and windowing processing on the speech signal to obtain each frame of speech signal segment after the frame division and windowing processing;

[0127] S102, performing a fast Fourier transform on each frame of the speech signal segment to obtain spectrum information of each frame of the speech signal segment;

[0128] S103, using a Mel filter bank to map the spectrum information of each frame of the speech signal segment to the Mel frequency domain, to obtain the spectrum data of each frame of the speech signal segment;

[0129] S104, performing logarithmic transformation and discrete cosine transformation on the energy spectrum of the frequency spectrum data of each frame of the speech signal segment in the Mel frequency domain to obtain Mel frequency cepstrum coefficient features;

[0130] S105, analyzing the dynamic change of logarithmic energy of the spectrum data of each frame of the speech signal segment to obtain Mel energy cepstral coefficient features;

[0131] S106, combining the Mel-frequency cepstral coefficient and the Mel-energy cepstral coefficient features to generate a complete emotion feature.

[0132] In the present embodiment, the speech signal is framed and windowed to segment the continuous speech signal into a plurality of short-time frames. The speech signal can be considered to be stable in a short time, and the framed and windowed processing is helpful for effective frequency domain analysis. The length of each frame is usually 20-40 milliseconds, and there will be a certain overlap (such as 10-20 milliseconds) to retain the continuity of the speech. Windowing (such as Hamming window or Hamming window) is used to smooth the boundary of each frame and reduce the spectrum leakage effect. Framed and windowed are realized using the short-time Fourier transform (STFT) framework. The input speech signal can be frame segmented within a specified time interval, and a window function, such as Hamming window, is applied to each frame to reduce the influence of boundary effects on spectrum analysis.

[0133] Perform a fast Fourier transform (FFT) on each frame of windowed speech signal fragments to convert the time domain signal to the frequency domain to obtain the spectrum information of each frame. The spectrum information can show the energy distribution of the signal at different frequencies and is the basis for subsequent feature extraction. Perform spectrum analysis on each frame of the signal through the FFT algorithm to obtain the amplitude spectrum and phase spectrum. Usually only the amplitude spectrum is used for subsequent processing. FFT can significantly improve computing efficiency and is suitable for real-time sentiment analysis applications.

[0134] The spectral information of each frame is mapped to the Mel frequency domain through a Mel filter bank to simulate the auditory perception of the human ear. The frequency distribution of the Mel frequency domain is more consistent with the auditory characteristics of the human ear and can better capture emotion-related audio features. The spectrum is processed using a set of Mel filters (usually 20-40 triangular filters) to convert the linear frequency scale into a nonlinear Mel frequency scale. The output of each filter represents the energy at a specific Mel frequency, and the Mel frequency energy spectrum of each frame is obtained.

[0135] The Mel frequency domain energy spectrum of each frame is logarithmically transformed to compress the dynamic range and enhance the robustness of the feature. Then the logarithmic energy spectrum is transformed using discrete cosine transform (DCT) to obtain Mel frequency cepstral coefficients (MFCC). MFCC features can effectively represent the timbre characteristics of speech signals and are key parameters for sentiment analysis. First, the logarithm of the energy output of each Mel filter is taken, and then the DCT is used to convert these logarithmic energies to the cepstral domain. Usually only the first few MFCC coefficients are retained to remove high-frequency noise and irrelevant information.

[0136] The dynamic changes of the logarithmic energy of the spectrum data of each frame are analyzed to obtain the Mel Energy Cepstrum Coefficient (MEDC). MEDC reflects the time change of energy in the speech signal and is an important feature used to identify the strength of speech and emotional fluctuations in sentiment analysis. The logarithmic energy of each frame and its first-order and second-order differences are calculated to extract the energy change features. This method can capture the rising and falling trends of energy in speech signals and provide strong support for emotion recognition.

[0137] Mel frequency cepstral coefficient (MFCC) and Mel energy cepstral coefficient (MEDC) features are combined to generate a complete emotional feature vector. This vector can fully describe the emotional information in the speech signal and serve as the input of the emotional classification and regression model. The extracted MFCC and MEDC features are spliced ​​in time series to form a complete emotional feature matrix for subsequent emotional analysis tasks.

[0138] This embodiment can effectively extract Mel-frequency cepstral coefficients and energy dynamic change characteristics related to emotions by performing frame segmentation and windowing processing, spectrum analysis and feature extraction on the speech signal.

[0139] In one embodiment, in the above S10, after collecting the user's voice signal, the process further includes:

[0140] S107, dividing the speech signal into a plurality of short time frames according to a preset frame length;

[0141] S108, analyzing the short-time energy value of each short-time frame, and determining an energy threshold according to the background noise level in the speech signal;

[0142] S109, comparing the short-time energy value of each short-time frame with the energy threshold, if the short-time energy value of a short-time frame is lower than the energy threshold, identifying the short-time frame as a silent segment;

[0143] S110, removing short-time frames identified as silence segments from the speech signal.

[0144] In this embodiment, the collected speech signal is divided into a plurality of short-time frames according to a preset frame length for subsequent analysis. The frame division of the speech signal is based on the assumption of short-time stability. The frame length is usually selected between 20-40 milliseconds, and a certain frame overlap (such as 10-20 milliseconds) is set to ensure the continuity and smoothness between adjacent frames. Through programming or digital signal processing algorithms, the speech signal is divided into a plurality of frames on the time axis, and each frame contains a certain number of sampling points. Frame division can be achieved through a sliding window, and the starting and ending positions of each frame are recorded for subsequent processing.

[0145] The short-time energy value is calculated for each short-time frame. The short-time energy value is used to measure the amplitude strength of the speech signal in the frame, reflecting the loudness or power of the speech. Short-time energy analysis is a basic step in speech signal processing, which is used to identify the difference between speech activity and silence. The method for calculating short-time energy is to sum the square of the amplitude of each frame signal and then take the average. The specific implementation can use simple mathematical operations to add the square of the amplitude value of each sampling point and then divide it by the frame length to get the short-time energy value of the frame.

[0146] An energy threshold is determined based on the background noise level in the speech signal. The energy threshold is used to distinguish between valid speech segments and silent segments. The background noise level can be estimated by analyzing the silent segments in which the noise ratio is relatively large. A noise estimation algorithm can be used or an initial background noise level can be set to dynamically adjust the energy threshold based on the short-term energy value distribution of the silent segment. The threshold determination strategy should be adaptive to adapt to different noise environments.

[0147] The short-time energy value of each short-time frame is compared with the energy threshold. If the energy value of a short-time frame is lower than the threshold, the frame is identified as a silent segment. This process is used to remove invalid segments in the speech signal and reduce unnecessary data processing and analysis burden. Through a simple comparison operation, the frames with energy values ​​lower than the threshold are marked as silent segments, and the start and end positions of these frames are recorded to facilitate the deletion or further processing of silent segments.

[0148] Remove short time frames identified as silence segments from the speech signal to obtain clearer and more useful speech data. Silence segment removal can improve the accuracy and processing efficiency of subsequent emotional feature extraction. According to the label information of silence segments, silence segments are deleted from the original speech signal, and valid speech segments are retained to form a new speech signal sequence. The deletion operation can be implemented using the slicing method of arrays or vectors to ensure the temporal continuity of the speech signal.

[0149] This embodiment can effectively remove invalid silent segments and reduce the impact of background noise on speech processing by performing frame segmentation and windowing processing, short-time energy analysis, energy threshold determination, and silent segment removal on speech signals. It improves the quality of speech signals, reduces data redundancy, and significantly improves the accuracy and efficiency of subsequent emotion feature extraction and emotion recognition, providing a more reliable speech input signal for the smart home system.

[0150] In one embodiment, before the above S60, the method further includes:

[0151] S601, collecting environmental data using a sensor module;

[0152] S602, based on the environmental data, analyzing the light intensity, the ambient temperature, the background noise level and the user's location to obtain a plurality of scene features;

[0153] S603, assigning weights to different scene features according to a preset scene feature standard, and performing a comprehensive analysis based on the scene features and the weights to determine overall environment features;

[0154] S604: Match the overall environment characteristics with a predefined scene template to determine a current application scene.

[0155] In this embodiment, real-time data in the surrounding environment is collected through sensor modules (such as light sensors, temperature sensors, noise sensors and position sensors), and these data are used to analyze and identify the current application scenario. The functions of the sensor module include detecting the ambient light intensity, indoor temperature, background noise and the specific location of the user. A variety of sensors are integrated into the smart home system to collect data regularly or under trigger conditions. The light sensor measures the intensity of the ambient light, the temperature sensor obtains the current indoor or outdoor temperature, the noise sensor captures the background volume level in the environment, and the position sensor (such as GPS or Bluetooth positioning) determines the user's physical location.

[0156] Analyze the collected environmental data and extract specific scene features, including light intensity, ambient temperature, background noise level, and user location. This step is used to convert physical environment information into recognizable scene features, providing basic data for subsequent scene recognition and strategy formulation. Use data processing algorithms to analyze the data output by sensors. For example, light sensor data can be used to determine whether it is day or night, temperature sensors can be used to determine the indoor temperature status, noise sensors can be used to distinguish between quiet or noisy environments, and location sensors can be used to identify the room or area where the user is located.

[0157] According to the preset scene feature criteria, weights are assigned to different scene features, and these weights are applied in the comprehensive analysis to determine the overall environmental characteristics. The scene feature weights determine the importance of each environmental factor in scene recognition. For example, light intensity may have a larger weight in adjusting the lighting scene, while the background noise level has a larger weight when selecting audio content to play. Design a weight assignment algorithm or use a predefined weight standard to assign weights to each scene feature according to the needs of the application scenario. Then, multiply the different features by their respective weights and perform a weighted average or comprehensive analysis to form an overall description of the current environment.

[0158] The overall environmental characteristics obtained through comprehensive analysis are matched with predefined scene templates to determine the current application scenario. A scene template is a set of preset environmental characteristics that are used to identify specific application scenarios, such as "home rest mode", "work mode" or "party mode". Using the template matching algorithm, the current overall environmental characteristics are compared with the templates in the scene library to calculate the matching degree. If the matching degree exceeds a certain threshold, the corresponding scene is determined, and the scene information is recorded or fed back for subsequent strategy use.

[0159] This embodiment can accurately identify and determine the current application scenario by collecting environmental data, analyzing scenario characteristics, assigning weights and performing comprehensive analysis.

[0160] In one embodiment, the above S60 includes:

[0161] S605, based on the current application scenario, filter out a basic strategy table corresponding to the application scenario from a preset strategy table;

[0162] S606, determining the intensity level corresponding to the emotion intensity based on a preset emotion intensity standard;

[0163] S607, determining a plurality of candidate response strategies in the basic strategy table based on the emotion category label and the intensity level;

[0164] S608: Determine an optimal response strategy from the multiple candidate response strategies based on the user's historical preference data.

[0165] In this embodiment, according to the current application scenario, the basic strategy table related to it is screened out from the preset strategy table. The basic strategy table contains the response strategies available in the current scenario, such as lighting settings, sound modes, etc. This step ensures that the response strategy is adapted to the user's current environment. The current application scenario is used as a query condition, and the strategy database is screened to extract the basic strategy table related to the scenario. For example, in the "rest mode" scenario, the basic strategy table may include options for adjusting the lighting to a soft mode and playing soothing music.

[0166] Based on the preset emotion intensity standard, emotion intensity is divided into different levels (such as low, medium, and high) for further screening and optimization of response strategies. Different emotion intensity levels determine the execution intensity and effect of the strategy, such as the volume or the brightness of the light. Standardization is performed according to the emotion intensity value and divided into several levels. For example, a threshold is used to judge the intensity level: emotion intensity between 0-0.3 is low intensity, 0.3-0.7 is medium intensity, and 0.7-1 is high intensity. This level is used to guide the selection and adjustment of subsequent strategies.

[0167] Based on the emotion category label and emotion intensity level, multiple candidate response strategies are screened out from the basic strategy table. The emotion category label (such as "happy", "angry", "calm", etc.) determines the strategy type, while the emotion intensity level affects the specific parameter configuration of the strategy, such as brightness and volume. Using the multi-condition screening mechanism, candidate strategies that match the current emotion and intensity level are screened out from the basic strategy table. For example, if the emotion category is "happy" and the intensity level is "high", the candidate strategies may include turning up the volume to play cheerful music and setting the room lights to bright mode.

[0168] Based on the user's historical preference data, the best strategy is selected from multiple candidate response strategies. This step optimizes the strategy execution based on the user's past choices and feedback, so that the response is more in line with the user's preferences and needs. Using a machine learning model or a weighted scoring mechanism, each candidate strategy is scored based on the user's historical behavior data, and the strategy with the highest score is selected as the best response strategy. For example, if the user has repeatedly selected high-brightness lights and dynamic music under the "happy" emotion in the past, these preferences will be given priority.

[0169] This embodiment enables the smart home system to provide more accurate and personalized emotional response services by querying a preset strategy table according to the emotion category label, emotion intensity and current application scenario, and determining the optimal response strategy in combination with the user's historical preference data.

[0170] In one embodiment, the above S70 includes:

[0171] S701, parsing the response strategy, and identifying specific operation instructions from the response strategy;

[0172] S702, establishing a connection with the execution module through the control module, and checking whether the connection with the execution module is successfully established;

[0173] S703, if the connection with the execution module is not successfully established, executing a preset backup strategy and / or sending a failure prompt notification;

[0174] S704, if the connection with the execution module is successfully established, sending the specific operation instruction to the execution module;

[0175] S705: Execute a corresponding response operation based on the specific operation instruction through the execution module.

[0176] In this embodiment, the response strategy obtained from the preset strategy table needs to be parsed to extract specific operation instructions. The operation instructions may involve specific behaviors of controlling home appliances, such as adjusting lights, changing volume, setting temperature, etc. A strategy parsing module or rule engine is used to convert the descriptive information in the response strategy into an instruction format that the device can understand and execute. For example, "adjust the lights to soft mode" is parsed into specific brightness and color temperature setting parameters.

[0177] Establish a communication connection between the control module of the smart home system and the device execution module, and check whether the connection is successful. This step is used to ensure that the device can receive and execute instructions to avoid the situation where the device is offline or the communication fails. Use standard communication protocols (such as Wi-Fi, Bluetooth or Zigbee) to connect the device and send a heartbeat signal or handshake request to verify the connection status. If the connection is successfully established, the system will receive confirmation feedback from the device; otherwise, the failure status is recorded.

[0178] If the connection cannot be established successfully, the system will execute the preset backup strategy or send a notification to the user to prompt the connection failure. The backup strategy can include using other available devices or delaying the operation. The system sets up an error handling mechanism, and when the connection fails, the backup strategy is automatically triggered, such as using a mobile phone notification or voice to remind the user that the device is unavailable. The backup strategy can be pre-defined and selected according to the importance of the device and user needs.

[0179] If the connection is successfully established, the parsed specific operation instructions are sent to the execution module, which instructs the device to perform specific functions or behaviors. The instructions are sent to the target device through the communication interface. The format of the instructions can be JSON, XML or other standard protocol formats to ensure that the device can correctly receive and parse the instructions. For example, the instruction received by the smart lighting system may be "set the brightness to 50% and the color temperature to 3000K".

[0180] After receiving the instruction, the device execution module completes the corresponding operation, such as adjusting the light brightness, playing music, starting the air conditioner, etc. After the execution is completed, the device can feedback the execution status to the system to ensure the success of the operation and the continuous monitoring of the system. The built-in controller of the device adjusts the settings according to the instruction and returns the status information (such as success, failure or abnormality) to the smart home system. The system can record the execution results and provide feedback to the user when necessary.

[0181] This embodiment can achieve accurate control and timely feedback in smart home devices by analyzing and executing response strategies. The system has the ability to handle device connection failures, can flexibly select backup strategies or remind users, and enhance system reliability and user experience.

[0182] In one embodiment, a response strategy generation and execution device is provided, and the response strategy generation and execution device corresponds one-to-one to the response strategy generation and execution method in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the response strategy generation and execution device of the present invention. Voice collection module 10, multi-task learning structure construction module 20, multi-task learning model training module 30, feature extraction module 40, sentiment analysis module 50, strategy matching module 60 and strategy execution module 70. The functional modules are described in detail as follows:

[0183] The voice collection module 10 is used to collect the user's voice signal and extract the emotional features from the voice signal;

[0184] A multi-task learning structure construction module 20, used to construct a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0185] A multi-task learning model training module 30, used to train the multi-task learning structure to obtain a multi-task learning model;

[0186] A feature extraction module 40, configured to input the emotion feature into the multi-task learning model, and obtain the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model;

[0187] A sentiment analysis module 50, configured to process the spatiotemporal features through a sentiment classification module of the multi-task learning model to obtain a sentiment category label, and obtain sentiment intensity through a sentiment intensity regression module of the multi-task learning model;

[0188] A strategy matching module 60 is used to query a preset strategy table and determine a corresponding response strategy according to the emotion category label, emotion intensity and current application scenario;

[0189] The policy execution module 70 is used to execute the response policy.

[0190] In one embodiment, the multi-task learning model training module 30 is specifically used to:

[0191] Using a unified sentiment metric, the sentiment categories in the original training data are divided into multiple categories and the corresponding sentiment category labels are generated;

[0192] Numerically represent the emotion intensity in the original training data and generate the corresponding emotion intensity value;

[0193] Matching the emotion category label and the emotion intensity value with the spatiotemporal features of the original training data to generate processed training data;

[0194] Using the processed training data as input of the multi-task learning structure to train the multi-task learning structure;

[0195] In each training iteration, the loss values ​​of the emotion intensity regression task, the emotion classification task, the gender recognition task and the phoneme recognition task are determined respectively, the emotion intensity regression task adopts the consistency correlation coefficient loss function, the emotion classification task and the gender recognition task both adopt the focus loss function, and the phoneme recognition task adopts the connection temporal classification loss function;

[0196] According to the preset weights, all loss values ​​are weighted and summed to generate the total loss value;

[0197] Based on the total loss value, using a gradient descent algorithm to optimize and update the parameters of the multi-task learning structure;

[0198] The training is continued iteratively until a preset stopping condition is met, the training is completed and the multi-task learning model is obtained.

[0199] In one embodiment, the voice collection module 10 is specifically used for:

[0200] Performing frame division and windowing processing on the speech signal to obtain each frame of speech signal fragment after the frame division and windowing processing;

[0201] Performing a fast Fourier transform on each frame of the speech signal fragment to obtain the spectrum information of each frame of the speech signal fragment;

[0202] The spectral information of each frame of speech signal fragment is mapped to the Mel frequency domain using a Mel filter bank to obtain the spectral data of each frame of speech signal fragment;

[0203] Perform logarithmic transformation and discrete cosine transformation on the energy spectrum of the spectral data of each frame of speech signal fragment in the Mel frequency domain to obtain Mel frequency cepstrum coefficient features;

[0204] Analyze the dynamic changes of the logarithmic energy of the spectrum data of each frame of the speech signal fragment to obtain the Mel energy cepstral coefficient characteristics;

[0205] The Mel frequency cepstral coefficient and Mel energy cepstral coefficient features are combined to generate a complete emotion feature.

[0206] In one embodiment, the voice collection module 10 is specifically used for:

[0207] Dividing the speech signal into a plurality of short time frames according to a preset frame length;

[0208] Analyzing the short-time energy value of each short-time frame and determining the energy threshold according to the background noise level in the speech signal;

[0209] Comparing the short-time energy value of each short-time frame with the energy threshold, if the short-time energy value of a short-time frame is lower than the energy threshold, identifying the short-time frame as a silent segment;

[0210] Short-time frames identified as silence segments are removed from the speech signal.

[0211] In one embodiment, the policy matching module 60 is specifically configured to:

[0212] Use sensor modules to collect environmental data;

[0213] Based on the environmental data, respectively analyzing the light intensity, the ambient temperature, the background noise level and the user's location to obtain a plurality of scene features;

[0214] According to the preset scene feature standards, weights are assigned to different scene features, and based on the scene features and weights, a comprehensive analysis is performed to determine the overall environment features;

[0215] The overall environmental features are matched with a predefined scene template to determine the current application scene.

[0216] In one embodiment, the policy matching module 60 is specifically configured to:

[0217] Based on the current application scenario, a basic strategy table corresponding to the application scenario is selected from a preset strategy table;

[0218] Based on a preset emotion intensity standard, determining an intensity level corresponding to the emotion intensity;

[0219] Determining a plurality of candidate response strategies in the basic strategy table based on the emotion category label and the intensity level;

[0220] Based on the user historical preference data, an optimal response strategy is determined from the multiple candidate response strategies.

[0221] In one embodiment, the policy execution module 70 is specifically configured to:

[0222] parsing the response strategy and identifying specific operation instructions from the response strategy;

[0223] Establishing a connection with the execution module through the control module, and checking whether the connection with the execution module is successfully established;

[0224] If the connection with the execution module is not successfully established, a preset backup strategy is executed and / or a failure prompt notification is sent;

[0225] If the connection with the execution module is successfully established, the specific operation instruction is sent to the execution module;

[0226] The execution module executes a corresponding response operation based on the specific operation instruction.

[0227] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a response strategy generation and execution method service side.

[0228] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side side of a response strategy generation and execution method

[0229] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0230] Collecting a user's voice signal and extracting emotional features from the voice signal;

[0231] Constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0232] Training the multi-task learning structure to obtain a multi-task learning model;

[0233] Inputting the emotion feature into the multi-task learning model, and obtaining the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model;

[0234] The spatiotemporal features are processed by the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and the sentiment intensity is obtained by the sentiment intensity regression module of the multi-task learning model;

[0235] According to the emotion category label, emotion intensity and current application scenario, query the preset strategy table to determine the corresponding response strategy;

[0236] The response strategy is executed.

[0237] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0238] Collecting a user's voice signal and extracting emotional features from the voice signal;

[0239] Constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task;

[0240] Training the multi-task learning structure to obtain a multi-task learning model;

[0241] Inputting the emotion feature into the multi-task learning model, and obtaining the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model;

[0242] The spatiotemporal features are processed by the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and the sentiment intensity is obtained by the sentiment intensity regression module of the multi-task learning model;

[0243] According to the emotion category label, emotion intensity and current application scenario, query the preset strategy table to determine the corresponding response strategy;

[0244] The response strategy is executed.

[0245] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0246] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0247] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0248] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A response strategy generation and execution method, characterized in that: The following steps are involved: Collecting a user's voice signal and extracting emotional features from the voice signal; Constructing a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task; Training the multi-task learning structure to obtain a multi-task learning model; Inputting the emotion feature into the multi-task learning model, and obtaining the spatiotemporal feature through the shared spatiotemporal feature extraction module of the multi-task learning model; The spatiotemporal features are processed by the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and the sentiment intensity is obtained by the sentiment intensity regression module of the multi-task learning model; According to the emotion category label, emotion intensity and current application scenario, query the preset strategy table to determine the corresponding response strategy; The response strategy is executed.

2. The response strategy generation and execution method according to claim 1, characterized in that: The multi-task learning structure is trained to obtain a multi-task learning model, including: Using a unified sentiment metric, the sentiment categories in the original training data are divided into multiple categories and the corresponding sentiment category labels are generated; Numerically represent the emotion intensity in the original training data and generate the corresponding emotion intensity value; Matching the emotion category label and the emotion intensity value with the spatiotemporal features of the original training data to generate processed training data; Using the processed training data as input of the multi-task learning structure to train the multi-task learning structure; In each training iteration, the loss values ​​of the emotion intensity regression task, the emotion classification task, the gender recognition task and the phoneme recognition task are determined respectively, the emotion intensity regression task adopts the consistency correlation coefficient loss function, the emotion classification task and the gender recognition task both adopt the focus loss function, and the phoneme recognition task adopts the connection temporal classification loss function; According to the preset weights, all loss values ​​are weighted and summed to generate the total loss value; Based on the total loss value, using a gradient descent algorithm to optimize and update the parameters of the multi-task learning structure; The training is continued iteratively until a preset stopping condition is met, the training is completed and the multi-task learning model is obtained.

3. The response strategy generation and execution method according to claim 1, characterized in that: Extracting emotional features from the speech signal includes: Performing frame division and windowing processing on the speech signal to obtain each frame of speech signal fragment after the frame division and windowing processing; Performing a fast Fourier transform on each frame of the speech signal fragment to obtain the spectrum information of each frame of the speech signal fragment; The spectral information of each frame of speech signal fragment is mapped to the Mel frequency domain using a Mel filter bank to obtain the spectral data of each frame of speech signal fragment; Perform logarithmic transformation and discrete cosine transformation on the energy spectrum of the spectral data of each frame of speech signal fragment in the Mel frequency domain to obtain Mel frequency cepstrum coefficient features; Analyze the dynamic changes of the logarithmic energy of the spectrum data of each frame of the speech signal fragment to obtain the Mel energy cepstral coefficient characteristics; The Mel frequency cepstral coefficient and Mel energy cepstral coefficient features are combined to generate a complete emotion feature.

4. The response strategy generation and execution method according to claim 1, characterized in that: After collecting the user's voice signal, it also includes: Dividing the speech signal into a plurality of short time frames according to a preset frame length; Analyzing the short-time energy value of each short-time frame and determining the energy threshold according to the background noise level in the speech signal; Comparing the short-time energy value of each short-time frame with the energy threshold, if the short-time energy value of a short-time frame is lower than the energy threshold, identifying the short-time frame as a silent segment; Short-time frames identified as silence segments are removed from the speech signal.

5. The response strategy generation and execution method according to claim 1, characterized in that: According to the emotion category label, emotion intensity and current application scenario, before searching the preset strategy table and determining the corresponding response strategy, the method further includes: Use sensor modules to collect environmental data; Based on the environmental data, respectively analyzing the light intensity, the ambient temperature, the background noise level and the user's location to obtain a plurality of scene features; According to the preset scene feature standards, weights are assigned to different scene features, and based on the scene features and weights, a comprehensive analysis is performed to determine the overall environment features; The overall environmental features are matched with a predefined scene template to determine the current application scene.

6. The response strategy generation and execution method according to claim 1, characterized in that: According to the emotion category label, emotion intensity and current application scenario, a preset strategy table is queried to determine a corresponding response strategy, including: Based on the current application scenario, a basic strategy table corresponding to the application scenario is filtered out from a preset strategy table; Based on a preset emotion intensity standard, determining an intensity level corresponding to the emotion intensity; Determining a plurality of candidate response strategies in the basic strategy table based on the emotion category label and the intensity level; Based on the user historical preference data, an optimal response strategy is determined from the multiple candidate response strategies.

7. The response strategy generation and execution method according to claim 1, characterized in that: Executing the response strategy includes: parsing the response strategy and identifying specific operation instructions from the response strategy; Establishing a connection with the execution module through the control module, and checking whether the connection with the execution module is successfully established; If the connection with the execution module is not successfully established, a preset backup strategy is executed and / or a failure prompt notification is sent; If the connection with the execution module is successfully established, the specific operation instruction is sent to the execution module; The execution module executes a corresponding response operation based on the specific operation instruction.

8. A response strategy generation and execution device, characterized in that: The response strategy generating and executing device comprises: A voice collection module is used to collect the user's voice signal and extract emotional features from the voice signal; A multi-task learning structure building module, used to build a multi-task learning structure based on an enhanced spatiotemporal feature extraction model, wherein the multi-task learning structure includes a sentiment classification task, a sentiment intensity regression task, a phoneme recognition task, and a gender recognition task; A multi-task learning model training module, used to train the multi-task learning structure to obtain a multi-task learning model; A feature extraction module, used to input the emotional features into the multi-task learning model, and obtain spatiotemporal features through a shared spatiotemporal feature extraction module of the multi-task learning model; A sentiment analysis module, used to process the spatiotemporal features through the sentiment classification module of the multi-task learning model to obtain a sentiment category label, and obtain sentiment intensity through the sentiment intensity regression module of the multi-task learning model; A strategy matching module is used to query a preset strategy table and determine a corresponding response strategy according to the emotion category label, emotion intensity and current application scenario; A policy execution module is used to execute the response policy.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a response strategy generation and execution program stored in the memory and executable on the processor. When the response strategy generation and execution program is executed by the processor, the steps of the response strategy generation and execution method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a response strategy generation and execution program, which, when executed by a processor, implements the steps of the response strategy generation and execution method according to any one of claims 1 to 7.

Images