Multi-dimensional emotion judgment method and device based on robot, equipment and medium

Abstract

The invention relates to a multi-dimensional emotion judgment method and device based on a robot, equipment and a medium, and the method comprises the steps: capturing multi-dimensional behavior data in real time through a preset behavior algorithm, and automatically coordinating the time synchronization of the behavior data of each dimension based on a multi-source data processing framework; performing parallel preprocessing on the multi-dimensional behavior data to obtain standard behavior data, and performing multi-modal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; inputting the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction to obtain an emotional prediction result; dynamically capturing a change signal of an emotional state based on an emotional prediction result, and analyzing an emotional change condition according to the change signal; according to the emotion change condition, a behavior adjusting signal is generated, and a preset adjusting action is executed through the adjusting signal. The method has the effect of improving the emotion judgment accuracy of the robot.

Description

Technical Field

[0001] The present application relates to the technical field of robot emotional interaction, and in particular to a robot-based multi-dimensional emotion judgment method, device, equipment and medium. Background Art

[0002] At present, intelligent robots have been widely used in many fields such as commercial office, family management, psychological diagnosis and treatment, and education. However, most of the existing intelligent robots can only execute simple commands or tasks, and lack the ability to deeply understand and respond to user emotions. As users' requirements for interactive experience increase, the performance of robots in emotional interaction becomes particularly important.

[0003] Most existing robot emotion judgment technologies rely on single-dimensional data, such as judgment through facial expressions or voice emotion recognition, but are easily affected by noise, data loss or individual differences among users, resulting in inaccurate emotion judgment results. At the same time, when existing technologies capture multi-dimensional behavioral data, the time synchronization of data and the real-time nature of data processing are difficult to guarantee, affecting the accurate analysis and judgment of emotional states.

[0004] The above-mentioned existing technical solutions have the following defects: the existing robot emotion judgment technology is difficult to fully and accurately reflect the user's real emotional changes, so there is room for improvement. Summary of the invention

[0005] In order to improve the accuracy of robot emotion judgment, the present application provides a robot-based multi-dimensional emotion judgment method, device, equipment and medium.

[0006] The above-mentioned invention objective of the present application is achieved through the following technical solutions: A robot-based multi-dimensional emotion judgment method, the method comprising: Capture multi-dimensional behavioral data in real time through preset behavioral algorithms, and automatically coordinate the time synchronization of the behavioral data in each dimension based on a multi-source data processing framework; Preprocessing the multi-dimensional behavior data in parallel to obtain standard behavior data, and extracting multimodal features from the standard behavior data using an automatic feature extraction algorithm to generate a behavior feature set; Inputting the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction to obtain an emotional prediction result; Based on the emotion prediction results, dynamically capture the change signal of the emotion state, and analyze the emotion change situation according to the change signal; According to the emotion change, a behavior adjustment signal is generated, and a preset adjustment action is executed through the adjustment signal.

[0007] By adopting the above technical solution, multi-dimensional behavioral data can be captured in real time through a preset behavioral algorithm, and based on a multi-source data processing framework, the time synchronization of each dimensional behavioral data can be automatically coordinated, which can ensure the comprehensiveness and time consistency of multi-dimensional data, thereby improving the system's processing efficiency and accuracy of behavioral data; by preprocessing multi-dimensional behavioral data in parallel, standard behavioral data can be obtained, and multimodal feature extraction can be performed through an automatic feature extraction algorithm, which can quickly extract key information, thereby improving the accuracy of sentiment analysis; by inputting multi-dimensional features into the emotional state model for analysis and prediction, the user's emotional state can be accurately judged, thereby making the robot more humane in its interaction capabilities; by dynamically capturing emotional change signals based on emotional prediction results and analyzing emotional changes, the emotional fluctuations can be grasped in real time, thereby adjusting the robot's behavior in a timely manner, and improving the naturalness and effectiveness of interaction.

[0008] In one example, the present application may be further configured as follows: the multi-dimensional behavior data is captured in real time by a preset behavior algorithm, and the time synchronization of the behavior data in each dimension is automatically coordinated based on a multi-source data processing framework, specifically including: Obtaining the task requirements of the robot, and dynamically adjusting the collection frequency and collection parameters of the behavior data according to the task requirements; The visual, voice, motion and environmental interaction data are captured in real time by the behavior algorithm to obtain multi-dimensional behavior data; Marking the behavior data of each dimension with a timestamp, and performing time alignment on the data based on the timestamp information; Through the data synchronization algorithm, multi-source data collected at different time points are resampled to eliminate the time deviation between the data.

[0009] By adopting the above technical solutions, by obtaining the robot's task requirements and dynamically adjusting the collection frequency and collection parameters of behavioral data according to the task requirements, the system can reasonably optimize data collection in different task scenarios, ensure data accuracy and efficient use of resources, and thus improve the system's adaptability in different tasks; by using behavioral algorithms to capture visual, voice, motion and environmental interaction data in real time, it is possible to fully obtain user behavior information, thereby providing comprehensive data support for emotional judgment; by using timestamps to mark behavioral data of each dimension and perform time alignment, it is possible to ensure synchronous processing of multi-source data, thereby improving the accuracy and consistency of data fusion; by resampling multi-source data through data synchronization algorithms, it is possible to eliminate time deviations between different data, thereby ensuring consistency in the data processing process.

[0010] In one example, the present application may be further configured as follows: performing multimodal feature extraction on the standard behavior data by using an automatic feature extraction algorithm specifically includes: Use convolutional neural networks to extract features from visual data and recognize facial expressions and body gestures; The natural language processing algorithm is used to process the voice data and extract features such as intonation, volume, and speaking speed.

[0011] By adopting the above technical solution, by using convolutional neural networks to extract features from visual data, facial expressions and body postures can be accurately identified, thereby providing reliable visual information for accurate prediction of emotional states; by using natural language processing algorithms to process voice data and extract features such as intonation, volume, and speaking speed, the emotional information in the voice can be effectively captured, thereby enhancing the accuracy of multimodal emotion judgments.

[0012] In one example, the present application may be further configured as follows: the construction of the emotional state model specifically includes: Obtain a historical behavior feature data set, and use a recurrent neural network to process the time series data in the historical behavior feature data set to ensure that the model can capture and analyze emotional changes in the time dimension; The historical behavior feature data set is classified in combination with a support vector machine to generate the emotional state prediction model.

[0013] By adopting the above technical solution, by obtaining historical behavior feature data sets and using recurrent neural networks to process time series data, it is possible to capture the changing trends of user emotions over time, thereby improving the expressiveness of the emotion model in long-term sentiment analysis; by combining support vector machines to classify historical behavior feature data sets, it is possible to effectively construct an emotion state prediction model, thereby providing high-precision model support for real-time emotion judgment.

[0014] In one example, the present application may be further configured as follows: dynamically capturing the change signal of the emotional state based on the emotion prediction result, and analyzing the emotion change situation according to the change signal, specifically including: Based on the emotion prediction results, monitoring changes in the emotional state at consecutive time points to detect abnormal emotion fluctuations; By comparing the current emotional state with the historical emotional state, the scope of emotional change is determined; According to the abnormal mood fluctuation and / or the range of the mood change, the mood change situation is analyzed and constructed.

[0015] By adopting the above technical solution, by monitoring the changes of emotional states at continuous time points based on the emotion prediction results, it is possible to capture emotional fluctuations in a timely manner, ensure real-time response to emotional changes, and thus effectively monitor abnormal emotional fluctuations; by comparing the current emotional state with the historical emotional state, it is possible to accurately judge the scope of emotional changes, thereby providing a basis for further emotional feedback and regulation; by analyzing abnormal emotional fluctuations or the scope of emotional changes, it is possible to comprehensively construct the emotional change situation, thereby providing a decision-making basis for subsequent emotional regulation.

[0016] In one example, the present application may be further configured as follows: monitoring the change of the emotional state at consecutive time points based on the emotion prediction result specifically includes: Through threshold setting and time sliding window, continuous emotional state changes are dynamically monitored to ensure that abnormal fluctuations are detected in a short time.

[0017] By adopting the above technical solution, continuous emotional state changes are dynamically monitored through threshold setting and time sliding windows, which can quickly detect abnormal emotional fluctuations in a short period of time, thereby improving the sensitivity and response speed of emotional fluctuation monitoring.

[0018] In one example, the present application may be further configured as follows: generating a behavior adjustment signal according to the emotion change situation specifically includes: According to the changes in emotions, the preset behavior regulation rules are called to generate specific regulation signals; The regulation signal includes a combination of adjustments to voice, action and expression, ensuring that the robot can make feedback behaviors that are consistent with emotional changes.

[0019] By adopting the above technical solution, by calling the preset behavior regulation rules to generate regulation signals according to the changes in emotions, the system can respond quickly to changes in emotions, thereby ensuring the timeliness and effectiveness of behavior regulation; by adjusting the combination of voice, action and expression by regulation signals, the robot can make feedback behaviors that match the emotional state, thereby enhancing the naturalness of interaction and the richness of emotional expression.

[0020] The second object of the invention is achieved by the following technical solutions: A robot-based multi-dimensional emotion judgment device, the robot-based multi-dimensional emotion judgment device comprising: A behavior capture module is used to capture multi-dimensional behavior data in real time through a preset behavior algorithm, and automatically coordinate the time synchronization of the behavior data in each dimension based on a multi-source data processing framework; A data synchronization module is used to perform parallel preprocessing on the multi-dimensional behavior data to obtain standard behavior data, and to perform multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; A data preprocessing module, used to input the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction, to obtain an emotional prediction result; An emotion analysis module, used to dynamically capture the change signal of the emotion state based on the emotion prediction result, and analyze the emotion change situation according to the change signal; The behavior regulation module is used to generate a behavior regulation signal according to the emotion change, and execute a preset regulation action through the regulation signal.

[0021] By adopting the above technical solution, multi-dimensional behavioral data can be captured in real time through a preset behavioral algorithm, and based on a multi-source data processing framework, the time synchronization of each dimensional behavioral data can be automatically coordinated, which can ensure the comprehensiveness and time consistency of multi-dimensional data, thereby improving the system's processing efficiency and accuracy of behavioral data; by preprocessing multi-dimensional behavioral data in parallel, standard behavioral data can be obtained, and multimodal feature extraction can be performed through an automatic feature extraction algorithm, which can quickly extract key information, thereby improving the accuracy of sentiment analysis; by inputting multi-dimensional features into the emotional state model for analysis and prediction, the user's emotional state can be accurately judged, thereby making the robot more humane in its interaction capabilities; by dynamically capturing emotional change signals based on emotional prediction results and analyzing emotional changes, the emotional fluctuations can be grasped in real time, thereby adjusting the robot's behavior in a timely manner, and improving the naturalness and effectiveness of interaction.

[0022] The third objective of the present application is achieved through the following technical solutions: A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned robot-based multi-dimensional emotion judgment method when executing the computer program.

[0023] The fourth objective of the present application is achieved through the following technical solutions: A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned robot-based multi-dimensional emotion judgment method.

[0024] In summary, this application includes the following beneficial technical effects: 1. Real-time capture of multi-dimensional behavior data through preset behavior algorithms, and automatic coordination of the time synchronization of behavior data of each dimension based on a multi-source data processing framework, can ensure the comprehensiveness and time consistency of multi-dimensional data, thereby improving the system's processing efficiency and accuracy of behavior data; by parallel preprocessing of multi-dimensional behavior data, standard behavior data is obtained, and multi-modal feature extraction is performed through an automatic feature extraction algorithm, which can quickly extract key information, thereby improving the accuracy of sentiment analysis; by inputting multi-dimensional features into the emotion state model for analysis and prediction, the user's emotional state can be accurately judged, thereby making the robot more humane in its interaction capabilities; by dynamically capturing emotion change signals based on emotion prediction results and analyzing emotion changes, the emotion fluctuations can be grasped in real time, thereby adjusting the robot's behavior in a timely manner, and improving the naturalness and effectiveness of interaction; 2. By obtaining the robot's task requirements and dynamically adjusting the collection frequency and collection parameters of behavioral data according to the task requirements, the system can reasonably optimize data collection in different task scenarios, ensure data accuracy and efficient use of resources, and thus improve the system's adaptability in different tasks; through the behavioral algorithm, real-time capture of visual, voice, motion and environmental interaction data can be carried out to fully obtain user behavior information, thereby providing comprehensive data support for emotional judgment; by using timestamps to mark the behavioral data of each dimension and perform time alignment, it can ensure the synchronous processing of multi-source data, thereby improving the accuracy and consistency of data fusion; by resampling multi-source data through data synchronization algorithms, it can eliminate the time deviation between different data, thereby ensuring consistency in the data processing process; 3. By using convolutional neural networks to extract features from visual data, facial expressions and body postures can be accurately identified, thereby providing reliable visual information for accurate prediction of emotional states; by using natural language processing algorithms to process voice data and extract features such as intonation, volume, and speaking speed, the emotional information in the voice can be effectively captured, thereby enhancing the accuracy of multimodal emotion judgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a multi-dimensional emotion judgment method based on a robot in one embodiment of the present application; Figure 2 is a flowchart for implementing step S10 in a robot-based multi-dimensional emotion judgment method in an embodiment of the present application; Figure 3 is a flowchart for implementing step S20 in the robot-based multi-dimensional emotion judgment method in one embodiment of the present application; Figure 4 is a flowchart for implementing step S30 in the robot-based multi-dimensional emotion judgment method in one embodiment of the present application; Figure 5 is a flowchart for implementing step S40 in the robot-based multi-dimensional emotion judgment method in one embodiment of the present application; Figure 6 is a flowchart for implementing step S41 in a robot-based multi-dimensional emotion judgment method in an embodiment of the present application; Figure 7 is a flowchart for implementing step S50 in the robot-based multi-dimensional emotion judgment method in one embodiment of the present application; Figure 8 This is a principle block diagram of a multi-dimensional emotion judgment device based on a robot in one embodiment of the present application; Fig. 9 It is a schematic diagram of a device in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The present application is further described in detail below in conjunction with the accompanying drawings.

[0027] In one embodiment, if Figure 1 As shown, the present application discloses a robot-based multi-dimensional emotion judgment method, which specifically includes the following steps: S10: Capture multi-dimensional behavioral data in real time through preset behavioral algorithms, and automatically coordinate the time synchronization of behavioral data in each dimension based on a multi-source data processing framework.

[0028] Specifically, the robot automatically calls the preset behavior algorithm through the built-in multi-source data acquisition module to capture multi-dimensional behavior data in real time. The behavior data comes from vision, voice, motion and environmental sensors, and can provide comprehensive user status and environmental interaction information. While capturing behavior data, the robot automatically synchronizes the data through the multi-source data processing framework. Each time data is collected, the system will assign a timestamp and align the data based on these timestamps to ensure that all data enter the processing flow synchronously.

[0029] S20: performing parallel preprocessing on the multi-dimensional behavior data to obtain standard behavior data, and performing multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set.

[0030] Specifically, after acquiring multi-dimensional behavioral data, the system immediately starts the parallel processing module to perform parallel preprocessing on different types of data, perform noise filtering and data cleaning on each type of behavioral data, remove invalid or interfering factors, and then standardize these data to ensure that behavioral data of different dimensions are analyzed uniformly under the same standard. The preprocessed standard behavioral data will be used as input, and the behavioral features will be further extracted through the automatic feature extraction algorithm to form a multimodal behavioral feature set for subsequent sentiment analysis and prediction.

[0031] S30: Input the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction to obtain an emotional prediction result.

[0032] Specifically, the generated multimodal behavioral feature set is used as input and passed to the preset emotional state model for analysis. Through multiple data-driven algorithm modules in the model, the input multi-dimensional behavioral characteristics are analyzed in real time. According to the changing trend of the behavioral characteristics, the learning ability of the model is used to predict the emotional state, calculate the user's current emotional state, reflect the user's emotional fluctuations, and provide basic support for subsequent feedback adjustment.

[0033] S40: Based on the emotion prediction result, dynamically capture the change signal of the emotion state, and analyze the emotion change situation according to the change signal.

[0034] Specifically, after the emotional state prediction is completed, the emotional state is dynamically captured and monitored through the emotion monitoring module, and the changes in the emotional state are continuously tracked. Especially when the user shows obvious emotional fluctuations, the change signal can be detected immediately. By real-time analysis of the emotional change signal, the specific circumstances of the emotional change can be determined, including the amplitude, direction and frequency of the change, providing an accurate basis for formulating corresponding feedback strategies.

[0035] S50: Generate a behavior adjustment signal according to the emotion change, and execute a preset adjustment action through the adjustment signal.

[0036] Specifically, based on the emotional changes obtained through analysis and combined with preset behavioral regulation rules, a behavioral regulation signal that matches the current emotional state is generated. The behavioral regulation signal is executed by the control module, and by adjusting regulation actions in multiple dimensions such as voice intonation, facial expressions, and body movements, the robot can respond in accordance with emotional changes, enhance the naturalness and effectiveness of human-computer interaction, continuously monitor the execution of the regulation signal, and make adjustments based on user feedback to ensure the accuracy of the regulation action.

[0037] In one embodiment, if Figure 2 As shown, in step S10, the multi-dimensional behavior data is captured in real time through a preset behavior algorithm, and the time synchronization of the behavior data of each dimension is automatically coordinated based on a multi-source data processing framework, specifically including: S11: Obtain the robot's task requirements and dynamically adjust the collection frequency and collection parameters of the behavior data according to the task requirements.

[0038] Specifically, the task management module identifies the current robot's task requirements, such as user interaction, scene monitoring, dialogue analysis and other different scenarios, and automatically adjusts the collection frequency and collection parameters of behavioral data according to the task type and requirements. For example, in interactive dialogue scenarios, the collection frequency of voice and expression data is increased, while in environmental monitoring tasks, the collection settings of visual and motion sensors are optimized to reduce unnecessary computing overhead and improve the real-time execution of task execution and resource utilization efficiency.

[0039] S12: Use behavioral algorithms to capture visual, voice, motion and environmental interaction data in real time to obtain multi-dimensional behavioral data.

[0040] Specifically, the preset behavioral algorithm is called to capture various types of behavioral data in real time while the robot is performing tasks. The visual sensor captures changes in the user's facial expressions, the voice module collects tone, volume, and speaking speed, the motion sensor records body movements and postures, and the environmental sensor monitors changes in the surrounding environment. The behavioral algorithm integrates these multi-source data to ensure the comprehensiveness and accuracy of the data and generates multi-dimensional behavioral data for subsequent analysis.

[0041] S13: Use timestamps to mark the behavior data of each dimension, and perform time alignment on the data based on the timestamp information.

[0042] Specifically, after each data point is captured, a unique timestamp is immediately assigned to it to ensure that the data from all sensors can be synchronously marked. The timestamp marking enables the system to accurately record the collection moment of each behavioral data, ensuring that data of different dimensions can be processed consistently according to the timeline during analysis. Subsequently, through the time alignment module, data of different dimensions are aligned on the same timeline according to the timestamp information, ensuring the synchronization of multi-source data and providing consistent input for subsequent data fusion and feature extraction.

[0043] S14: Through the data synchronization algorithm, the multi-source data collected at different time points are resampled to eliminate the time deviation between the data.

[0044] Specifically, a data synchronization algorithm is used to process multi-source data with different acquisition frequencies or data loss to ensure the time point consistency of all data. The low-frequency sampling data is interpolated through resampling technology to make it consistent with the high-frequency data. At the same time, the high-frequency sampling data is downsampled to reduce data redundancy.

[0045] In one embodiment, if Figure 3 As shown, in step S20, multimodal feature extraction is performed on the standard behavior data by an automatic feature extraction algorithm, specifically including: S21: Use convolutional neural networks to extract features from visual data and recognize facial expressions and body gestures.

[0046] Specifically, the convolutional neural network model is called to process the visual data through layer-by-layer convolution and pooling operations, and features such as the user's facial expressions and body postures are extracted from the captured image data, such as smiling, frowning, waving and other actions. The convolutional neural network extracts image features layer by layer to form a visual feature map, ensuring that the robot can accurately identify the user's emotions and action states, providing visual support for emotional analysis.

[0047] S22: Use natural language processing algorithms to process voice data and extract features such as intonation, volume, and speech speed.

[0048] Specifically, the collected voice data is deeply analyzed through the natural language processing algorithm, and the basic features of the voice signal such as intonation, volume, and speaking speed are extracted. Then, the emotional expression in the voice content is further analyzed through the semantic analysis module. This process enables the robot to judge the user's emotional tension through changes in volume, or identify the user's anxiety state through changes in speaking speed, providing voice feature support for multimodal emotion judgment.

[0049] In one embodiment, if Figure 4 As shown, in step S30, the construction of the emotional state model specifically includes: S31: Obtain a historical behavioral feature dataset and use a recurrent neural network to process the time series data in the historical behavioral feature dataset to ensure that the model can capture and analyze emotional changes in the time dimension.

[0050] Specifically, we extract labeled behavioral feature data sets from the historical behavior database and input them into a recurrent neural network for processing. This process processes the time dependency of the behavioral data and analyzes the changing trends of user behaviors at different times. By capturing the regularity of emotional changes in time series, we ensure that the emotional state model can provide more accurate predictions based on historical data when processing current behavioral data.

[0051] S32: Combine support vector machine to classify the historical behavior feature data set and generate an emotional state prediction model.

[0052] Specifically, the behavioral features processed by time series are input into the support vector machine classifier, and the behavioral features are further classified into emotions. According to the similarity of the behavioral features, a multidimensional classification hyperplane is constructed, and the behavioral features are classified into different emotional state categories. After multiple training and optimization, an emotional state model that can predict the emotional state in real time is generated for subsequent real-time emotional analysis.

[0053] In one embodiment, if Figure 5As shown, in step S40, based on the emotion prediction result, the change signal of the emotion state is dynamically captured, and the emotion change situation is analyzed according to the change signal, which specifically includes: S41: Based on the emotion prediction results, monitor the changes of the emotion state at consecutive time points and detect abnormal emotion fluctuations.

[0054] Specifically, based on the prediction results of the emotional state model, the changes in emotional state at continuous time points are monitored in real time. By detecting the trend of emotional changes, it is possible to quickly identify whether there are abnormal fluctuations in the emotional state. For example, when it is monitored that the user's emotion changes rapidly from calm to anxious, this fluctuation signal can be captured in real time.

[0055] S42: Determine the scope of the emotion change by comparing the current emotion state with the historical emotion state.

[0056] Specifically, by comparing the current emotional state with the historical emotional state, analyzing the scope and amplitude of the user's emotional fluctuations, using similar emotional states in historical emotional data as a reference, evaluating the severity of the current emotional fluctuations, and by calculating the amplitude and time span of emotional changes, judging whether the emotional changes are short-term fluctuations or long-term emotional changes, providing data support for the generation of emotional regulation strategies.

[0057] S43: Analyze and construct a mood change situation based on the range of abnormal mood fluctuations and / or mood changes.

[0058] Specifically, combined with the amplitude and range of abnormal emotional fluctuations, the emotional changes are further analyzed. The emotional change analysis module constructs a complete emotional change map by calculating parameters such as the frequency, amplitude and duration of emotional fluctuations, and integrates various parameters of emotional fluctuations to ensure that the system has a comprehensive understanding of current emotional changes and provide a basis for the formulation of subsequent regulatory behaviors.

[0059] In one embodiment, if Figure 6 As shown, in step S41, based on the emotion prediction result, monitoring the change of the emotion state at consecutive time points specifically includes: S411: Dynamically monitor continuous emotional state changes through threshold setting and time sliding window to ensure that abnormal fluctuations are detected in a short time.

[0060] Specifically, by setting specific emotion fluctuation thresholds and combining them with time sliding window technology, changes in emotional states are dynamically monitored. The sliding window enables the system to continuously monitor fluctuations in emotional states within multiple short time segments, ensuring timely response when abnormal emotion fluctuations occur. For example, when a user's emotions fluctuate significantly multiple times over a period of time, these abnormal fluctuations can be captured through the sliding window, and feedback adjustments can be made immediately.

[0061] In one embodiment, if Figure 7 As shown, in step S50, a behavior adjustment signal is generated according to the emotion change, specifically including: S51: According to the emotion change, the preset behavior adjustment rules are called to generate specific adjustment signals.

[0062] Specifically, according to the emotion changes obtained by analysis, a preset behavior regulation rule library is called, and the regulation rule library contains regulation schemes for different emotion changes. According to the current emotion fluctuation type and change range, a suitable regulation scheme is selected to generate a specific regulation signal.

[0063] S52: The regulation signal includes the combined adjustment of voice, action and expression to ensure that the robot can make feedback behaviors consistent with emotional changes.

[0064] Specifically, according to the generated adjustment signals, the robot's feedback behaviors in voice, movement and expression are comprehensively adjusted. For example, when the user is in an anxious state, the robot's speaking speed is reduced, the tone is adjusted to a calmer timbre, and soothing movements are performed in body movements.

[0065] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0066] In one embodiment, a robot-based multi-dimensional emotion judgment device is provided, and the robot-based multi-dimensional emotion judgment device corresponds one-to-one to the robot-based multi-dimensional emotion judgment method in the above embodiment. Figure 8 As shown, the robot-based multi-dimensional emotion judgment device includes a behavior capture module, a data synchronization module, a data preprocessing module, an emotion analysis module, and a behavior regulation module. The detailed description of each functional module is as follows: A behavior capture module is used to capture multi-dimensional behavior data in real time through a preset behavior algorithm, and automatically coordinate the time synchronization of the behavior data in each dimension based on a multi-source data processing framework; A data synchronization module is used to perform parallel preprocessing on the multi-dimensional behavior data to obtain standard behavior data, and to perform multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; A data preprocessing module, used to input the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction, to obtain an emotional prediction result; An emotion analysis module, used to dynamically capture the change signal of the emotion state based on the emotion prediction result, and analyze the emotion change situation according to the change signal; The behavior regulation module is used to generate a behavior regulation signal according to the emotion change, and execute a preset regulation action through the regulation signal.

[0067] Optionally, the behavior capture module specifically includes: A task adjustment submodule is used to obtain the task requirements of the robot and dynamically adjust the collection frequency and collection parameters of the behavior data according to the task requirements; A behavior capture submodule, used to capture visual, voice, motion and environmental interaction data in real time through the behavior algorithm to obtain multi-dimensional behavior data; A time-stamping submodule, used to mark the behavior data of each dimension with a timestamp, and to perform time alignment on the data based on the timestamp information; The data resampling submodule is used to resample the multi-source data collected at different time points through a data synchronization algorithm to eliminate the time deviation between the data.

[0068] Optionally, the data synchronization module specifically includes: The visual feature submodule is used to extract features from visual data using convolutional neural networks and recognize facial expressions and body gestures; The speech feature submodule is used to process speech data using natural language processing algorithms and extract features such as intonation, volume, and speech speed.

[0069] Optionally, the construction of the emotional state model specifically includes: The behavior data processing submodule is used to obtain a historical behavior feature data set and process the time series data in the historical behavior feature data set using a recurrent neural network to ensure that the model can capture and analyze emotional changes in the time dimension; The emotion model submodule is used to classify the historical behavior feature data set in combination with a support vector machine to generate the emotion state prediction model.

[0070] Optionally, the sentiment analysis module specifically includes: An emotion monitoring submodule, for monitoring changes in the emotion state at consecutive time points based on the emotion prediction results, and detecting abnormal emotion fluctuations; The emotion comparison submodule is used to determine the range of emotion changes by comparing the current emotion state with the historical emotion state; The emotion change constructing submodule is used to analyze and construct the emotion change situation according to the abnormal emotion fluctuation and / or the range of the emotion change.

[0071] Optionally, the emotion monitoring submodule specifically includes: The emotion dynamic monitoring unit is used to dynamically monitor the continuous emotion state changes through threshold setting and time sliding window to ensure that abnormal fluctuations are detected in a short time.

[0072] Optionally, the behavior regulation module specifically includes: The regulation rule submodule is used to call preset behavior regulation rules according to the change of emotions and generate specific regulation signals; The feedback adjustment submodule is used for the adjustment signal including the combined adjustment of voice, action and expression to ensure that the robot can make feedback behaviors that are consistent with emotional changes.

[0073] For the specific limitations of the robot-based multidimensional emotion judgment device, please refer to the limitations of the robot-based multidimensional emotion judgment method above, which will not be repeated here. Each module in the above-mentioned robot-based multidimensional emotion judgment device can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0074] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a multi-dimensional emotion judgment method based on a robot is implemented.

[0075] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program: Capture multi-dimensional behavioral data in real time through preset behavioral algorithms, and automatically coordinate the time synchronization of behavioral data in each dimension based on a multi-source data processing framework; Perform parallel preprocessing on multi-dimensional behavior data to obtain standard behavior data, and perform multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; Input the multi-dimensional features into the preset emotional state model for analysis and emotional state prediction to obtain the emotional prediction results; Based on the emotion prediction results, dynamically capture the change signals of the emotional state, and analyze the emotion changes according to the change signals; According to the changes in emotions, behavioral adjustment signals are generated, and preset adjustment actions are executed through the adjustment signals.

[0076] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: Capture multi-dimensional behavioral data in real time through preset behavioral algorithms, and automatically coordinate the time synchronization of behavioral data in each dimension based on a multi-source data processing framework; Perform parallel preprocessing on multi-dimensional behavior data to obtain standard behavior data, and perform multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; Input the multi-dimensional features into the preset emotional state model for analysis and emotional state prediction to obtain the emotional prediction results; Based on the emotion prediction results, dynamically capture the change signals of the emotional state, and analyze the emotion changes according to the change signals; According to the changes in emotions, behavioral adjustment signals are generated, and preset adjustment actions are executed through the adjustment signals.

[0077] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0078] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0079] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A robot-based multi-dimensional emotion judgment method, characterized in that: The robot-based multi-dimensional emotion judgment method includes: Capture multi-dimensional behavioral data in real time through preset behavioral algorithms, and automatically coordinate the time synchronization of the behavioral data in each dimension based on a multi-source data processing framework; Preprocessing the multi-dimensional behavior data in parallel to obtain standard behavior data, and extracting multimodal features from the standard behavior data using an automatic feature extraction algorithm to generate a behavior feature set; Inputting the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction to obtain an emotional prediction result; Based on the emotion prediction results, dynamically capture the change signal of the emotion state, and analyze the emotion change situation according to the change signal; According to the emotion change, a behavior adjustment signal is generated, and a preset adjustment action is executed through the adjustment signal.

2. The robot-based multi-dimensional emotion judgment method according to claim 1, characterized in that: The method of capturing multi-dimensional behavior data in real time through a preset behavior algorithm and automatically coordinating the time synchronization of the behavior data in each dimension based on a multi-source data processing framework specifically includes: Obtaining the task requirements of the robot, and dynamically adjusting the collection frequency and collection parameters of the behavior data according to the task requirements; The visual, voice, motion and environmental interaction data are captured in real time by the behavior algorithm to obtain multi-dimensional behavior data; Marking the behavior data of each dimension with a timestamp, and performing time alignment on the data based on the timestamp information; Through the data synchronization algorithm, multi-source data collected at different time points are resampled to eliminate the time deviation between the data.

3. The robot-based multi-dimensional emotion judgment method according to claim 1, characterized in that: The step of extracting multimodal features from the standard behavior data using an automatic feature extraction algorithm specifically includes: Use convolutional neural networks to extract features from visual data and recognize facial expressions and body gestures; The natural language processing algorithm is used to process the voice data and extract features such as intonation, volume, and speaking speed.

4. The robot-based multi-dimensional emotion judgment method according to claim 1, characterized in that: The construction of the emotional state model specifically includes: Obtain a historical behavior feature data set, and use a recurrent neural network to process the time series data in the historical behavior feature data set to ensure that the model can capture and analyze emotional changes in the time dimension; The historical behavior feature data set is classified in combination with a support vector machine to generate the emotional state prediction model.

5. The robot-based multi-dimensional emotion judgment method according to claim 1, characterized in that: The method of dynamically capturing the change signal of the emotional state based on the emotion prediction result and analyzing the emotion change situation according to the change signal specifically includes: Based on the emotion prediction results, monitoring changes in the emotional state at consecutive time points to detect abnormal emotion fluctuations; By comparing the current emotional state with the historical emotional state, the scope of emotional change is determined; According to the abnormal mood fluctuation and / or the range of the mood change, the mood change situation is analyzed and constructed.

6. The robot-based multi-dimensional emotion judgment method according to claim 5, characterized in that: The step of monitoring the change of the emotional state at consecutive time points based on the emotion prediction result specifically includes: Through threshold setting and time sliding window, continuous emotional state changes are dynamically monitored to ensure that abnormal fluctuations are detected in a short time.

7. The robot-based multi-dimensional emotion judgment method according to claim 1, characterized in that: Generating a behavior adjustment signal according to the emotion change specifically includes: According to the changes in emotions, the preset behavior regulation rules are called to generate specific regulation signals; The regulation signal includes a combination of adjustments to voice, action and expression, ensuring that the robot can make feedback behaviors that are consistent with emotional changes.

8. A robot-based multi-dimensional emotion judgment device, characterized in that: The robot-based multi-dimensional emotion judgment device includes: A behavior capture module is used to capture multi-dimensional behavior data in real time through a preset behavior algorithm, and automatically coordinate the time synchronization of the behavior data in each dimension based on a multi-source data processing framework; A data synchronization module is used to perform parallel preprocessing on the multi-dimensional behavior data to obtain standard behavior data, and to perform multimodal feature extraction on the standard behavior data through an automatic feature extraction algorithm to generate a behavior feature set; A data preprocessing module, used to input the multi-dimensional features into a preset emotional state model for analysis and emotional state prediction, to obtain an emotional prediction result; An emotion analysis module, used to dynamically capture the change signal of the emotion state based on the emotion prediction result, and analyze the emotion change situation according to the change signal; The behavior regulation module is used to generate a behavior regulation signal according to the emotion change, and execute a preset regulation action through the regulation signal.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the robot-based multi-dimensional emotion judgment method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the robot-based multi-dimensional emotion judgment method as described in any one of claims 1 to 7 are implemented.

Images