Robot emotion analysis method, system and robot

Abstract

The present invention discloses a robot emotion analysis method, system, and robot. The method comprises: acquiring image data to be analyzed; locating a target area of ​​the image data based on an accelerated image recognition model to obtain a human frame; inputting the human frame and the image data into a sentiment analysis model so that the sentiment analysis model performs a fusion calculation on the human frame and the image data to obtain a first emotion classification result; wherein the sentiment analysis model includes an accelerated human feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model; and controlling the robot's human-computer interaction behavior based on the first emotion classification result. Using embodiments of the present invention, a robot equipped with a sentiment analysis model can accurately, in real time, and rapidly identify a user's emotional information, enabling the robot to better understand the user's conversational intent in human-computer interaction and providing the user with a better communication experience.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robots, and in particular to a robot emotion analysis method, system and robot. Background Art

[0002] Sentiment analysis is the computational study of people's opinions, emotions, feelings, evaluations, and attitudes toward products, services, organizations, individuals, issues, events, topics, and their attributes. Deep learning, a powerful machine learning technique that learns multi-layered features or representations of data, has achieved remarkable results in many application areas. Its branches include computer vision, speech recognition, and natural language processing (NLP). In recent years, text-based natural language processing (NLP) and image-based computer vision have been widely used in sentiment analysis tasks.

[0003] Current sentiment analysis methods primarily focus on analyzing facial expressions and body posture. However, when people are in various scenarios, facial expressions alone are insufficient to identify a person's emotional state. Furthermore, sentiment analysis models that integrate human features and scene information are slow when deployed on robots due to hardware computing limitations. Summary of the Invention

[0004] The present invention provides a robot emotion analysis method and system to solve the technical problem of slow operation speed of the robot emotion analysis model.

[0005] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides a robot emotion analysis method, comprising:

[0006] Acquiring image data to be analyzed;

[0007] Locating the target area of ​​the image data according to the accelerated image recognition model to obtain a human body frame;

[0008] Inputting the human body frame and the image data into a sentiment analysis model so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result; wherein the sentiment analysis model includes: an accelerated human body feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model;

[0009] Control the robot's human-computer interaction behavior based on the first emotion classification result.

[0010] The present invention uses an accelerated image recognition model to identify the subject from image data and output a human frame. The sentiment analysis model then performs a fusion calculation based on the human frame and image data to produce a first sentiment classification result. In addition to performing sentiment analysis based on the human body, the present invention also performs a fusion calculation based on the image data combined with the human frame, ensuring that the final sentiment classification result takes into account the impact of scene factors on the subject's emotion recognition. Furthermore, the models used in both the image recognition and fusion calculation stages of the present invention are accelerated, resulting in faster computational speeds.

[0011] Furthermore, before acquiring the image data to be analyzed, the process includes:

[0012] Load the first image data and the first label data in the COCO dataset and normalize the first image data and the first label data; initialize the first backbone network, the feature pyramid network, and the first fully connected network of the YOLOv3 model to obtain an initialized YOLOv3 model;

[0013] Inputting the first image data into the initialized YOLOv3 model to obtain a three-dimensional tensor embedding;

[0014] Selecting a first loss function to calculate the loss according to the first label and the three-dimensional tensor embedding, and obtaining a first parameter gradient through back propagation;

[0015] Parameters are adjusted according to the first parameter gradient until the initialized YOLOv3 model converges to obtain the image recognition model.

[0016] Furthermore, before acquiring the image data to be analyzed, the method further includes:

[0017] Load the second image data and second label data from the EMOTIC dataset, load the human feature extraction model and the scene feature extraction model, normalize the second image data and the second label data, and initialize the second backbone network and the second fully connected network of the baseline CNN model; wherein, the first ResNet50 model pre-trained with the ImageNet dataset is used as the human feature extraction model, and the second ResNet50 model pre-trained with Places365 is used as the scene feature extraction model;

[0018] The human feature extraction model extracts a second human feature of the second image data, the scene feature extraction model extracts a second scene feature of the second image data, and the baseline CNN model fuses the human feature and the scene feature to obtain a second emotion classification result; wherein the second emotion classification result includes a second continuous emotion classification result and a second discrete emotion classification result;

[0019] Calculating the loss using a weighted square error loss function based on the second label data and the second discrete sentiment classification result, and obtaining a second parameter gradient through back propagation; calculating the loss using an L1 loss function based on the second label data and the second continuous sentiment classification result, and obtaining a third parameter gradient through back propagation;

[0020] Parameters are adjusted according to the second parameter gradient and the third parameter gradient until the baseline CNN model converges to obtain the fusion classification model.

[0021] Before using the image recognition model and sentiment analysis model, the present invention pre-trains both models. The COCO dataset is used as the training set for the image recognition model. A first loss function and parameter adjustment are used to converge the image recognition model. The resulting image recognition model is able to more accurately select the human frame in the target area. A baseline CNN model fuses the human features and scene features to produce a second sentiment classification result, which is further divided into a second discrete sentiment classification result and a second continuous sentiment classification result. This classification method is more detailed than single discrete sentiment classification and more accurately identifies the subject's emotions.

[0022] Furthermore, the accelerated image recognition model, the accelerated human feature extraction model, the accelerated scene feature extraction model and the accelerated fusion classification model are specifically:

[0023] Confirm the CUDA version of the robot host that stores the image recognition model, the human feature extraction model, the scene feature extraction model, and the fusion classification model, and load the tensorRT optimizer corresponding to the CUDA version;

[0024] Convert the image recognition model to an image recognition model in ONNX format; convert the human feature extraction model to a human feature extraction model in ONNX format; convert the scene feature extraction model to a scene feature extraction model in ONNX format; convert the fusion classification model to a fusion classification model in ONNX format;

[0025] Convert the image recognition model in ONNX format to an image recognition model in tensorRT format; convert the human feature extraction model in ONNX format to a human feature extraction model in tensorRT format; convert the scene feature extraction model in ONNX format to a scene feature extraction model in tensorRT format; convert the fusion classification model in ONNX format to a fusion classification model in tensorRT format;

[0026] Deserialize the image recognition model in the tensorRT format, the human feature extraction model in the tensorRT format, the scene feature extraction model in the tensorRT format, and the fusion classification model in the tensorRT format, and create a runtime engine;

[0027] The accelerated image recognition model, the accelerated human feature extraction model, the accelerated scene feature extraction model and the accelerated fusion classification model are obtained.

[0028] The present invention uses the tensorRT optimizer to accelerate the models used in the sentiment analysis process, converting these models into the universal ONNX format, then converting the ONNX format model into the tensorRT format, and creating a runtime engine after deserialization; this method can convert the image recognition model, human feature extraction model, scene feature extraction model, and fusion classification model used in the present invention into a format that occupies less storage space and requires fewer resources for calculation, thereby improving the model's calculation speed.

[0029] Furthermore, the target area of ​​the image data is located according to the accelerated image recognition model to obtain a human body frame, specifically:

[0030] Reading a video frame of the image data;

[0031] Cropping the video frame to a size of a preset resolution;

[0032] Inputting the cropped video frame into the backbone module of the backbone network of the accelerated image recognition model, and extracting features through the feature pyramid network of the accelerated image recognition model;

[0033] The bounding box coordinates, scores, and multiple human body frames of the human body region are output according to the features, and after performing a non-maximum suppression operation on the multiple human body frames, the human body frame with the highest confidence is selected.

[0034] Furthermore, the human body frame and the image data are input into a sentiment analysis model so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result, specifically:

[0035] Inputting the human body frame into the accelerated human body feature extraction model, and inputting the image data into the scene feature extraction model;

[0036] The accelerated human feature extraction model extracts a first human feature of the human frame, and the scene feature extraction model extracts a first scene feature of the image data;

[0037] The accelerated fusion classification model fuses the first human features and the first scene features to obtain a first emotion classification result.

[0038] Furthermore, the first emotion classification result includes: a first discrete emotion classification result and a first continuous emotion classification result;

[0039] The first discrete emotion classification results include one of the following: calm, gentle, expectant, respectful, confident, interested, happy, excited, surprised, frustrated, hostile, jealous, tired, painful, hesitant, fearful, hurt, worried, annoyed, angry, disgusted, sad, bored, or embarrassed;

[0040] The first continuous emotion classification results include: happiness level, excitement level, and control level, wherein each first continuous emotion classification result takes an integer value within a preset range.

[0041] The accelerated fusion classification model fuses the human body features and the scene features to obtain a first emotion classification result, which is further divided into a first discrete emotion classification result and a first continuous emotion classification result. This classification method is more detailed than a single discrete emotion classification and can more accurately identify the subject's emotions.

[0042] In a second aspect, an embodiment of the present invention provides a robot emotion analysis system, comprising: an image acquisition module, a first control module, a second control module, and an interactive function module;

[0043] The image acquisition module is used to acquire image data to be analyzed;

[0044] The first control module is used to locate the target area of ​​the image data according to the accelerated image recognition model to obtain a human body frame;

[0045] The second control module is used to input the human body frame and the image data into the sentiment analysis model, so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result; wherein the sentiment analysis model includes: an accelerated human body feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model;

[0046] The interactive function module is used to control the human-computer interaction behavior of the robot according to the first emotion classification result.

[0047] Furthermore, the second control module includes: a data input unit, a feature extraction unit and a fusion classification unit;

[0048] The data input unit is used to input the human body frame into the accelerated human body feature extraction model, and input the image data into the scene feature extraction model;

[0049] The feature extraction unit is used to extract the first human feature of the verification body frame using the accelerated human feature extraction model, and the scene feature extraction model is used to extract the first scene feature of the image data;

[0050] The fusion classification unit is used to fuse the first human features and the first scene features with the accelerated fusion classification model to obtain a first emotion classification result.

[0051] In a third aspect, an embodiment of the present invention provides a robot, comprising: a central controller, an image acquisition device, and a human-computer interaction device;

[0052] The central controller is configured to execute any one of the robot emotion analysis methods described in the embodiments of the present invention;

[0053] The image acquisition device is used to acquire image data to be analyzed and send the image data to the central controller;

[0054] The human-computer interaction device is used to perform human-computer interaction behavior according to the control instructions of the central controller.

[0055] The present invention uses an accelerated image recognition model to identify the subject from image data and output a human frame. The sentiment analysis model then performs a fusion calculation based on the human frame and image data to produce a first sentiment classification result. In addition to performing sentiment analysis based on the human body, the present invention also performs a fusion calculation based on the image data combined with the human frame, ensuring that the final sentiment classification result takes into account the impact of scene factors on the subject's emotion recognition. Furthermore, the models used in both the image recognition and fusion calculation stages of the present invention are accelerated, resulting in faster computational speeds. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an embodiment of the robot emotion analysis method provided by the present invention;

[0057] Figure 2 This is a flowchart of an embodiment of image recognition model pre-training provided by the present invention;

[0058] Figure 3 This is a flowchart of an embodiment of the sentiment analysis model pre-training provided by the present invention;

[0059] Figure 4 This is a flow chart of an embodiment of model acceleration provided by the present invention;

[0060] Figure 5 1 is a flow chart of another embodiment of the robot emotion analysis method provided by the present invention;

[0061] Figure 6 1 is a schematic structural diagram of an embodiment of the robot emotion analysis system provided by the present invention;

[0062] Figure 7 1 is a schematic structural diagram of another embodiment of the robot emotion analysis system provided by the present invention;

[0063] Figure 8 1 is a schematic structural diagram of an embodiment of a robot provided by the present invention; DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0065] Example 1

[0066] Please refer to Figure 1 , is a flow chart of an embodiment of the robot emotion analysis method provided by the present invention, which mainly includes steps 101 to 104, as follows:

[0067] Step 101: Obtain image data to be analyzed.

[0068] Step 102: Locate the target area of ​​the image data according to the accelerated image recognition model to obtain a human body frame.

[0069] In this embodiment, the deep learning model used for image recognition is the YOLOv3 model. YOLOv3 is the third version of the YOLO (You Only Look Once) series of object detection algorithms. It has fast inference speed and can accurately identify small objects. The image recognition model is used to detect and locate human body regions and obtain human body frames.

[0070] In this embodiment, the target area of ​​the image data is located according to the accelerated image recognition model to obtain a human body frame, specifically by: reading the video frame of the image data; cropping the video frame to a preset resolution size; inputting the cropped video frame into the backbone module of the backbone network of the accelerated image recognition model, extracting features through the feature pyramid network of the accelerated image recognition model; outputting the bounding box coordinates, scores, and multiple human body frames of the human body area according to the features, and performing a non-maximum suppression operation on the multiple prior frames, and selecting the human body frame with the highest confidence. Among them, the preset resolution can be selected as 640*640px or 320*320px; when performing the non-maximum suppression operation, the multiple human body frames are sorted according to the confidence, the human body frame with the highest confidence is screened out, and other human body frames in the neighborhood of the human body frame with the highest confidence are screened out.

[0071] Step 103: Input the human body frame and the image data into a sentiment analysis model, so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result; wherein the sentiment analysis model includes: an accelerated human body feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model.

[0072] In this embodiment, the sentiment analysis model combines environmental information in the image data with information about the person in the human frame to identify the person's emotions. After receiving the human frame and image data, the sentiment analysis model extracts features from each, performs a fusion calculation on these features, and generates a first sentiment classification result.

[0073] In this embodiment, the first emotion classification result includes: a first discrete emotion classification result and a first continuous emotion classification result; the first discrete emotion classification result includes: one of: peace, gentleness, expectation, respect, confidence, interest, happiness, excitement, surprise, frustration, hostility, jealousy, fatigue, pain, hesitation, fear, injury, worry, annoyance, anger, disgust, sadness, boredom or embarrassment; the first continuous emotion classification result includes: happiness level, excitement level and control level, wherein each first continuous emotion classification result takes an integer value within a preset range.

[0074] Step 104: Control the human-machine interaction behavior of the robot according to the first emotion classification result.

[0075] In this embodiment, the robot can realize human-computer interaction through movement, voice, and expression.

[0076] In this embodiment, an accelerated image recognition model identifies the subject from the image data and outputs a human frame. The sentiment analysis model then performs a fusion calculation based on the human frame and image data to produce a first sentiment classification result. In addition to performing sentiment analysis based on the human body, the present invention also performs a fusion calculation based on the image data combined with the human frame, allowing the final sentiment classification result to take into account the impact of scene factors on the subject's emotion recognition. Furthermore, the models used in both the image recognition and fusion calculation stages of the present invention are accelerated, resulting in faster computation speeds.

[0077] Please refer to Figure 2 , is a flowchart of an embodiment of image recognition model pre-training provided by the present invention, which mainly includes steps 201 to 204, as follows:

[0078] In this embodiment, steps 201 to 204 are completed before step 101 .

[0079] Step 201: Load the first image data and the first label data in the COCO dataset, and normalize the first image data and the first label data; initialize the first backbone network, the feature pyramid network, and the first fully connected network of the YOLOv3 model to obtain an initialized YOLOv3 model.

[0080] In this example, the training dataset is the MS COCO dataset, a large-scale object detection, segmentation, and captioning dataset designed for scene understanding. The images are primarily captured from complex everyday scenes, and the locations of objects within the images are precisely segmented and calibrated. The dataset includes 91 target categories, including common objects found in daily life. Using this dataset to train the YOLOv3 object detection model enables the model to achieve good generalization and accuracy.

[0081] Step 202: Input the first data into the initialized YOLOv3 model to obtain a three-dimensional tensor embedding.

[0082] In this embodiment, after obtaining the first data, the YOLOv3 model outputs three 3D tensor embeddings of different sizes, corresponding to three different sizes, and each 3D tensor embedding includes: a classification score, a bounding box coordinate, and a predicted category.

[0083] Step 203: A first loss function is selected to calculate the loss based on the first label and the three-dimensional tensor embedding, and a first parameter gradient is obtained through back propagation.

[0084] Step 204: Adjust parameters according to the first parameter gradient until the initialized YOLOv3 model converges to obtain the image recognition model.

[0085] In this embodiment, before using the image recognition model and the sentiment analysis model, the models used are pre-trained, wherein the COCO dataset is used as the training set of the image recognition model, and the image recognition model is converged through the first loss function and adjustment parameters. The image recognition model finally used can more accurately select the human body frame of the target area.

[0086] Please refer to Figure 3 , is a flowchart of an embodiment of sentiment analysis model pre-training provided by the present invention, which mainly includes steps 301 to 304, as follows:

[0087] In this embodiment, steps 301 to 304 are completed before step 101 .

[0088] Step 301: Load the second image data and second label data in the EMOTIC dataset, load the human feature extraction model and the scene feature extraction model, normalize the second image data and the second label data, and initialize the second backbone network and the second fully connected network of the baseline CNN model; wherein, the first resnet50 model pre-trained with the ImageNet dataset is used as the human feature extraction model, and the second resnet50 model pre-trained with places365 is used as the scene feature extraction model.

[0089] In this embodiment, the training set for the baseline CNN model is the EMOTIC dataset, which is a collection of images labeled according to people's obvious emotional states. The scenes in the images are natural and show people doing different things in different environments. The human feature extraction model uses the ResNet50 model, which takes the target human body as input and generates human features. These features include: face, head, posture, and body appearance. To capture these aspects, the human feature extraction model is pre-trained using the ImageNet dataset, which is an object-centric dataset. The scene feature extraction model also uses the ResNet50 model, which takes the entire image data as input and generates scene features; these scene features can be interpreted as attributes and objects appearing in the scene. To capture these scene features, the scene feature extraction model is pre-trained using the Places365 dataset.

[0090] Step 302: The human feature extraction model extracts the second human feature of the second image data, the scene feature extraction model extracts the second scene feature of the second image data, and the baseline CNN model fuses the human feature and the scene feature to obtain a second emotion classification result; wherein, the second emotion classification result includes a second continuous emotion classification result and a second discrete emotion classification result.

[0091] In this embodiment, the baseline CNN model flattens and concatenates the feature maps obtained by the first two human feature extraction models and the scene feature extraction model into a 1024-dimensional vector, and then passes through two fully connected layers to finally output a 26-dimensional vector and a 3-dimensional vector. The baseline CNN model combines the features of the two feature extraction modules to estimate discrete emotions and continuous emotions. The EMOTIC dataset combines two different types of emotion categories: 26 discrete emotions and 3 continuous emotions. The scene feature extraction model further divides the identified scene information into three scene types: positive, negative, and normal. At the same time, the 26 discrete emotions are artificially divided into three scene types. Each set contains multiple discrete emotion categories corresponding to the type of scene information. When finally obtaining the emotion classification results, the discrete emotion categories in the emotion type corresponding to the scene are given priority.

[0092] Step 303: Calculate the loss using the weighted square error loss function based on the second label data and the second discrete sentiment classification result, and obtain the second parameter gradient through back propagation; calculate the loss using the L1 loss function based on the second label data and the second continuous sentiment classification result, and obtain the third parameter gradient through back propagation.

[0093] In this embodiment, for 26 discrete sentiment classification tasks, the weighted square error loss function is used to calculate the loss; for 3 continuous sentiment classification and regression tasks, the L1 loss function is used to calculate the loss.

[0094] Step 304: Adjust parameters according to the second parameter gradient and the third parameter gradient until the CNN baseline model converges to obtain the fusion classification model.

[0095] In this embodiment, the baseline CNN model fuses the human body features and the scene features to obtain a second emotion classification result, which is further divided into a second discrete emotion classification result and a second continuous emotion classification result. This classification method is more detailed than a single discrete emotion classification and can more accurately identify the emotions of the subject.

[0096] Please refer to Figure 4 , is a flow chart of an embodiment of model acceleration provided by the present invention, which mainly includes steps 401 to 405, as follows:

[0097] Step 401: Confirm the CUDA version of the robot host that stores the image recognition model, the human feature extraction model, the scene feature extraction model, and the fusion classification model, and load the tensorRT optimizer corresponding to the CUDA version.

[0098] In this embodiment, the image recognition model and sentiment analysis model are deployed on intelligent robot hosts such as NX and AGX. These hosts are equipped with a Linux system to develop and call the robot's underlying control packages. However, deep model inference on these devices is relatively inefficient, so TensorRT acceleration of the model is required to ensure that the inference speed on the edge hardware device is close to that on the computer host. TensorRT is a deep learning inference optimizer that optimizes trained models and provides low-latency, high-throughput deployment and inference for deep learning applications.

[0099] Step 402: Convert the image recognition model to an image recognition model in ONNX format; convert the human feature extraction model to a human feature extraction model in ONNX format; convert the scene feature extraction model to a scene feature extraction model in ONNX format; and convert the fusion classification model to a fusion classification model in ONNX format.

[0100] Step 403: Convert the image recognition model in ONNX format to an image recognition model in tensorRT format; convert the human feature extraction model in ONNX format to a human feature extraction model in tensorRT format; convert the scene feature extraction model in ONNX format to a scene feature extraction model in tensorRT format; convert the fusion classification model in ONNX format to a fusion classification model in tensorRT format.

[0101] In this embodiment, the ONNX model is converted to a TensorRT model for acceleration and deployment. The model conversion completes the inter-layer fusion described in the optimization process. Step 403 yields a TensorRT model optimized for a specific GPU platform and network model. This TensorRT model can be serialized and stored on disk or in memory.

[0102] Step 404: Deserialize the image recognition model in the tensorRT format, the human feature extraction model in the tensorRT format, the scene feature extraction model in the tensorRT format, and the fusion classification model in the tensorRT format, and create a running engine.

[0103] In this embodiment, the various TensorRT models obtained in step 403 are deserialized and a runtime engine is created. Image data can then be input into the TensorRT model to obtain the first sentiment classification result. Simultaneously, in step 404, kernel auto-tuning is performed to select the optimal underlying implementation of the various TensorRT models based on the number of data samples and the amount of input data. Dynamic tensor memory is also implemented, with TensorRT assigning video memory to each tensor, reducing memory usage and improving reuse efficiency.

[0104] Step 405: Obtain an accelerated image recognition model, an accelerated human feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model.

[0105] In this embodiment, the models used in the sentiment analysis process are accelerated according to the tensorRT optimizer, and these models are converted into the general ONNX format, and then the ONNX format model is converted into the tensorRT format, and the running engine is created after deserialization; this method can convert the image recognition model, human feature extraction model, scene feature extraction model and fusion classification model used in the present invention into a format that occupies less storage space and calls fewer resources for calculation, thereby improving the calculation speed of the model.

[0106] Please refer to Figure 5 , is a flow chart of another embodiment of the robot emotion analysis method provided by the present invention. Figure 5 and Figure 1 The difference is that step 103 is specifically steps 501-503, which are as follows:

[0107] Step 501: input the human body frame to the accelerated human body feature extraction model, and input the image data to the scene feature extraction model.

[0108] Step 502: The accelerated human feature extraction model extracts a first human feature of the verification body frame, and the scene feature extraction model extracts a first scene feature of the image data.

[0109] In this embodiment, the accelerated scene feature extraction model further divides the identified scene information into three scene types: negative, positive, and normal.

[0110] Step 503: The accelerated fusion classification model fuses the first human features and the first scene features to obtain a first emotion classification result.

[0111] In this embodiment, the first emotion classification result includes: a first discrete emotion classification result and a first continuous emotion classification result; the first discrete emotion classification result includes: one of: peace, gentleness, expectation, respect, confidence, interest, happiness, excitement, surprise, frustration, hostility, jealousy, fatigue, pain, hesitation, fear, injury, worry, annoyance, anger, disgust, sadness, boredom or embarrassment; the first continuous emotion classification result includes: happiness level, excitement level and control level, wherein each first continuous emotion classification result takes an integer value within a preset range.

[0112] In this example, 26 discrete emotions are artificially divided into three scene types. Each set contains multiple discrete emotion categories corresponding to the type of scene information. When obtaining the final emotion classification result, the discrete emotion categories within the emotion type corresponding to the scene are prioritized. When displaying the analysis results, the first emotion classification result is printed in the human frame.

[0113] Please refer to Figure 6 , which is a structural diagram of a robot emotion analysis system provided by the present invention. The robot emotion analysis system includes: an image acquisition module 601, a first control module 602, a second control module 603 and an interactive function module 604.

[0114] The image acquisition module 601 is used to acquire image data to be analyzed.

[0115] The first control module 602 is configured to locate the target area of ​​the image data using the accelerated image recognition model to obtain a human body frame.

[0116] The second control module 603 is used to input the human body frame and the image data into the sentiment analysis model, so that the sentiment analysis model obtains a first sentiment classification result based on the fusion calculation of the human body frame and the image data.

[0117] The interactive function module 604 is used to control the human-computer interaction behavior of the robot according to the first emotion classification result.

[0118] Please refer to Figure 7 , is another structural diagram of the robot emotion analysis system provided by the present invention. Figure 7 and Figure 6 The difference is that the second control module 603 further includes: a data input unit 701, a feature extraction unit 702 and a fusion classification unit 703.

[0119] In this embodiment, the data input unit 701 is used to input the human body frame into the accelerated human body feature extraction model, and input the image data into the scene feature extraction model.

[0120] The feature extraction unit 702 is configured to, after receiving the human frame input by the data input unit 701 , extract the first human feature of the human frame using the accelerated human feature extraction model and extract the first scene feature of the image data using the scene feature extraction model.

[0121] The fusion classification unit 703 is used to obtain a first emotion classification result by fusing the first human feature and the first scene feature with the accelerated fusion classification model after the feature extraction unit extracts the first human feature and the first scene feature.

[0122] Please refer to Figure 8 , is a schematic structural diagram of an embodiment of the robot provided by the present invention. The robot comprises: a central controller 801, an image acquisition device 802 and a human-computer interaction device 803.

[0123] In this embodiment, the central controller 801 is used to execute the robot emotion analysis method described in any one of steps 101-104, steps 201-204, steps 301-304, steps 401-405 and steps 501-503.

[0124] In this embodiment, the central controller 801 serves as the decision-making center of the robot system, can obtain the working status information of the remaining devices, send instructions to the remaining devices to coordinate the work of each device, and complete the computational reasoning of the accelerated image recognition model and sentiment analysis model.

[0125] The image acquisition device 802 is used to acquire image data to be analyzed and send the image data to the central controller.

[0126] In this embodiment, the image acquisition device 802 receives instructions from the central controller, completes the setting of parameters such as the height and angle of the camera, obtains image data and transmits it to the central controller.

[0127] The human-computer interaction device 803 is used to perform human-computer interaction behavior according to the control instructions of the central controller.

[0128] In this embodiment, the human-computer interaction device 803 communicates with the central controller in a two-way manner, and provides real-time feedback of status and external information, thereby realizing interactive functions of movement, voice, and expression.

[0129] As can be seen from the above, the robot emotion analysis method, system and robot provided by the present invention extract scene features and scene information that are useful for emotion analysis with the scene as the center, and the fusion classification model performs calculations based on human body features and scene features to obtain emotion classification results, so that the robot that deploys the image recognition model and the emotion analysis model can accurately identify the user's emotional information, allowing the robot to better understand the user's conversational intentions in human-computer interaction, allowing the user to obtain a better communication experience. The present invention mainly uses image data, specifically by intercepting video frames from image data and extracting human features and scene features from the video frames, and calculates the emotion classification results based on the fusion of the two features. Therefore, no additional traditional sensors are required except for the camera, and the use of deep learning solutions greatly reduces the dependence on the hardware system. In the process of deploying the image recognition model and the emotion analysis model on the robot side, the tensorRT optimizer is used to introduce model compression, model acceleration and other methods to provide the optimal model for the emotion analysis function of the intelligent robot, improve the model's calculation and reasoning speed and reduce the hardware deployment cost.

[0130] The specific embodiments described above further illustrate the objectives, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A robot sentiment analysis method, characterized in that: include: Acquiring image data to be analyzed; Locating the target area of ​​the image data according to the accelerated image recognition model to obtain a human body frame; Inputting the human body frame and the image data into a sentiment analysis model so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result; wherein the sentiment analysis model includes: an accelerated human body feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model; The accelerated human feature extraction model, the accelerated scene feature extraction model, and the accelerated fusion classification model are based on the CUDA version of the robot host storing the image recognition model, the human feature extraction model, the scene feature extraction model, and the fusion classification model, and the tensorRT optimizer corresponding to the CUDA version is loaded; Convert the image recognition model to an image recognition model in ONNX format; convert the human feature extraction model to a human feature extraction model in ONNX format; convert the scene feature extraction model to a scene feature extraction model in ONNX format; convert the fusion classification model to a fusion classification model in ONNX format; Convert the image recognition model in ONNX format to an image recognition model in tensorRT format; convert the human feature extraction model in ONNX format to a human feature extraction model in tensorRT format; convert the scene feature extraction model in ONNX format to a scene feature extraction model in tensorRT format; convert the fusion classification model in ONNX format to a fusion classification model in tensorRT format; Deserialize the image recognition model in the tensorRT format, the human feature extraction model in the tensorRT format, the scene feature extraction model in the tensorRT format, and the fusion classification model in the tensorRT format, and create a runtime engine to build the model; Control the robot's human-computer interaction behavior based on the first emotion classification result.

2. A robot emotion analysis method according to claim 1, characterized in that: Before acquiring the image data to be analyzed, including: Load the first image data and the first label data in the COCO dataset and normalize the first image data and the first label data; initialize the first backbone network, the feature pyramid network, and the first fully connected network of the YOLOv3 model to obtain an initialized YOLOv3 model; Inputting the first image data into the initialized YOLOv3 model to obtain a three-dimensional tensor embedding; Selecting a first loss function to calculate the loss according to the first label and the three-dimensional tensor embedding, and obtaining a first parameter gradient through back propagation; Parameters are adjusted according to the first parameter gradient until the initialized YOLOv3 model converges to obtain the image recognition model.

3. A robot emotion analysis method according to claim 1, characterized in that: Before acquiring the image data to be analyzed, the method further includes: Load the second image data and second label data from the EMOTIC dataset, load the human feature extraction model and the scene feature extraction model, normalize the second image data and the second label data, and initialize the second backbone network and the second fully connected network of the baseline CNN model; wherein, the first ResNet50 model pre-trained with the ImageNet dataset is used as the human feature extraction model, and the second ResNet50 model pre-trained with Places365 is used as the scene feature extraction model; The human feature extraction model extracts a second human feature of the second image data, the scene feature extraction model extracts a second scene feature of the second image data, and the baseline CNN model fuses the human feature and the scene feature to obtain a second emotion classification result; wherein the second emotion classification result includes a second continuous emotion classification result and a second discrete emotion classification result; Calculating the loss using a weighted square error loss function based on the second label data and the second discrete sentiment classification result, and obtaining a second parameter gradient through back propagation; calculating the loss using an L1 loss function based on the second label data and the second continuous sentiment classification result, and obtaining a third parameter gradient through back propagation; Parameters are adjusted according to the second parameter gradient and the third parameter gradient until the baseline CNN model converges to obtain the fusion classification model.

4. A robot emotion analysis method according to claim 1, characterized in that: The target area of ​​the image data is located according to the accelerated image recognition model to obtain a human body frame, specifically: Reading a video frame of the image data; Cropping the video frame to a size of a preset resolution; Inputting the cropped video frame into the backbone module of the backbone network of the accelerated image recognition model, and extracting features through the feature pyramid network of the accelerated image recognition model; The bounding box coordinates, scores, and multiple human body frames of the human body region are output according to the features, and after performing a non-maximum suppression operation on the multiple human body frames, the human body frame with the highest confidence is selected.

5. A robot emotion analysis method according to claim 1, characterized in that: The human body frame and the image data are input into the sentiment analysis model so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result, specifically: Inputting the human body frame into the accelerated human body feature extraction model, and inputting the image data into the scene feature extraction model; The accelerated human feature extraction model extracts a first human feature of the human frame, and the scene feature extraction model extracts a first scene feature of the image data; The accelerated fusion classification model fuses the first human features and the first scene features to obtain a first emotion classification result.

6. A robot emotion analysis method according to claim 1 or 5, characterized in that: The first emotion classification result includes: a first discrete emotion classification result and a first continuous emotion classification result; The first discrete emotion classification results include one of the following: calm, gentle, expectant, respectful, confident, interested, happy, excited, surprised, frustrated, hostile, jealous, tired, painful, hesitant, fearful, hurt, worried, annoyed, angry, disgusted, sad, bored, or embarrassed; The first continuous emotion classification results include: happiness level, excitement level, and control level, wherein each first continuous emotion classification result takes an integer value within a preset range.

7. A robot emotion analysis system, characterized in that: include: Image acquisition module, first control module, second control module and interactive function module; The image acquisition module is used to acquire image data to be analyzed; The first control module is used to locate the target area of ​​the image data according to the accelerated image recognition model to obtain a human body frame; The second control module is used to input the human body frame and the image data into the sentiment analysis model, so that the sentiment analysis model performs a fusion calculation on the human body frame and the image data to obtain a first sentiment classification result; wherein the sentiment analysis model includes: an accelerated human body feature extraction model, an accelerated scene feature extraction model, and an accelerated fusion classification model; The accelerated human feature extraction model, the accelerated scene feature extraction model, and the accelerated fusion classification model are based on the CUDA version of the robot host storing the image recognition model, the human feature extraction model, the scene feature extraction model, and the fusion classification model, and the tensorRT optimizer corresponding to the CUDA version is loaded; Convert the image recognition model to an image recognition model in ONNX format; convert the human feature extraction model to a human feature extraction model in ONNX format; convert the scene feature extraction model to a scene feature extraction model in ONNX format; convert the fusion classification model to a fusion classification model in ONNX format; Convert the image recognition model in ONNX format to an image recognition model in tensorRT format; convert the human feature extraction model in ONNX format to a human feature extraction model in tensorRT format; convert the scene feature extraction model in ONNX format to a scene feature extraction model in tensorRT format; convert the fusion classification model in ONNX format to a fusion classification model in tensorRT format; Deserialize the image recognition model in the tensorRT format, the human feature extraction model in the tensorRT format, the scene feature extraction model in the tensorRT format, and the fusion classification model in the tensorRT format, and create a runtime engine to build the model; The interactive function module is used to control the human-computer interaction behavior of the robot according to the first emotion classification result.

8. A robot emotion analysis system according to claim 7, characterized in that: The second control module includes: a data input unit, a feature extraction unit and a fusion classification unit; The data input unit is used to input the human body frame into the accelerated human body feature extraction model, and input the image data into the scene feature extraction model; The feature extraction unit is used to extract the first human feature of the human frame using the accelerated human feature extraction model, and the scene feature extraction model is used to extract the first scene feature of the image data; The fusion classification unit is used to fuse the first human features and the first scene features with the accelerated fusion classification model to obtain a first emotion classification result.

9. A robot, characterized in that: include: Central controller, image acquisition equipment and human-computer interaction equipment; Wherein, the central controller is used to execute the robot emotion analysis method according to any one of claims 1 to 6; The image acquisition device is used to acquire image data to be analyzed and send the image data to the central controller; The human-computer interaction device is used to perform human-computer interaction behavior according to the control instructions of the central controller.

Images