Unified attention mechanism-based interactive robot multi-task learning method

Abstract

The application provides a unified interactive robot multitask learning method based on an attention mechanism, and a model can identify facial expressions, age distribution and gender information in an image.The unified interactive robot multitask learning method based on the attention mechanism comprises the following steps: S1. a unified deep convolutional neural network model based on the attention mechanism is used for multitask learning, and facial expressions, age and gender are simultaneously identified; S2. the model in step S1 is trained and optimized: 1) a loss function is set for the model implemented in step 1; 2) a data set is input, and the model is trained; 3) the training is stopped when the model converges or the accuracy reaches the application requirement; and S3. the model trained in S2 is deployed to a robot.

Description

Technical Field

[0001] The present invention relates to a unified interactive robot multi-task learning method based on an attention mechanism, and belongs to the technical field of artificial intelligence. Background Art

[0002] With the application of deep learning technology in human-robot interaction, robots are becoming increasingly intelligent and human-like. During emotional interaction between robots and humans, the primary challenge is that the robot must first identify basic information about the person interacting, such as their emotional state, gender, and age. This allows the robot to select different interaction strategies to achieve greater intelligence and human-like behavior, thereby enhancing the robot's value. To achieve this goal without pre-existing information, the robot often requires cameras to capture the person's presence and deploy multiple deep neural network models to separately identify emotion, gender, and age information within facial regions. Deploying multiple models increases the memory overhead of the robot's computing platform, resulting in increased computational overhead and higher inference latency. This is unfriendly to resource-constrained robotic computing platforms and negatively impacts the real-time interaction experience. Summary of the Invention

[0003] The purpose of this invention is to provide a unified multi-task learning method for interactive robots based on the attention mechanism. The model can recognize the facial expressions, age distribution and gender information of the human face in the image.

[0004] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions:

[0005] A unified attention-based multi-task learning method for interactive robots, including the following steps:

[0006] S1. A unified attention-based deep convolutional neural network model for multi-task learning to simultaneously recognize facial expressions and estimate age and gender.

[0007] S2. Train and optimize the model in step S1:

[0008] 1) Set the loss function for the model implemented in step 1;

[0009] 2) Input data set and train the model;

[0010] 3) Stop training when the model converges or the accuracy meets the application requirements;

[0011] S3. Deploy the model trained in S2 to the robot.

[0012] Based on the above unified interactive robot multi-task learning method based on the attention mechanism, the specific steps of step S1 are as follows:

[0013] The model architecture consists of a feature extractor and three local networks. The number of channels of the feature map output by the feature extractor is consistent with the number of channels of the feature maps output by the three local networks. The attention mechanism is that the feature extractor abstracts the facial image and outputs a high-dimensional feature map through multiple layers of convolution. The three local networks are responsible for constructing local focus features that are strongly correlated with the corresponding tasks according to three different tasks, and are coupled with the feature maps output by the feature extraction network respectively, and further output to the classification branches of the three tasks.

[0014] Based on the above unified attention-based interactive robot multi-task learning method, the basic operation of the attention mechanism is matrix multiplication, which is implemented as follows:

[0015] The feature map output by the feature extractor is denoted as F1 and converted into a feature matrix denoted as M1 in spatial order. The dimension of M1 is C rows and W*H columns, where C is the number of channels of F1, and W and H are the width and height of F1 respectively;

[0016] In the same way, the feature maps output by the three local networks are recorded as F2, F3, and F4 respectively, and converted into three feature matrices recorded as M2, M3, and M4 respectively;

[0017] The number of channels of the feature maps output by the feature extractor and the local network is the same, so the number of rows of M2, M3 and M4 is C, so M1 is matrix multiplied with M2, M3 and M4 respectively to obtain three correlation matrices;

[0018] The three correlation matrices are expanded into column vectors and then input into the three classification task head networks to recognize expression, age and gender.

[0019] The advantages of the present invention are:

[0020] The model infers facial images and obtains facial expressions, age distribution and gender information at the same time. It is a high-efficiency real-time inference model that can be used for robot emotional interaction. The inference information provides perception capabilities for robot interaction, making the interaction more intelligent. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0022] Figure 1 Schematic diagram of the deep convolutional neural network architecture of an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0024] As mentioned in the background of this invention, multiple deep neural network models are deployed in a robot to identify emotion, gender, age, and other information in facial images. Deploying multiple models increases the memory overhead of the robot's computing platform, resulting in more computation and higher inference latency. This is not user-friendly for resource-limited robotic computing platforms and negatively impacts the real-time interactive experience.

[0025] In order to solve the problem of model accuracy, the present invention adopts the following technical solutions:

[0026] A unified attention-based multi-task learning method for interactive robots, including the following steps:

[0027] S1. A unified attention-based deep convolutional neural network model for multi-task learning to simultaneously recognize facial expressions and estimate age and gender.

[0028] S2. Train and optimize the model in step S1:

[0029] 1) Set the loss function for the model implemented in step 1;

[0030] 2) Input data set and train the model;

[0031] 3) Stop training when the model converges or the accuracy meets the application requirements;

[0032] S3. Deploy the model trained in S2 to the robot.

[0033] The model architecture consists of a feature extractor and three local networks. The number of channels of the feature map output by the feature extractor is consistent with the number of channels of the feature maps output by the three local networks. The attention mechanism is that the feature extractor abstracts the facial image and outputs a high-dimensional feature map through multiple layers of convolution. The three local networks are responsible for constructing local focus features that are strongly correlated with the corresponding tasks according to three different tasks, and are coupled with the feature maps output by the feature extraction network respectively, and further output to the classification branches of the three tasks.

[0034] Example 1

[0035] refer to Figure 1 , a unified multi-task learning method for interactive robots based on attention mechanism, including the following steps:

[0036] S1. Use a deep learning programming framework such as TensorFlow or PyTorch to establish a model architecture, and the basic operation of the attention mechanism is matrix multiplication, which is implemented as follows:

[0037] The feature map output by the feature extractor is denoted as F1, which is converted into a feature matrix denoted as M1 in spatial order, and the dimension of M1 is C rows and W*H columns, where C is the channel number of F1, and W and H are the width and height of F1, respectively.

[0038] In the same way, the feature maps output by the three local networks are denoted as F2, F3, and F4, respectively, and the three feature matrices are denoted as M2, M3, and M4, respectively.

[0039] The channel numbers of the feature maps output by the feature extractor and the local networks are consistent, so the row numbers of M2, M3, and M4 are all C, so M1 is multiplied by M2, M3, and M4, respectively, to obtain three correlation matrices.

[0040] The three correlation matrices are respectively expanded into column vectors, and then input into three classification task head networks to identify expressions, ages, and genders.

[0041] S2. Train and optimize the model in step S1:

[0042] Set the loss function for the model implemented in step 1, input the data set, train the model, and stop training when the model converges or the accuracy meets the application requirements.

[0043] S3. Deploy the trained model in S2 to the robot: set the trained model from the training state to the inference state and deploy it as an HTTP or other type of callable service, with the service input being a face image and the return content being the expression, age, and gender information of the person in the face image; when there is an interactive task that needs to identify the expression, age, or gender information of the interactor, the service can be called to obtain the information.

[0044] Finally, it should be noted that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A unified attention-based multi-task learning method for interactive robots, characterized by: The following steps are involved: S1. A unified attention-based deep convolutional neural network model for multi-task learning to simultaneously recognize facial expressions and estimate age and gender. S2. Train and optimize the model in step S1: 1) Set the loss function for the model implemented in step 1; 2) Input data set and train the model; 3) Stop training when the model converges or the accuracy meets the application requirements; S3. Deploy the model trained in S2 to the robot; The specific steps of step S1 are as follows: The model architecture consists of a feature extractor and three local networks. The number of channels of the feature map output by the feature extractor is consistent with the number of channels of the feature maps output by the three local networks. The attention mechanism is that the feature extractor abstracts the facial image and outputs a high-dimensional feature map through multiple layers of convolution. The three local networks are responsible for constructing local focus features that are strongly correlated with the corresponding tasks according to three different tasks, and are coupled with the feature maps output by the feature extraction network respectively, and further output to the classification branches of the three tasks.

2. The unified attention-based interactive robot multi-task learning method according to claim 1, characterized in that: The basic operation of the attention mechanism is matrix multiplication, which is implemented as follows: The feature map output by the feature extractor is denoted as F1 and converted into a feature matrix denoted as M1 in spatial order. The dimension of M1 is C rows and W*H columns, where C is the number of channels of F1, and W and H are the width and height of F1 respectively; In the same way, the feature maps output by the three local networks are recorded as F2, F3, and F4 respectively, and converted into three feature matrices recorded as M2, M3, and M4 respectively; The number of channels of the feature maps output by the feature extractor and the local network is the same, so the number of rows of M2, M3 and M4 is C, so M1 is matrix multiplied with M2, M3 and M4 respectively to obtain three correlation matrices; The three correlation matrices are expanded into column vectors and then input into the three classification task head networks to recognize expression, age and gender.

Images