Robot facial expression generation method based on deep reinforcement learning
By using deep reinforcement learning and generative adversarial network methods in virtual environments, robots can independently learn and generate natural and realistic facial expressions, solving the problems of insufficient diversity and adaptability of expression generation and serious hardware wear in the existing technology, and achieving a more efficient and economical robot facial expression generation.
Patent Information
- Application Number
- CN202411967841.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-28
- Publication Date
- 2025-04-25
AI Technical Summary
Existing robot facial expression generation technology relies on predefined templates or specific input data, making it difficult to adapt to dynamic and changeable emotional expressions. The generated expressions are diverse and adaptable, with high system complexity and severe hardware wear, resulting in short service life and high maintenance costs.
Using a method based on deep reinforcement learning, the robot can learn independently in a virtual environment and generate natural and realistic facial expressions by generating adversarial network (GAN) and deep reinforcement learning (DRL) models, and combine the generative model and visual evaluation standards.
The naturalness and fidelity of the robot's facial expressions have been significantly improved, and the diversity and adaptability of the generated expressions have been greatly improved, which reduces the dependence on physical hardware, extends the service life of the robot, and reduces maintenance costs and development complexity.
Smart Images
Figure CN119888025A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of robots and artificial intelligence, and in particular to a method for generating robot facial expressions based on deep reinforcement learning. Background Art
[0002] Robots are increasingly used in social interaction, companionship, education and other fields. With the rapid development of robotics and artificial intelligence technologies, facial expression generation technology, as an important means of robot emotional expression, has become a hot topic in research and development.
[0003] Currently, the methods for generating robot facial expressions can be mainly divided into the following categories.
[0004] The first is manual pre-programming. Basic expressions are simulated by writing a series of predefined facial movement instructions. This method is relatively simple to implement, but the naturalness and diversity of expressions are limited. It is difficult to accurately capture the complex changes of human expressions, resulting in the robot's expressions appearing stiff and lacking in realism.
[0005] The second method is facial feature mapping. This method generates expressions by mapping human expression features to the robot's facial structure. This method can improve the naturalness of expressions, but its flexibility and diversity are limited by input data and preset templates, making it difficult to generate rich and diverse emotional expressions.
[0006] The third method is automatic generation of facial expressions. Facial expressions are generated through automated methods, usually relying on machine learning algorithms to extract features from predefined expression datasets. However, most existing methods are still based on the extension of manual pre-programming, and only generate predefined expression instruction sets through algorithms, which fails to fully simulate the complexity and dynamics of the human expression generation process.
[0007] Although the existing robot facial expression generation technology helps the robot expressions to simulate human natural expressions to a certain extent, it has at least the following shortcomings: 1. The existing technology relies on predefined templates or specific input data, and it is difficult to adapt to the dynamic and changeable emotional expression needs. Especially in complex social scenes, it is difficult to capture subtle emotional changes, and the diversity and adaptability of expression generation are poor. 2. The existing technology also relies on external capture devices or preset template libraries, which increases the complexity of the system and limits the robot's ability to generate expressions autonomously. 3. The existing technology is to directly map facial features to explicit instruction sets, which cannot fully simulate the complexity and dynamics of the human expression generation process. The generated expressions are not natural enough in continuity and transition, and cannot achieve the performance effect of real human expressions. 4. The implementation of the existing technology requires frequent activation and control of the robot's facial structure, which is easy to cause hardware wear, shorten the robot's service life, and increase maintenance costs.
[0008] In summary, the shortcomings of the existing technology are that the processing method for facial expression generation is too simplified and fails to fully consider the multi-dimensional factors involved in the expression generation process, such as emotional state, psychological changes, social context, etc. The direct mapping method that relies on explicit instruction sets is also relatively simple in technical implementation, which greatly limits the flexibility and diversity of expression generation. In addition, the hardware loss problem caused by the existing technology when training or executing on actual robots also needs to be solved urgently. Summary of the invention
[0009] The present invention proposes a robot facial expression generation method based on deep reinforcement learning. By introducing a generation model and a reward function calculation method of a visual judgment standard, the robot can learn autonomously in a virtual environment, thereby generating more natural and realistic facial expressions. To solve the above technical problems, the present application provides a robot facial expression generation method based on deep reinforcement learning, comprising the following steps:
[0010] S1: Build a virtual robot model,
[0011] Generative adversarial networks (GANs) are used as generative models to simulate facial expressions to build virtual robot models;
[0012] The generative model includes a generator and a discriminator;
[0013] The input of the generator is the motor command and the real robot facial expression image, and the output is two intermediate masks, which are combined to generate the robot's virtual facial expression image; the motor command is the target expression representation vector, and the real robot facial expression image is the original input image;
[0014] The input of the discriminator is the virtual facial expression image generated by the generator or the real robot facial expression image directly extracted from the real data set. It is processed by the multi-layer shared convolutional layer of the discriminator and outputs the target expression representation vector prediction result and authenticity score.
[0015] S2: Construct a virtual training environment to simulate the physical structure and characteristics of the robot's facial expression generation, so that the output robot facial expression image is consistent with the motor command;
[0016] Constructing a simulated virtual training environment in a computer system, wherein the virtual training environment includes the virtual robot model and its control interface;
[0017] The input of the virtual training environment is motor commands, which are used to control the virtual robot model to generate robot facial expression images;
[0018] The control interface communicates via messages in the Robot Operating System (ROS);
[0019] S3: Steps for training a deep reinforcement learning (DRL) model to generate a model
[0020] The deep reinforcement learning (DRL) model uses the SAC algorithm (Soft Actor-Critic), whose input is the robot's current state representation vector, that is, an n-dimensional vector, where n is an integer greater than or equal to 1, describing the servo angle corresponding to the robot's facial features;
[0021] Its output is the current action vector that the robot should make in the current state, that is, the change of each robot motor relative to the current servo angle;
[0022] The generation model is a sub-part of the deep reinforcement learning (DRL) model algorithm framework. After the deep reinforcement learning model outputs an action, it is used to generate a facial expression image of the virtual robot after the action is executed, thereby obtaining the similarity between the current facial expression image of the virtual robot and the target expression image, and calculating the current reward.
[0023] As an improved technical solution of the present application, the generator integrates the methods of convolutional layer, residual layer, and deconvolution layer to extract high-level features. The specific process is: the generator network model has two key inputs: the target expression representation vector and the original input image. The target expression representation vector is expanded to the same spatial dimension as the original input image after preprocessing so as to be effectively fused with the original image to form a comprehensive input. After the comprehensive input is processed through the hierarchical structure of convolutional layer, residual layer and deconvolution layer respectively, the generator outputs the two intermediate masks described in S1, and combines the two intermediate masks to generate the robot's virtual facial expression image.
[0024] As an improved technical solution of the present application, it also includes a physical robot deployment step S4 of the virtual training result: deploying the deep reinforcement learning (DRL) model trained in step S3 to the physical robot so that it can generate corresponding expressions;
[0025] Generate a virtual robot facial expression image corresponding to the motor command according to the motor command and a real robot facial expression image through the generation model described in S1;
[0026] The generated virtual robot facial expression image is used to train the deep reinforcement learning model described in S3, and the trained deep reinforcement learning model is deployed on the physical robot. The actions output by the control model are directly applied to the physical robot, presenting a real facial expression image on the robot's face.
[0027] As an improved technical solution of the present application, it also includes a feedback and optimization step S5: the physical robot deploys the reinforcement learning model in step S4 and applies it to generate corresponding expressions. The generation model evaluates the expression generation effect through visual sensors and user feedback, adjusts the current action representation vector that the robot should make, and optimizes the deep reinforcement learning model; the optimized deep reinforcement learning model chooses to iterate to step S3 for further training or directly deploy it to the physical robot from step S4 according to the naturalness and realism of the generated expression.
[0028] As an improved technical solution of the present application, the generative model is a variational autoencoder (VAE).
[0029] As an improved technical solution of the present application, the way in which the deep reinforcement learning (DRL) model interacts with the generation model is to use the simulated facial expression image output by the generation model for the reward function of the deep reinforcement learning (DRL) model. The specific process is: the facial expression generation algorithm framework of the deep reinforcement learning (DRL) model includes an Actor network and a Critic network. The target expression label and the current state representation vector are input into the Actor network, and the action vector of the next step is output; in the Critic network, the target expression label, the current state representation vector and the current action representation vector are input, and the evaluation value of the effect of the current action representation vector in the current state under the constraint of the target expression label is output; the current action representation vector acts on the current state to obtain the next state representation vector, and the next state representation vector is compared with the virtual facial expression image generated by S1 to obtain the similarity vector, and finally the reward function is used to calculate the immediate reward.
[0030] As an improved technical solution of the present application, the deep reinforcement learning (DRL) model adopts a proximal policy optimization (PPO) algorithm, the input of which is a target expression label and a current state representation vector, and the output is a representation vector of the action that the robot should take under the current target expression label and the current state.
[0031] As an improved technical solution of the present application, the deep reinforcement learning (DRL) model adopts the asynchronous advantage actor-critic (A3C) algorithm or the deep Q learning (DQN) algorithm, the input of which is the target expression label and the current state representation vector, and the output is the action representation vector that the robot should take under the current target expression label and the current state.
[0032] As an improved technical solution of the present application, the generation model is replaced by simulation platforms Gazebo and Unity3D.
[0033] Beneficial Effects
[0034] This application combines generative models with deep reinforcement learning to generate more natural and realistic facial expressions. The generative model is trained with a large amount of real human expression data, and can capture subtle changes in human expressions, thereby generating highly realistic expressions. Experimental results show that using the method of this application, the facial expressions generated by the robot achieve a score that is highly consistent with real human expressions in visual evaluation, which is significantly better than traditional pre-programmed methods and template-based generation techniques.
[0035] This application uses deep reinforcement learning to enable robots to dynamically generate expressions that meet the needs of emotional expression according to different social situations, breaking through the limitations of existing technologies in preset templates or fixed instruction sets. This method enables robots to generate rich and diverse expressions and adapt to complex interactive scenarios, significantly improving the flexibility and diversity of expression generation.
[0036] This application significantly reduces the hardware dependence on the physical robot by training in a virtual environment, thereby effectively reducing the risk of hardware wear and tear during the training process. Experimental data shows that the method of the present invention can extend the service life of the hardware by at least 50%, significantly reduce maintenance costs and replacement frequency, and improve the overall economy of the system.
[0037] This application greatly simplifies the development process of the robot facial expression generation model by training in a virtual environment. Developers can directly debug and optimize the expression generation model in the virtual environment, greatly reducing the dependence on physical equipment, thereby greatly improving the flexibility and convenience of the development process. At the same time, the virtual training environment is easy to set up and adjust, and can adapt to different types of robot platforms, further improving the applicability and promotion of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram showing a method for generating robot facial expressions based on deep reinforcement learning.
[0039] Figure 2 Given the current state s t In the case of t+1 Follow action a t Changes in the relationship diagram. DETAILED DESCRIPTION
[0040] Example 1
[0041] A robot facial expression generation method based on deep reinforcement learning, such as Figure 1 As shown, the following steps are included:
[0042] S1: Build a virtual robot model,
[0043] Generative adversarial networks (GANs) are used as generative models to simulate facial expressions to build virtual robot models;
[0044] When the generative model is a variational autoencoder (VAE), the generative model includes a generator and a discriminator; in practical applications, the generative model can be replaced by simulation platforms Gazebo and Unity3D.
[0045] The generator combines the convolutional layer, residual layer, and deconvolution layer methods to extract high-level features. The specific process is as follows: the input of the generator is the motor command x′ tar and real robot facial expression image (original input image) I g_src , the target expression representation vector (the input is the motor command x′ tar ) is expanded to the same spatial dimension as the original input image after preprocessing, so as to be effectively fused with the original input image to form a comprehensive input. After the comprehensive input is processed by the hierarchical structure of the convolution layer, the residual layer and the deconvolution layer, the output is two intermediate masks, namely, the single-channel G a and three-channel G c ; Combine the two intermediate masks to generate the robot's virtual facial expression image I g_out , the calculation formula of the generation process: I g_out =(1-G a )·G c +G a I g_src ; Among them, the input motor command is the target expression vector; as Figure 1 As shown, a color image is actually composed of three channels. Here, the single-channel G a and three-channel G c They are also images. They are when the motor command x′ is input to the generative model tar and real robot facial expression images I g_src , the intermediate result obtained after being processed by the generator in the generative model.
[0046] The input of the discriminator is the virtual facial expression image I generated by the generator g_out Or directly extract the real robot facial expression image I from a real data set (the data set is collected by the user in advance, one sample is collected each time, and the total number of samples is more than 20,000. Specifically, a motor command is randomly generated each time, sent to the physical robot for execution, and then the facial expression image of the robot after executing the motor command is captured, so that a data set containing more than 20,000 robot facial images and their corresponding motor command data is collected.) d_src , through the multi-layer shared convolutional layer processing of the discriminator, two key indicators are output: one is for the input image I d_inputThe target expression vector prediction result D aus ; The second is about the input image I d_input The authenticity score of D score ; Here, the input image I d_input Virtual facial expression image I generated by the generator g_out Or real robot facial expression images directly extracted from real datasets I d_src ;
[0047] The model learns real robot facial expression sample data to form the robot's facial expression image and its corresponding motor command data, and then generates the robot's virtual facial expression image.
[0048] S2: Construct a virtual training environment to simulate the physical structure and characteristics of the robot's facial expression generation, so that the output robot facial expression image is consistent with the motor command;
[0049] Constructing a simulated virtual training environment in a computer system, wherein the virtual training environment includes the virtual robot model and its control interface;
[0050] The input of the virtual training environment is motor commands, which are used to control the virtual robot model to generate robot facial expression images;
[0051] The control interface communicates via messages in the Robot Operating System (ROS).
[0052] S3: Steps for training a deep reinforcement learning (DRL) model to generate a model.
[0053] The deep reinforcement learning (DRL) model adopts an asynchronous advantage actor-critic (A3C) algorithm or a deep Q-learning (DQN) algorithm or a SAC algorithm (Soft Actor-Critic).
[0054] When the deep reinforcement learning (DRL) model uses the SAC algorithm (Soft Actor-Critic), its input is the robot's current state representation vector s t, that is, an n-dimensional vector, where n is an integer greater than or equal to 1, describing the servo angle corresponding to the robot's facial features, that is, representing the current robot facial expression image; (In reinforcement learning, the agent needs to decide the next action based on the state it is in. For a robot, the state can include multiple parameters such as speed and acceleration. The robot in this application is a head component that changes facial features through a number of servos driving a silicone face skin. Since the change in facial features is directly related to the servo angle, the servo angle is used to describe the amplitude of the change in facial features. For example, the amplitude of the mouth opening and closing can be measured by the servo angle that controls the mouth opening and closing, and it is normalized to a value between 0 and 1 according to the range of motion, where 0 means the mouth is completely closed and 1 means the mouth is completely open. In order to control multiple features of the robot's face, multiple variables are needed to describe these features. For example, if you need to describe the four facial features of the mouth, left eyebrow, right corner of the mouth, and left eye, you can describe them by the servo angles associated with these features. At this time, the number of variables n = 4.) Its output is the current action representation vector a that the robot should make in the current state t , which is the change of each motor relative to the current angle;
[0055] like Figure 1 As shown, the target expression vector c represents the facial expression category that needs to be generated, and the current action vector a t Indicates the current target expression representation vector c and the current state representation vector s t The robot system represents the action to be taken by the robot from the current state vector s t To the next state representation vector s t+1 The state transition process is represented by the current state vector s t and the current action performed represents the vector a t Joint decision, specifically expressed as follows: Where s t，i is the i-th element of the current state representation vector, s t+1，i is the i-th element of the next state representation vector, a t，i , is the i-th element of the current action representation vector, i is an integer from 1 to n, and n is an integer greater than or equal to 1. In the above formula, s t+1，i and a t，i The relationship can be obtained through Figure 2 Intuitive representation.
[0056] The facial expression generation algorithm framework of deep reinforcement learning includes Actor network and Critic network. The target expression label c is introduced into the input of Actor network and Critic network. After the embedding operation, the target expression label c is respectively combined with the original input of Actor network and Critic network (the original input here is Figure 1The state s in t ,
[0057] After the target c is introduced and fused together as the input of the network), the model is allowed to learn and generate the specified target expression; that is, in the Actor network, the input target expression label c and the current state representation vector s t Representation vector fusion post-processing to generate the action representation vector a for the next step t+1 ; In the Critic network, the target expression label c and the current state representation vector s t and the current action representation vector a t The fusion result of is processed to evaluate the execution of the current action representation vector a under the constraint of the target expression representation vector c. t In the current state, the vector s t The effect when the current state represents the vector s t Execute the current action representation vector a t After that, the robot enters the next state representation vector s t+1 , in order to calculate the next state representation vector s t+1 The instant reward under the expression generation model is used to generate the next state representation vector s t+1 The corresponding robot facial expression image is compared with the target expression image in key areas to obtain the similarity vector
[0058] The number of key areas is m, m ≥ 1. (If the robot's facial expression image is directly compared with the target expression image for similarity, since most areas of the face are usually similar, the overall similarity will be very high, thereby masking the impact of subtle changes in local facial features on the similarity. To solve this problem, the present application divides the robot's facial area and extracts key areas. These key areas are parts where facial features change significantly. The specific number of key areas to be divided needs to be determined based on the actual debugging situation). Therefore, the number m of key areas only needs to satisfy m ≥ 1 and does not need to be fixed to a specific value. Calculate the similarity sim(κ) between the generated expression image and the target expression image in this area i ), the value range is [0, 1]; finally, the basic reward (Calculation formula: In the formula represents an indicator function, δ 1 With δ 2 are the adjustment coefficients corresponding to rewards and penalties, respectively, where δ 1 Reward those key regions that reach the similarity threshold, δ 2Those areas that fail to meet the standards will be penalized, κ i is the ith key region, sim(κ i ) is the difference between the generated expression image and the target expression image in κ i The similarity of the region, θ is the set threshold, ranging from 0 to 1; i is an integer from 1 to m, and the minimum similarity reward is
[0059] (Calculation formula: In the formula is the similarity vector, θ is the set threshold, ranging from 0 to 1, ξ is a positive number, which is used to handle the situation where the similarity threshold conditions of all key regions are not met, i is an integer from 1 to m, and the number of key regions m only needs to satisfy m ≥ 1), variance penalty reward (Calculation formula: In the formula is the similarity vector, for The reward function formula composed of the standard deviation of t ,Right now (Calculation formula: In the formula As a basic reward, is the minimum similarity reward, is the variance penalty reward, η 1 , η 2 , η 3 For the corresponding The coefficient of ) provides the model with the feedback reward needed to optimize the accuracy of action generation. In addition, the experience of the current step (s t , a t ,s t+1 , r t , c) are stored in the experience pool to ensure that the training process can effectively use these accumulated experiences for learning, where s t is the current state representation vector, a t is the current action representation vector, s t+1 is the next state representation vector, r t is the immediate reward, and c is the target expression label.
[0060] S4: Deploy the deep reinforcement learning (DRL) model trained in step S3 to the physical robot so that it can generate corresponding expressions;
[0061] Generate a robot facial expression image corresponding to the motor command according to the output motor command and a real robot facial expression image through the generation model described in S1;
[0062] The generated robot facial expression image is used to train the deep reinforcement learning model described in S3, and the trained deep reinforcement learning model is deployed on the physical robot. The actions output by the control model are directly applied to the physical robot, presenting a real facial expression image on the robot's face.
[0063] S5: Feedback and optimization step. After applying the generated expression model on the physical robot, the expression generation effect is evaluated through visual sensors and user feedback. According to the reward formula in S3: The comprehensive reward function framework is composed of As a basic reward, is the minimum similarity reward, is the variance penalty reward, η 1 , η 2 , η 3 For the corresponding The coefficient of ; focuses on the optimization of the similarity of key areas, and introduces expression consistency and outlier processing to guide the model to generate images that are highly consistent with the target expression in visual terms, and adjust the current action representation vector a that the robot should make t (the change of each motor relative to the current angle), improve the naturalness and realism of the generated expression in the physical environment, and optimize the deep reinforcement learning model. The optimized deep reinforcement learning model chooses to iterate to step S3 for further training or directly deploy it to the physical robot from step S4 according to the naturalness and realism of the generated expression.
[0064] In summary, the entire method not only overcomes the deficiencies of the existing technology in terms of naturalness, flexibility and hardware protection, but also effectively improves development efficiency and simplifies the use process through virtual training. These advantages give this application a significant technical advantage in the field of robot facial expression generation, and provide more powerful technical support for the application of robots in the fields of social interaction, companionship and education.
Claims
1. A robot facial expression generation method based on deep reinforcement learning, characterized in that: The following steps are involved: S1: Build a virtual robot model, Generative adversarial networks (GANs) are used as generative models to simulate facial expressions to build virtual robot models; The generative model includes a generator and a discriminator; The input of the generator is the motor command and the real robot facial expression image, and the output is two intermediate masks, which are combined to generate the robot's virtual facial expression image; the motor command is the target expression representation vector, and the real robot facial expression image is the original input image; The input of the discriminator is the virtual facial expression image generated by the generator or the real robot facial expression image directly extracted from the real data set. It is processed by the multi-layer shared convolutional layer of the discriminator and outputs the target expression representation vector prediction result and authenticity score. S2: Construct a virtual training environment to simulate the physical structure and characteristics of the robot's facial expression generation, so that the output robot facial expression image is consistent with the motor command; Constructing a simulated virtual training environment in a computer system, wherein the virtual training environment includes the virtual robot model and its control interface; The input of the virtual training environment is motor commands, which are used to control the virtual robot model to generate robot facial expression images; The control interface communicates via messages in the Robot Operating System (ROS); S3: Steps for training a deep reinforcement learning (DRL) model to generate a model The deep reinforcement learning (DRL) model uses the SAC algorithm (Soft Actor-Critic), whose input is the robot's current state representation vector, that is, an n-dimensional vector, where n is an integer greater than or equal to 1, describing the servo angle corresponding to the robot's facial features; Its output is the current action vector that the robot should make in the current state, that is, the change of each robot motor relative to the current servo angle; The generation model is a sub-part of the deep reinforcement learning (DRL) model algorithm framework, and is used to generate a robot facial expression image after the deep reinforcement learning model outputs an action, thereby obtaining the similarity between the current robot facial expression image and the target expression image, and calculating the current reward.
2. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: The generator combines the methods of convolutional layers, residual layers, and deconvolutional layers to extract high-level features. The specific process is as follows: the generator network model has two key inputs: a target expression representation vector and an original input image. The target expression representation vector is expanded to the same spatial dimension as the original input image after preprocessing so as to be effectively fused with the original image to form a comprehensive input. After the comprehensive input is processed in the hierarchical structure of the convolutional layer, the residual layer, and the deconvolution layer, the generator outputs the two intermediate masks described in S1, and combines the two intermediate masks to generate a virtual facial expression image of the robot.
3. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: It also includes a physical robot deployment step S4 of the virtual training result: deploying the deep reinforcement learning (DRL) model trained in step S3 to the physical robot so that it can generate corresponding expressions; Generate a robot facial expression image corresponding to the motor command according to the motor command and a real robot facial expression image through the generation model described in S1; The generated robot facial expression image is used to train the deep reinforcement learning model described in S3, and the trained deep reinforcement learning model is deployed on the physical robot. The actions output by the control model are directly applied to the physical robot, presenting a real facial expression image on the robot's face.
4. A robot facial expression generation method based on deep reinforcement learning according to claim 3, characterized in that: It also includes a feedback and optimization step S5: the physical robot is applied to generate corresponding expressions after the reinforcement learning model is deployed in step S4, and the generation model evaluates the expression generation effect through visual sensors and user feedback, adjusts the current action representation vector that the robot should make, and optimizes the deep reinforcement learning model; the optimized deep reinforcement learning model chooses to iterate to step S3 for further training or directly deploy it to the physical robot from step S4 according to the naturalness and realism of the generated expression.
5. A robot facial expression generation method based on deep reinforcement learning according to any one of claims 1-4, characterized in that: The generative model is a variational autoencoder (VAE).
6. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: The method of interacting the deep reinforcement learning (DRL) model with the generative model is to use the simulated facial expression image output by the generative model for the reward function of the deep reinforcement learning (DRL) model, and the specific process is as follows: the facial expression generation algorithm framework of the deep reinforcement learning (DRL) model includes an Actor network and a Critic network, the target expression label and the current state representation vector are input into the Actor network, and the action vector of the next step is output; in the Critic network, the target expression label, the current state representation vector and the current action representation vector are input, and the evaluation value of the effect of the current action representation vector in the current state under the constraint of the target expression label is output; The current action representation vector acts on the current state to obtain the next state representation vector. After the next state representation vector passes through the generative model, the virtual facial expression image is compared with the target expression image to obtain the similarity vector. Finally, the reward function is used to calculate the immediate reward.
7. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning (DRL) model adopts the proximal policy optimization (PPO) algorithm. The input of the algorithm is the target expression label and the current state representation vector, and the output is the action representation vector that the robot should do under the current target expression label and the current state.
8. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning (DRL) model adopts the asynchronous advantage actor-critic (A3C) algorithm or the deep Q learning (DQN) algorithm, the input of which is the target expression label and the current state representation vector, and the output is the action representation vector that the robot should do under the current target expression label and the current state.
9. A robot facial expression generation method based on deep reinforcement learning according to claim 1, characterized in that: The generative model is replaced by the simulation platforms Gazebo and Unity3D.