Robot facial expression virtual simulation system

Abstract

The invention belongs to the technical field of robot face control, and relates to a robot face expression virtual simulation system which comprises a robot face, a generative model and a virtual simulation engine. A plurality of face motors are arranged in the face of the robot, and the generative adversarial network (GAN) is adopted as the generative model and comprises a generator and a discriminator; the method comprises the following specific working steps that firstly, a virtual simulation engine collects a data set composed of instruction signals of a face motor and real image signals of the face of the robot; 2, the generation model generates a virtual image of the face of the robot corresponding to the face motor instruction signal; and step 3, training and optimizing the facial expression of the robot and generating a control algorithm for controlling the face of the robot. By adopting the virtual simulation system, the instability of a physical simulation training model on an entity robot is overcome, the additional training and fine tuning work is reduced, the flexible material modeling problem is solved, and the deviation between simulation and practical application is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of robot facial control, and in particular to a robot facial expression virtual simulation system. Background Art

[0002] With the rapid development of robotics, especially in the field of facial expression generation and control, how to realistically simulate and control the facial expressions of robots has become one of the main challenges faced by researchers and engineers. Facial expression generation and control not only directly affects the performance of robots in human-computer interaction, but also determines their applicability in scenarios such as emotional expression and social interaction.

[0003] In order to achieve highly realistic facial expression generation, the more prominent method is the robot facial expression generation algorithm training method based on physical simulation engine. Researchers usually rely on physical simulation engines (such as Unity, UnrealEngine, etc.) to build a three-dimensional model of the robot head in a simulation environment and use the three-dimensional model to train the expression generation algorithm. These simulation engines can provide a controllable virtual environment to simulate the movement and facial features of the robot head, thereby supporting the development and verification of control algorithms.

[0004] However, this method has significant limitations in practical applications: 1. Sim2Real (Simulation to Reality) problem. Due to the inevitable differences between the physical simulation environment and the real world, the physical characteristics and behaviors in the physical simulation model are usually simplified and approximated, which leads to unstable performance of the model trained in the physical simulation environment on the actual robot; in order to make up for this difference, a lot of additional training and fine-tuning on the physical robot is usually required, which not only increases the development cost and time, but may also affect the robustness of the algorithm. 2. The head of the humanoid robot is usually made of flexible materials (such as silicone) to simulate human facial expressions and muscle movements. The complex physical properties of flexible materials are difficult to accurately model through traditional physical simulation engines. Traditional simulation models often have difficulty capturing the nonlinear elasticity, dynamic response and complex interactions with internal mechanical structures of flexible materials, resulting in significant differences between the facial expressions generated by simulation and the actual robot facial performance. This difference further exacerbates the Sim2Real problem, making it difficult for facial expression generation algorithms based on physical simulation to achieve the expected results in practical applications. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a robot facial expression virtual simulation system, which solves the problem that the model trained by the training method in the prior art in a physical simulation environment cannot fully and accurately reflect the physical robot and performs unstable on the actual robot; it also avoids the problem of needing to perform a large amount of additional training and fine-tuning on the physical robot; it also solves the problem that traditional simulation models are difficult to capture the nonlinear elasticity, dynamic response and complex interaction with the internal mechanical structure of flexible materials, resulting in significant differences between the facial expressions generated by simulation and the actual robot facial expressions, and it also avoids the problem that the model generated by simulation has significant deviations in real applications.

[0006] In order to solve the above technical problems, the present invention provides the following core technical solutions: a robot facial expression virtual simulation system, including a robot face, a generation model; a plurality of facial motors are arranged in the robot face; the generation model adopts a generative adversarial network (GAN), including a generator and a discriminator; the generator is used to receive the command signal of the facial motor and generate a virtual image of the robot face, and the discriminator is used to evaluate the difference between the generated virtual image and the real image of the robot face; and also includes a virtual simulation engine, and the specific working steps are as follows:

[0007] In the first step, the virtual simulation engine collects a data set consisting of command signals from facial motors and real image signals from the robot’s face;

[0008] In the second step, the generative model generates a virtual image of the robot’s face corresponding to the facial motor command signals;

[0009] The third step is to train and optimize the robot's facial expressions and generate a control algorithm for robot facial control.

[0010] Furthermore, based on the above core technical solution, the network structure of the generator is composed of a multi-layer convolutional neural network (CNN), the input is a multi-dimensional vector signal of facial motor instructions, and the output is a virtual image signal of the robot's face.

[0011] Furthermore, based on the above core technical solution, the network structure of the discriminator is composed of a multi-layer convolutional neural network (CNN), the input is a virtual image signal of the robot's face and a real image signal of the robot's face, and the output is comparative evaluation result information.

[0012] Furthermore, based on the above core technical solution, the specific method of the first step is: the data set collected by the virtual simulation engine is a plurality of signal instruction pairs, and the signal instruction pairs are composed of the real image signal of the robot's face and the corresponding command signal of the facial motor.

[0013] Furthermore, based on the above core technical solution, the specific method of the third step is: first, perform model training in a virtual simulation engine to minimize the difference between the virtual image of the robot's face and the real image; then, optimize and adjust the specific parameters of the generator according to the virtual image of the robot's face with the smallest difference; finally, regenerate a virtual image corresponding to the facial motor command signal.

[0014] Furthermore, based on the above core technical solution, the training process in the third step adopts an adversarial training strategy, that is, the specific parameters of the generator and the discriminator are optimized alternately until the virtual image generated by the generator is evaluated in the discriminator as being indistinguishable from the real image.

[0015] Furthermore, based on the above core technical solution, the production control algorithm in the third step is optimized using the reinforcement learning method. The reward function in the control algorithm is determined by the visual similarity between the generated virtual image and the real image, and repeated experiments and optimization adjustments are carried out.

[0016] Furthermore, based on the above core technical solution, the generative model adopts a variational autoencoder (VAE), a generative model based on an attention mechanism, or a generative model based on a diffusion strategy;

[0017] Furthermore, based on the above core technical solution, the training process in the third step adopts a multi-task learning strategy or a transfer learning strategy.

[0018] The robot facial expression virtual simulation system proposes a virtual simulation engine based on a generative adversarial network (GAN) or other generative models; the virtual simulation engine uses the robot motor command as input, generates a virtual image that is highly consistent with the real robot facial expression through the generative model, and significantly improves the simulation accuracy and the accuracy of expression generation. Since the corresponding virtual robot facial image is directly generated by inputting the robot facial motor command, it can ensure that the simulation result is highly consistent with the actual robot operation; this mechanism significantly reduces the Sim2Real error and is the key to achieving high-precision facial expression generation. The virtual robot facial image is directly generated by the generative model, without relying on complex three-dimensional modeling and physical modeling of flexible materials, thereby simplifying the training process of the expression generation algorithm; since the generative model is directly trained from the real data set, and the image generation process does not require complex physical modeling, the development cycle is significantly shortened and the robustness of the system is improved. The virtual simulation system of the present invention directly generates facial expression images through motor commands, simplifying the operation and control process. Users do not need to master the complex simulation environment settings and adjustment knowledge, and only need to input motor commands to generate the required expression images. This simplicity makes the application range of the system wider, especially suitable for the rapid development and iteration of expression generation algorithms. The generation model of the present invention is not limited to the GAN structure, but can also be extended to other generation models (such as VAE, attention mechanism, diffusion model, etc.), which enhances the flexibility and adaptability of the system; users can choose the most suitable generation model according to different application requirements to further enhance the adaptability of the system. This flexibility makes the present invention not only suitable for robot expression generation, but also can be extended to other complex multi-dimensional simulation scenes. It is not only suitable for humanoid robots, but also for robots of other forms (such as cartoon characters). As long as it involves the generation and control of facial expressions, the technology of the present invention can be used. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application:

[0020] Figure 1 This is a working principle diagram of Embodiment 1 of the present invention;

[0021] Figure 2 In the first embodiment of the present invention, under the same motor command, the similarity data between the generated image and the real image is compared;

[0022] Figure 3 This is an overview diagram of the generative model architecture based on a generative adversarial network according to Embodiment 1 of the present invention. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Glossary:

[0025] "Sim2Real": The full English name is Simulation to Reality, which means from simulation to reality. It is a research concept in the field of robotics and automation. It involves transferring the skills or strategies learned by the robot in a simulation environment to the application in the real world.

[0026] "Generative Adversarial Network (GAN)": It is a deep learning model (Generative Adversarial Network, referred to as GAN), proposed by Goodfellow et al. in 2014. It consists of a generator and a discriminator, which compete with each other during the training process to improve their respective performance. The dataset is composed of the command signal of the robot's facial motor and the real image signal of the robot's face; the generator receives the facial motor command signal and generates a virtual image, the discriminator evaluates the difference between the virtual image and the real image, and optimizes it through the adversarial training strategy; the output is to generate realistic robot facial expression images, which are used to train and optimize the robot facial expression control algorithm.

[0027] "Variational Autoencoder (VAE)": is a generative model, the full name of which is Variational Autoencoder, which is used to learn the latent representation of input data and can generate new data instances. VAE maps the data to a latent space distribution through the encoder, and then generates data by sampling from this latent distribution through the decoder. Unlike GAN, VAE uses a probabilistic generation framework, usually assuming that the distribution of the latent space is Gaussian, and uses variational inference to optimize model parameters. Variational Autoencoder (VAE) is used to collect facial expression datasets; using a probabilistic generation framework, the encoder maps the data to the latent space distribution, and the decoder samples and generates data from the latent distribution; generates new data instances for generating robotic facial expressions.

[0028] "Generative models based on attention mechanism": This type of model introduces an attention mechanism in the generation process, allowing the model to pay more attention to the important parts of the input data. The attention mechanism allows the model to dynamically focus on specific parts of the input sequence when generating each output element. This mechanism is particularly important in the field of natural language processing. For example, in tasks such as machine translation, text summarization, and image description generation, it can help the model better capture long-distance dependencies and contextual information. The generative model based on the attention mechanism collects a dataset of robot facial expressions; the introduction of the attention mechanism allows the model to pay attention to the important parts of the input data during the generation process, improving the accuracy and relevance of the generated expressions; the generated robot facial expression images contain more details and contextual information.

[0029] "Diffusion-based generative model": is an emerging generative model, the English name is Diffusion Models, which generates data through a step-by-step denoising method. The model first introduces noise to the data to generate disordered noisy data, and then trains a neural network to gradually reverse this process, that is, gradually reduce the impact of noise, and finally generate clear data. This method performs well in image and audio generation and can generate high-quality samples. A key advantage of the diffusion model is that it can generate data by precisely controlling the process of adding and removing noise, which makes the generation process more stable and predictable. The diffusion-based generative model collects a robot facial expression dataset; generates data through a step-by-step denoising method, first introducing noise, and then training a neural network to gradually reduce the impact of noise to generate clear data; generates high-quality robot facial expression images.

[0030] "Multi-task learning strategy": is a machine learning paradigm, the English name is Multi-task Learning, MTL, which improves the generalization ability of the model by learning multiple related tasks in the same model at the same time. In multi-task learning, the model tries to find a shared representation that is useful for all tasks, thereby improving the performance of each task. For example, in natural language processing, a multi-task model may learn multiple tasks such as language modeling, sentiment analysis, and named entity recognition at the same time. The key advantage of multi-task learning is that it allows the knowledge learned by the model on one task to be applied to other related tasks, which usually leads to better performance and less overfitting.

[0031] "Transfer learning strategy": is a machine learning paradigm, the English name is Transfer Learning, which allows the knowledge of a pre-trained model (usually trained on a large dataset) to be transferred to a different but related task. This method is particularly suitable for tasks with less data. In transfer learning, certain layers of the model (usually the lower layers) are trained to extract common features on the source task and then fine-tuned on the target task. For example, a convolutional neural network (CNN) pre-trained on a large-scale image dataset (such as ImageNet) can be transferred to a specific image classification task and adapted to the new task by further training the top layer of the network on that task. Transfer learning is widely used in fields such as computer vision, natural language processing, and speech recognition. It can help models learn quickly and improve performance on new tasks.

[0032] Embodiment 1:

[0033] A robot facial expression virtual simulation system includes a robot face, a generation model, and a plurality of facial motors are arranged in the robot face. The generation model adopts a generative adversarial network (GAN), including a generator and a discriminator; the generator is used to receive the command signal of the facial motor and generate a virtual image of the robot face, and the discriminator is used to evaluate the difference between the generated virtual image and the real image of the robot face; the network structure of the generator is composed of a multi-layer convolutional neural network (CNN), the input is a multi-dimensional vector signal of the command of the facial motor, and the output is a virtual image signal of the robot face. The network structure of the discriminator is composed of a multi-layer convolutional neural network (CNN), the input is a virtual image signal of the robot face and a real image signal of the robot face, and the output is comparative evaluation result information.

[0034] It also includes a virtual simulation engine. The specific working steps are as follows:

[0035] In the first step, the virtual simulation engine collects a data set consisting of command signals of facial motors and real image signals of the robot's face; the data set collected by the virtual simulation engine is multiple sets of signal-command pairs, and the signal-command pairs are composed of real image signals of the robot's face and corresponding command signals of facial motors.

[0036] In the second step, the generative model generates a virtual image of the robot’s face corresponding to the facial motor command signals;

[0037] The third step is to train and optimize the robot's facial expressions and generate control algorithms. First, the model is trained in a virtual simulation engine to minimize the difference between the virtual image and the real image of the robot's face. Data sets consisting of robot facial expression images and motor commands are collected; these data sets contain multiple facial expressions and corresponding motor command pairs, providing the input required for model training. The generator of the generative adversarial network (GAN) receives the command signal of the facial motor and generates a virtual image, and the discriminator evaluates the difference between the virtual image and the real image and optimizes it through an adversarial training strategy. The purpose of the training process is to minimize the difference between the generated virtual image and the real image, optimize the parameters of the generator, and finally generate realistic virtual images corresponding to the facial motor command signal, and generate an algorithm for controlling the robot's facial expressions. The output of the model training includes: an optimized generative model that can generate highly realistic robot facial expression images based on motor commands.

[0038] Then, the specific parameters of the generator are optimized and adjusted according to the virtual image of the robot face with the smallest difference; finally, the virtual image corresponding to the facial motor command signal is regenerated. The training process adopts an adversarial training strategy, that is, the specific parameters of the generator and the discriminator are optimized alternately until the virtual image generated by the generator is evaluated as no different from the real image in the discriminator. The production control algorithm is optimized using a reinforcement learning method. The reward function in the control algorithm is determined by the visual similarity between the generated virtual image and the real image, and repeated trials and optimization adjustments are performed. The control algorithms generated during the training process can be used to adjust the robot's facial expressions in real time and improve the accuracy and authenticity of expression generation.

[0039] like Figure 3 As shown in FIG. 1 , the generative model architecture constructed in the first embodiment follows the basic structure of the generative adversarial network, including a generator and a discriminator, and makes key improvements on this basis to adapt to the task of generating robot facial expressions. In this architecture, the generator no longer simply relies on a priori data distribution (usually random noise p z ) to generate an image. Instead, it receives a specific target expression vector x' tar and real robot face image I g-src As input; these inputs pass through the feature generator G feature The intermediate mask G is generated by a and G c The combination of these two masks finally forms the desired generated image I g-out The discriminator receives input I d_input And through the feature evaluator D feature Processing to obtain D aus and D scoreHere, the input image I d_input It may be from the self-built data set. d-src Or the I produced by the generator g-out . D aus is responsible for predicting the target expression label of the input image, while D score The authenticity of the image is scored with an unbounded real value to determine the authenticity of the image.

[0040] The generator network in the generative model is the core part, which aims to generate a target expression vector x' tar and real robot face image I g-src (i.e. from the actual data distribution p data A randomly sampled RGB image of size Where H and W represent the image height and width respectively) to accurately control facial expression features and generate facial images. Figure 3 The generator network model is presented in , which combines a single-channel G a and three-channel G c Through the interaction between these two masks, the target facial expression image I is effectively synthesized g_out The mathematical expression of the synthesis process is shown in the following formula 1:

[0041] I g_out =(1-G a )·G c +G a I g-src

[0042] When constructing the structure of the generator network model, this study adopted a method of fusing convolutional layers, residual layers, and deconvolutional layers (i.e., G feature ) method to extract high-level features, aiming to effectively achieve the generation of target expressions through this structural design. Specifically, the design of the network model allows it to receive two key inputs: the target expression vector x' tar and the original input image I g-src The specific operation is: the target expression vector x' tar After preprocessing, it is expanded to the same size as the input image. g-src The spatial dimension of x' tar ∈ (where n is a natural number) in order to effectively fuse with the original image to form a comprehensive input I g_input , whose dimensions are I g_input In the hierarchical structure (i.e., D feature ) after processing, the generator outputs the intermediate results I c_mask and I a_maskAmong them, I c_mask ∈[-1,1] 3×H×W , I a_mask ∈[0,1] 1×H×W By using formula 1, we finally get the same g-src Output image I of the same size g_out .

[0043] Discriminator network model structure:

[0044] In the framework of the generative model, the discriminator network also plays a vital role. It is responsible for accurately evaluating the input image I d_input The authenticity of the input image I d_input It can be the image I synthesized by the generator g_out , or it can be a real image I directly extracted from a real dataset d_src Similar to the generator, the discriminator’s network model structure design also uses multiple layers of shared convolutional layers to process the input image I d_input , and finally outputs two key indicators: one is for the input image I d_input The target expression vector prediction result D aus ; The second is about the input image I d_input The authenticity score of D score One point that needs to be emphasized here is that I g_src and I d_src Both are real robot face images from the self-built dataset, so both have corresponding real robot servo angle vectors (i.e., target expression labels). score , whose output dimension is set to Predicted label D aus With the target expression label x' tar The dimensions of are the same. In this study, the discriminator also draws on the design idea of ​​PatchGAN, that is, by evaluating the authenticity of each local area in the image, rather than directly making a consistent judgment on the entire image. This method can optimize the discriminator's sensitivity to image details and textures, and enhance the model's ability to discriminate image quality. Therefore, the authenticity score D score It is essentially a two-dimensional score matrix, in which each element is an unrestricted real value, representing the authenticity score of the corresponding area.

[0045] In the process of building a generative model, designing an effective loss function is crucial to training high-quality generators and discriminators. An appropriate loss function can not only guide the model to accurately learn the data distribution, but also ensure that the quality and authenticity of the generated images meet the expected standards. Considering the goals of this study and the particularity of the self-built dataset [that is, each robot facial image corresponds to a servo angle vector x=(x1,x2,…,x n )】, the present invention designs specific loss functions for the generator and the discriminator respectively according to these characteristics, so as to achieve the purpose of optimizing the model performance by refining the objectives.

[0046] Generator loss function:

[0047] The loss function of the generator G includes five core parts: adversarial truth loss, label consistency loss, image reconstruction loss, attention mask regularization loss and total variation loss. Among them, adversarial truth loss ensures that the generated image is visually difficult to distinguish from the real image by improving its authenticity score in the discriminator. Label consistency loss ensures that the expression of the generated image is highly consistent with the target label, enhancing the accuracy of the expression. Image reconstruction loss focuses on reducing the visual difference between the generated image and the real target image, improving the authenticity of the image details. Attention mask regularization loss reduces overfitting and improves the generalization ability of the model by constraining the attention mask. Finally, the total variation loss improves the visual effect of the image by smoothing the image and reducing noise, ensuring that the generated image reduces unnecessary noise and discontinuity while maintaining details. The common goal of these loss functions is to improve the authenticity and visual quality of the generated image, ensure the consistency of the generated image with the target expression label, and reduce noise and overfitting. Through the comprehensive application of these loss functions, the generator can generate high-quality images that are difficult to distinguish from real images.

[0048] The role of these generator loss functions and their specific forms are as follows:

[0049] The adversarial truth loss aims to motivate the image I synthesized by the generator D g-out After being evaluated by the discriminator D, it is considered a real image. This process improves the score D given by the discriminator D to the generated image. score This can be achieved by improving the quality of generated images, making them more difficult to distinguish from real images.

[0050] Its corresponding mathematical expression is

[0051] The goal of the label consistency loss is to ensure that the discriminator D is consistent with the generated image I g-out The predicted target expression vector D ausAnd the target expression vector x' input by the generator G tar This design aims to enhance the expression accuracy of the generated image and ensure that the generated result closely corresponds to the expected target. The mathematical expression corresponding to this loss function is

[0052] The core goal of image reconstruction loss is to reduce the size of the generated image I g_out Its corresponding target expression vector x' in the self-built dataset tar Real image The difference between the two can improve the visual quality and realism of the generated image. This approach aims to ensure that the generated image is not only similar to the real image on the surface, but also can reproduce every detail of the target expression in detail. The mathematical representation of the corresponding loss function is

[0053] The attention mask regularization loss is obtained by applying the generated attention mask I a_mask The strategy is based on optimizing the efficiency of mask utilization in the attention mechanism, ensuring that the generated mask can not only accurately reflect the target features, but also maintain the wide applicability of the model while preventing over-learning of specific training samples. The mathematical expression corresponding to this goal is

[0054] The total variation loss is used to enhance the generated attention mask I a_mask The smoothness of the image is improved to improve the visual effect and quality of the generated image. The corresponding mathematical expression is Among them, I i,j Represents image I a_mask The pixel value at position (i, j). The loss function of the generator G is The above five different loss terms are integrated by multiplying these loss terms with their respective weight coefficients and summing them up. The mathematical expression can be expressed as follows:

[0055]

[0056] Among them, λ g-i Represents the weight coefficient of the corresponding loss function part. In this embodiment, the weight coefficients are all 1.

[0057] Discriminator loss function:

[0058] The loss function of the discriminator D is composed of four key parts: real image loss, label consistency loss, generated image loss, and gradient penalty loss. The loss function structure of the discriminator D is designed to refine its ability to distinguish between real and generated images, and enhance training stability through gradient penalty. The real image loss strengthens the discriminator's recognition accuracy of entity samples, ensuring that it can efficiently recognize real images. The label consistency loss ensures that the discriminator accurately predicts the expression labels of real images and enhances its judgment ability in expression consistency. The generated image loss encourages the discriminator to effectively distinguish synthetic images and strengthen its ability to distinguish between true and false. The gradient penalty loss ensures the stable judgment ability of the discriminator by imposing gradient constraints, avoiding mode collapse during training, thereby improving the discriminative performance and training stability of the model as a whole. The combined effect of these loss functions enables the discriminator to more accurately evaluate and distinguish different types of images, thereby effectively monitoring and improving the quality of generated images. The role of these discriminative loss functions and their specific forms are described as follows:

[0059] The core purpose of real image loss is to improve the discriminator D for real samples I in the data set d_src Rating D score , which in turn motivates the discriminator D to improve its ability to accurately score real images. In this way, the discriminator learns to more effectively improve the scores of real images, thereby strengthening its ability to discriminate the authenticity of real images. The mathematical expression corresponding to this goal is

[0060] The label consistency loss aims to ensure that the discriminator D can accurately identify the real samples I in the dataset. d_src The predicted target expression vector D aus The real target expression vector x' corresponding to the real sample in the dataset tar By this method, the discriminator learns how to accurately predict the target expression vector of the real image, thereby improving the accuracy of judging the consistency of expression. The mathematical expression of this loss function is

[0061] The generated image loss aims to promote the discriminator D to recognize the image I generated by the generator G g_out are non-real images. By reducing the scores of these generated images in the discriminator, D score , the discriminator can effectively distinguish the difference between the generated image and the real image. This mechanism strengthens the discriminator's ability to distinguish authenticity and ensures that it can accurately determine which images are synthesized. The mathematical expression corresponding to this purpose is

[0062] The gradient penalty loss is designed to improve the stability of model training and prevent mode collapse during training. g_out Compared with the real image I g_src The interpolation samples between calculate the gradient, which makes the discriminator more stable when judging the authenticity of the image. The corresponding mathematical expression is

[0063] Among them, I inter =αI g_src +(1-α)I g_out , α is a random number uniformly distributed between 0 and 1.

[0064] The loss function of the discriminator D The four loss functions are integrated by weighted summation, and the final loss function The mathematical expression is as follows:

[0065]

[0066] Among them, λ d-i (i=1,2,…,4) are the weight coefficients of the corresponding loss function, in this embodiment, λ d-i (i=1,2,…,4) are all 1.

[0067] So far, in order to realize the generation model given the target expression vector x' tar Under the condition of x' tar The total loss function of the model incorporates the loss of the generator G. The loss of the discriminator D Together they form the core optimization goal of the generative adversarial network:

[0068] This total loss function embodies the core idea of ​​GANs, which is to optimize the generator and discriminator through a continuous adversarial process to generate high-quality expression vectors x' tar Matching images.

[0069] Generate model training process:

[0070] The generator and discriminator are trained separately and alternately (train the discriminator first --> then train the generator --> then train the discriminator...). The steps are as follows:

[0071] Training the discriminator:

[0072] Fixed the parameters of the generator and selected real samples from the dataset (real image I d_src and its corresponding x=(x1,x2,…,xn ), real image I d_src The output result D after inputting the discriminator score should be as large as possible, and D aus should be as close as possible to x=(x1,x2,…,x n ).

[0073] Fixed the parameters of the generator and selected real samples from the dataset (real image I d_src and its corresponding x=(x1,x2,…,x n ), that is, x' tar ), x' tar and a random real image I from the dataset g_src After inputting the generator, the generated image I is obtained g_out , I g_out Output result D after input to the discriminator score should be as small as possible, and D aus should be as close as possible to x=(x1,x2,…,x n ).

[0074] Training the generator:

[0075] Fixed the parameters of the discriminator, selected real samples from the data set (real image I d_src and its corresponding x=(x1,x2,…,x n ), that is, x' tar ), x' tar and a random real image I from the dataset g_src After inputting the generator, the generated image I is obtained g_out , I g_out Should be x' tar The corresponding real image I d_src As similar as possible, and generate image I g_out Output result D after input to the discriminator score should be as large as possible, and D aus should be as close as possible to x=(x1,x2,…,x n ).

[0076] The present embodiment provides a humanoid robot facial expression virtual simulation system based on a generative model. Compared with the prior art, this simulation system has significant advantages and innovative effects in many aspects. By using a generative adversarial network (GAN) or other generative models, the motor instructions are directly mapped to virtual robot facial expression images, which significantly improves the simulation accuracy and the accuracy of expression generation. Compared with the traditional method that relies on a physical simulation engine, the simulation system eliminates the difference problem between the simulation environment and the real world, and greatly reduces the Sim2Real error. Experimental data show that the similarity between the generated virtual image and the real robot facial expression exceeds 95%, which greatly improves the accuracy of expression generation. Figure 2 Shown is the similarity data comparison between the generated image and the real image under the same motor command.

[0077] Existing physical simulation methods usually require complex 3D modeling and a large amount of simulation calculations, which is not only time-consuming, but also difficult to complete algorithm optimization within a limited development cycle. This simulation system uses a virtual simulation engine based on a generative model, which simplifies the training process and significantly shortens the development cycle. Since the generative model is trained directly from the real data set and the image generation process does not require complex physical modeling, the development time can be shortened to about 50% of the traditional method, and the training efficiency is increased by more than 2 times.

[0078] The virtual simulation system of this simulation system directly generates facial expression images through motor commands, which simplifies the operation and control process. Users do not need to master the complex simulation environment settings and adjustment knowledge, and only need to input motor commands to generate the required expression images. This simplicity makes the system more widely applicable, especially suitable for the rapid development and iteration of expression generation algorithms.

[0079] Embodiment 2:

[0080] A robot facial expression virtual simulation system includes a robot face, a generation model, a plurality of facial motors are arranged in the robot face, and the generation model adopts a variational autoencoder (VAE); and also includes a virtual simulation engine. The specific working steps are as follows:

[0081] In the first step, the virtual simulation engine collects a data set consisting of command signals of facial motors and real image signals of the robot's face; the data set collected by the virtual simulation engine is multiple sets of signal-command pairs, and the signal-command pairs are composed of real image signals of the robot's face and corresponding command signals of facial motors.

[0082] In the second step, the generative model generates a virtual image of the robot’s face corresponding to the facial motor command signals;

[0083] The third step is to train and optimize the robot's facial expressions and generate a control algorithm. First, the model is trained in a virtual simulation engine to minimize the difference between the virtual image of the robot's face and the real image. A dataset consisting of robot facial expression images and motor commands is collected. These datasets contain a variety of facial expressions and corresponding motor command pairs, providing the model with the input required for training. The data is mapped to a latent space distribution through a variational autoencoder (VAE), and then the decoder is used to sample and generate data from the latent distribution, which is suitable for generating new data instances. Then, the specific parameters of the generation model are optimized and adjusted according to the virtual image of the robot's face with the smallest difference. Finally, a virtual image corresponding to the facial motor command signal is regenerated, and an algorithm for controlling the robot's facial expression is generated. The training process adopts a multi-task learning strategy; the production control algorithm is optimized using a reinforcement learning method, and the reward function in the control algorithm is determined by the visual similarity between the generated virtual image and the real image, and repeated trials and optimization adjustments are performed.

[0084] The present embodiment provides a humanoid robot facial expression virtual simulation system based on a generative model. Compared with the prior art, this simulation system has significant advantages and innovative effects in many aspects. Existing physical simulation methods usually require complex three-dimensional modeling and a large amount of simulation calculations, which are not only time-consuming, but also difficult to complete algorithm optimization within a limited development cycle. The present simulation system adopts a virtual simulation engine based on a generative model, which simplifies the training process and significantly shortens the development cycle. Since the generative model is trained directly from the real data set, and the process of generating images does not require complex physical modeling, the development time can be shortened to about 50% of the traditional method, and the training efficiency is increased by more than 2 times.

[0085] The virtual simulation system of this simulation system directly generates facial expression images through motor commands, which simplifies the operation and control process. Users do not need to master the complex simulation environment settings and adjustment knowledge, and only need to input motor commands to generate the required expression images. This simplicity makes the system more widely applicable, especially suitable for the rapid development and iteration of expression generation algorithms.

[0086] The generation model of this embodiment adopts a variational autoencoder (VAE), which enhances the flexibility and adaptability of the system. Users can select the generation model according to application requirements to further enhance the adaptability of the system.

[0087] The generation model of the present invention is not limited to the GAN structure, but can also be extended to other generation models (such as VAE, attention mechanism, diffusion model, etc.), which enhances the flexibility and adaptability of the system. Users can choose the most suitable generation model according to different application requirements to further improve the adaptability of the system. This flexibility makes the present invention not only suitable for robot expression generation, but also can be extended to other complex multi-dimensional simulation scenes. In particular, the scheme of the present invention is not only applicable to humanoid robots, but also to robots of other forms (such as cartoon characters). As long as it involves the generation and control of facial expressions, the technology of the present invention can be used.

[0088] In summary, the present invention significantly improves the accuracy and efficiency of simulation and expression generation through innovative generative model virtual simulation technology, simplifies the operation process, saves energy consumption and development resources, and shows high flexibility and environmental protection. Compared with the existing technology, the present invention overcomes many limitations of traditional physical simulation methods and provides a more efficient, more accurate and more widely applicable solution.

[0089] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A robot facial expression virtual simulation system, comprising a robot face and a generative model; the robot face is provided with a plurality of facial motors; the generative model adopts a generative adversarial network (GAN), comprising a generator and a discriminator; the generator is used to receive a command signal of the facial motor and generate a virtual image of the robot face, and the discriminator is used to evaluate the difference between the generated virtual image and the real image of the robot face; the characteristics are: It also includes a virtual simulation engine. The specific working steps are as follows: In the first step, the virtual simulation engine collects a data set consisting of the command signal of the facial motor and the real image signal of the robot face; In a second step, the generative model generates a virtual image of the robot face corresponding to the facial motor command signal; The third step is to train and optimize the robot's facial expressions and generate a control algorithm for robot facial control.

2. A robot facial expression virtual simulation system according to claim 1, characterized in that: The network structure of the generator is composed of a multi-layer convolutional neural network (CNN), the input is a multi-dimensional vector signal of the instructions of the facial motor, and the output is a virtual image signal of the robot's face.

3. A robot facial expression virtual simulation system according to claim 1, characterized in that: The network structure of the discriminator is composed of a multi-layer convolutional neural network (CNN), the input is the virtual image signal of the robot face and the real image signal of the robot face, and the output is comparison evaluation result information.

4. A robot facial expression virtual simulation system according to claim 1, characterized in that: The specific method of the first step is: the data set collected by the virtual simulation engine is a plurality of signal instruction pairs, and the signal instruction pairs are composed of the real image signal of the robot's face and the corresponding instruction signal of the facial motor.

5. A robot facial expression virtual simulation system according to claim 1, characterized in that: The specific method of the third step is: first, perform model training in the virtual simulation engine to minimize the difference between the virtual image and the real image of the robot's face; then, optimize and adjust the specific parameters of the generator according to the virtual image of the robot's face with the smallest difference; finally, regenerate a virtual image corresponding to the facial motor command signal.

6. A robot facial expression virtual simulation system according to claim 1, characterized in that: The training process in the third step adopts an adversarial training strategy, that is, the specific parameters of the generator and the discriminator are optimized alternately until the virtual image generated by the generator is evaluated in the discriminator as being indistinguishable from the real image.

7. A robot facial expression virtual simulation system according to claim 1, characterized in that: The production control algorithm in the third step is optimized using a reinforcement learning method, and the reward function in the control algorithm is determined by the visual similarity between the generated virtual image and the real image, and repeated trials and optimization adjustments are performed.

8. A robot facial expression virtual simulation system according to claim 1, characterized in that: The generative model adopts a variational autoencoder (VAE), a generative model based on an attention mechanism, or a generative model based on a diffusion strategy.

9. A robot facial expression virtual simulation system according to claim 1, characterized in that: The training process in the third step adopts a multi-task learning strategy or a transfer learning strategy.

Images