A high-simulation robot facial expression control method

Abstract

The present application relates to the technical field of robot control algorithm, in particular to a high-simulation robot facial expression control method.The high-simulation robot facial expression control method comprises collecting user photos, generating different facial expressions of a high-simulation robot, obtaining facial expression-full-view image sequence-rudder rotation angle of the high-simulation robot, training user photos and facial expression-full-view image sequence of the high-simulation robot using a deep neural network, obtaining a user-high-simulation robot training model, and using the trained user-high-simulation robot training model and facial expression-rudder rotation angle of the high-simulation robot to output facial expression sequence of the user in real time and drive the high-simulation robot.The present application increases the universality of the expression driving of the simulation robot, solves the phenomenon that the facial expression of the high-simulation robot is difficult to quickly adjust and control, has high universality, and is suitable for facial control of various different high-simulation robots.

Description

Technical Field

[0001] This invention relates to the field of robot control algorithm technology, and in particular to a highly realistic method for controlling facial expressions in robots. Background Technology

[0002] 4D data refers to data composed of three-dimensional data plus a fourth time dimension. Three-dimensional data stores a three-dimensional representation of geometric data and is used to calculate and render the final three-dimensional image. Continuously acquiring three-dimensional data over a series of time periods expands the time dimension of the three-dimensional data, resulting in continuous time-series three-dimensional data; this is called 4D data acquisition.

[0003] Existing robot technologies typically include a motion drive mechanism, an expression drive mechanism, a facial shell, and elastic facial skin. The motion control system comprises a servo controller and multiple servos. Information perceived by a multi-sensor system is output to a main controller for processing. The main controller outputs corresponding command information to the servo controller, which outputs PWM pulses to drive the corresponding servos to rotate to a designated position. The servos then drive the motion drive mechanism to move the lip skin, enabling the robot to mimic lip movements. The servos also drive the expression drive mechanism to move the elastic facial skin, achieving various facial expressions. Alternatively, real-time imitation of human facial expressions can be achieved through robot simulation. Key points of the user are extracted using Kinect sensors and an SDK. Facial expressions are recognized and classified using an artificial neural network. A minimum layer of the neural network has been developed. The Kinect output array for each expression is collected to create a training dataset. Then, the optimal network for real-time recognition is found and trained. Finally, two compilers are connected to complete the imitation process. Existing robots have the following drawbacks:

[0004] 1. Facial expression and motion design still requires manual adjustment in the time domain, which incurs significant labor costs and low operational efficiency. Furthermore, the continuity of the adjusted servo rotation sequence is not high, requiring manual correction under prolonged continuous data input.

[0005] 2. Due to the limited number of servo motors in a humanoid robot, it's difficult to achieve a one-to-one correspondence between facial landmarks and corresponding positions on the robot's face. Decoupling facial expressions into different emotional dimensions using neural networks results in poor interpretability. Furthermore, the use of a small number of people to drive a single robot facial expression leads to low generalization ability. The robot performs poorly when driven by different users, limiting its potential for widespread application. Summary of the Invention

[0006] In view of the above-mentioned disadvantages of the prior art, the purpose of the present application is to provide a high-simulation robot facial expression control method, which increases the universality of the simulation robot expression driving by 4D data acquisition and training of the high-simulation robot face, has high universality, is suitable for various high-simulation robot face control, and has the characteristics of wide application potential.

[0007] To achieve the above-mentioned purpose and other related purposes, the present application provides a high-simulation robot facial expression control method, comprising:

[0008] S1, collecting a user photo P0;

[0009] S2, generating different facial expressions of the high-simulation robot to obtain the facial expression-full view image sequence-rudder rotation angle A-P-Π of the high-simulation robot;

[0010] S3, training the user photo P0 and the facial expression-full view image sequence A-P of the high-simulation robot using a deep neural network to obtain a user-high-simulation robot training model M;

[0011] S4, using the trained user-high-simulation robot training model M and the facial expression-rudder rotation angle A-Π of the high-simulation robot, outputting the facial expression sequence of the user in real time and driving the high-simulation robot.

[0012] In an embodiment of the present application, the step S2 of generating different facial expressions of the high-simulation robot to obtain the facial expression-full view image sequence-rudder rotation angle A-P-Π of the high-simulation robot comprises:

[0013] S21, using the driving control system of the high-simulation robot, randomly generating the rotation angle of each rudder of the high-simulation robot to generate different facial expressions A of the high-simulation robot i , recording the n rudder rotation angles Π i ={θ1,θ2,...,θ n} of the high-simulation robot at the current time, and recording the corresponding data pairs of the facial expression A and the rudder rotation angle Π of the high-simulation robot;

[0014] S22, using a 4D acquisition system to capture the full view image sequence P i ={p1,p2,...,p k} of k facial photos of the high-simulation robot at different angles at the current time, k≥6, to obtain the facial expression-full view image sequence-rudder rotation angle A-P-Π of the high-simulation robot.

[0015] In an embodiment of the present application, the step S3 of training the user-photo P0 and the facial expression-full view image sequence A-P of the high-simulation robot using the deep neural network to obtain the user-high-simulation robot training model M includes:

[0016] The auto-encoder training data pair of the generative adversarial network using unsupervised learning:

[0017] S31, encode the facial expression A of the high-simulation robot and the user photo P0 using the same encoder En to obtain the facial data L1 and L2 with consistent hidden image features, which is expressed as L1=En(A) and L2=En(P0);

[0018] S32, use two decoders De1 and De2 respectively to restore the obtained facial data L1 and L2 with consistent hidden image features to the full view image sequence A' and the user photo P0', which is expressed as A'=De1(L1) and P0'=De2(L2);

[0019] S33, optimize the encoder En and the decoders De1 and De2 in the way of gradient descent by minimizing the interpolation of the facial expression A of the high-simulation robot and the full view image A', the user photo P0 and P0' to obtain the user-high-simulation robot training model M.

[0020] In an embodiment of the present application, the step S3 of training the user-photo P0 and the facial expression-full view image sequence A-P of the high-simulation robot using the deep neural network to obtain the user-high-simulation robot training model M includes:

[0021] The deep fake network training data pair of the generative adversarial network using unsupervised learning:

[0022] S311, input a user photo P0 and a full view image sequence P of the high-simulation robot into the deep fake network, and train the deep fake network model in the way of gradient descent;

[0023] S312, input the user facial expression picture sequence into the deep fake network model to generate the multi-view image R of the high-simulation robot, which is expressed as R=M(P0,P).

[0024] In an embodiment of the present application, the step S4 of using the trained user-high-simulation robot training model M and the facial expression-rudder rotation angle A-Π of the high-simulation robot to output the user facial expression sequence in real time and drive the high-simulation robot includes:

[0025] S41, shoot the user facial expression picture sequence, i.e. the expression video, to obtain the user facial expression picture sequence P';

[0026] S42, using the user-high simulation robot training model M, replace the user face with a robot face video sequence R', which is expressed as: R'=M(P0, P');

[0027] S43, separate the high simulation robot face video sequence R' into t different pictures {r1, r2,..., rt}, and match each separated picture r t with the full-view image sequence P i , find the closest facial expression of the high simulation robot in the full-view image sequence P i . i

[0028] S44, find the corresponding rudder rotation angle A in the high simulation robot full-view image sequence-rudder rotation angle database P-Π. Use the rudder rotation angle A to drive the high simulation robot face movement and generate the corresponding high simulation robot facial expression.

[0029] In an embodiment of the present application, the number of t different pictures separated in step S43 is determined by the length of the expression video and the rudder rotation frequency of the high simulation robot.

[0030] In an embodiment of the present application, the high simulation robot face video sequence R' separated per second in step S43 is not less than ten different pictures.

[0031] In an embodiment of the present application, in step S43, the closest facial expression of the high simulation robot is found in the full-view image sequence P i by a feature matching algorithm including SIFT algorithm or SURF algorithm.

[0032] The present application also provides a high simulation robot facial expression control system, comprising:

[0033] A collection unit for collecting user photos P0;

[0034] A facial expression generation unit for generating different facial expressions of a high simulation robot, obtaining facial expression-full-view image sequence-rudder rotation angle A-P-Π of the high simulation robot.

[0035] A training unit for training user photos P0 and facial expression-full-view image sequence A-P of the high simulation robot using a deep neural network, obtaining a user-high simulation robot training model M.

[0036] ​​​The face expression sequence output unit is configured to output a face expression sequence of the user in real time and drive the high-simulation robot by using the trained user-high-simulation robot training model M and the face expression-rudder rotation angle A-Π of the high-simulation robot.

[0037] The application further provides an electronic device including a processor and a memory, wherein the memory stores program instructions, and the processor executes the program instructions to implement the high-simulation robot face expression control method.

[0038] As described above, the high-simulation robot face expression control method has the following beneficial effects:

[0039] The high-simulation robot face expression control method collects and trains 4D data of the high-simulation robot and drives the face of the high-simulation robot in real time.

[0040] The high-simulation robot face expression control method effectively shortens the time required for adjusting the rudder sequence design of the high-simulation robot face expression in the later stage through the data preparation and processing in the early stage. The high-simulation robot face expression control method drives the high-simulation robot in real time by using the real face expression sequence of the user, guarantees the dynamic reality of the face of the high-simulation robot, and has high fidelity. The high-simulation robot face expression control method trains the user-high-simulation robot face changing model by using 4D data, and has strong universality.

[0041] The high-simulation robot face expression control method drives the user in real time in the online stage, and realizes the control of the face expression of the high-simulation robot. The high-simulation robot face expression control method drives the high-simulation robot by using the face expression sequence of the user, and significantly improves the efficiency of the face expression editing of the high-simulation robot.

[0042] The high-simulation robot face expression control method separates the time-consuming 4D data collection and training process and the user real-time driving process, so that the method has good implementation speed and high fidelity when actually used, and improves the design efficiency of the face expression of the high-simulation robot.

[0043] The high-simulation robot facial expression control method constructs a face-changing neural network algorithm model of a user-simulation robot, and effectively improves the problem of dynamic distortion of robot expression by changing the face of the user into the face of the high-simulation robot in real time.

[0044] Different users of the high-simulation robot facial expression control method can quickly drive the same robot, and the same user can also quickly drive different types of robots, so that the facial expression design scheme is more flexible and has more application potential. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 A high-simulation robot facial expression control method provided for an embodiment of the application has a work flow chart.

[0046] Figure 2 A high-simulation robot facial expression control method provided for an embodiment of the application has a user face replaced by a robot face video sequence picture. (a) is a simulation robot picture, (b) is a user face information picture, and (c) is a user face information picture after face changing.

[0047] Figure 3 A high-simulation robot facial expression control method provided for an embodiment of the application has a full-view image sequence picture.

[0048] Figure 4 A high-simulation robot facial expression control method provided for an embodiment of the application has a 4D acquisition system structure diagram.

[0049] Figure 5 A high-simulation robot facial expression control method provided for another embodiment of the application has a work flow chart.

[0050] Figure 6 A high-simulation robot facial expression control method provided for an embodiment of the application has a work flow chart of step S2.

[0051] Figure 7 A high-simulation robot facial expression control method provided for an embodiment of the application has a work flow chart of step S3.

[0052] Figure 8 A high-simulation robot facial expression control method provided for another embodiment of the application has a work flow chart of step S3.

[0053] Figure 9 A high-simulation robot facial expression control method provided for an embodiment of the application has a work flow chart of step S4.

[0054] Figure 10 A structural block diagram of a high-simulation robot facial expression control system provided for an embodiment of the present application.

[0055] Figure 11 A structural principle block diagram of an electronic device provided for an embodiment of the present application.

[0056] Figure 12 A structural principle block diagram of a computer readable storage medium provided for an embodiment of the present application.

[0057] Element number explanation

[0058] 1 acquisition unit

[0059] 2 facial expression generation unit

[0060] 3 training unit

[0061] 4 facial expression sequence output unit

[0062] 10 processor

[0063] 20 memory

[0064] 30 computer readable storage medium

[0065] 40 computer instruction DETAILED DESCRIPTION

[0066] The present application will be described in more detail by the following specific examples. Other advantages and benefits of the present application will be easily understood by those skilled in the art from the disclosure of this specification. The present application can also be implemented or applied in other different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0067] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only show the components related to the present application in the diagrams, not the number, shape and size of the components when actually implemented. The shape, number and ratio of each component when actually implemented can be changed arbitrarily, and the layout pattern of the components can also be more complex.

[0068] Please refer to Figure 1 , Figure 1The working flow chart of a high-simulation robot facial expression control method is provided for an embodiment of the application. The application provides a high-simulation robot facial expression control method, which comprises 4D data acquisition and training of a high-simulation robot and real-time driving of the facial expression of the high-simulation robot. The 4D data acquisition and training is an offline stage and is a preparatory work for the high-simulation robot facial expression control. In this stage, the expression information of the high-simulation robot and the photos of users are collected, and the collected data is trained on a deep learning model to obtain a training model capable of real-time facial transformation, and a database of the rotation angle of each rudder corresponding to the generated expression information is generated. The real-time driving of the user is an online stage, and the high-simulation robot facial expression control is realized. In this stage, the facial expression sequence of the user is converted into a facial expression video sequence of the robot through the training model in the previous stage, and the closest expression in the 4D database is searched for each frame through the feature point matching method, and the rudder data of the expression is used to drive the simulation robot to move, so that the high-simulation robot facial expression control with high fidelity and fast realization is realized.

[0069] Please refer to Figure 2 、 Figure 3 、 Figure 4 、 Figure 5 , Figure 2 The user's face of the high-simulation robot facial expression control method provided by the embodiment of the application is replaced by a robot face video sequence picture. (a) is a simulation robot picture, (b) is a user face information picture, and (c) is a user face information picture after face replacement. Figure 3 The full-view image sequence picture of the high-simulation robot facial expression control method provided by the embodiment of the application. Figure 4 The 4D acquisition system structure diagram of the high-simulation robot facial expression control method provided by the embodiment of the application. Figure 5 The working flow chart of a high-simulation robot facial expression control method is provided for another embodiment of the application. The hardware structure of the high-simulation robot facial expression control method comprises a 4D face acquisition system, a high-simulation robot, a computing terminal, a storage and the like. The 4D acquisition system needs to be built in the early stage of the application, and is used to acquire a three-dimensional model of the robot face in a time sequence, as shown in Figure 3 In order to realize facial acquisition in a series of time periods, a plurality of pictures of the face at different angles need to be used. The current 4D acquisition system mainly uses 6 to 8 cameras to simultaneously shoot the acquisition data at a moment, and other 4D data acquisition systems, for example, a radiation field method (NeRF) can also be used, as long as 4D data can be acquired.

[0070] The high-simulation robot facial expression control method specifically comprises:

[0071] Step S1, collect user photo P0, specifically, one or more user face photos can be collected as the reference for replacement. One user front face photo can be collected, and the resolution of the front face photo is not less than 256*256 pixels.

[0072] Step S2, generate different facial expressions of the high-simulation robot, and obtain the facial expression-full-view image sequence-rudder rotation angle A-P-Π of the high-simulation robot. Specifically, the rotation angles of each rudder of the high-simulation robot are randomly generated using the driving control system of the high-simulation robot, different facial expressions are formed, the rotation angles of each rudder at the current time are recorded, the data pair of the rudder rotation angle-facial expression of the high-simulation robot is formed, and the data is stored in the storage.

[0073] Step S3, train the user photo P0 and the facial expression-full-view image sequence A-P of the high-simulation robot using the deep neural network, and obtain the user-high-simulation robot training model M.

[0074] Step S4, use the trained user-high-simulation robot training model M and the facial expression-rudder rotation angle A-Π of the high-simulation robot to output the facial expression sequence of the user in real time and drive the high-simulation robot.

[0075] Please refer to Figure 6 , Figure 6 The working flow chart of step S2 of the high-simulation robot facial expression control method provided in the embodiment of the application. The generation of different facial expressions of the high-simulation robot in step S2 obtains the facial expression-full-view image sequence-rudder rotation angle A-P-Π of the high-simulation robot, which includes:

[0076] Step S21, randomly generate the rotation angles of each rudder of the high-simulation robot using the driving control system of the high-simulation robot, and generate different facial expressions A of the high-simulation robot i Record the rotation angles of the n rudders of the high-simulation robot at the current time i ={θ1,θ2,...,θ n}, and record the corresponding data pair of the facial expression A and the rudder rotation angle of the high-simulation robot.

[0077] Step S22, capture the full-view image sequence P i ={p1,p2,...,p k}, k≥6, to obtain the facial expression of the high-simulation robot - full-view image sequence - rudder rotation angle A-P-Π. Since the time of each collection is very short, and the data collection is fully automated, the above collection process can be repeated a large number of times. Specifically, the facial expression data A that the high-simulation robot can reach is collected to obtain the facial expression - full-view image sequence - rudder rotation angle data A-P-Π.

[0078] Please refer to Figure 7 , Figure 7 The working flow chart of step S3 of a high-simulation robot facial expression control method provided by an embodiment of the present application. The use of a deep neural network in step S3 to train the user photo P0 and the facial expression - full-view image sequence A-P of the high-simulation robot obtains a user - high-simulation robot training model M, which includes:

[0079] The use of the autoencoder training data pair of the generative adversarial network of unsupervised learning enables the deep neural network to predict a realistic simulation robot expression image output sequence from the facial expression sequence of the user, and obtain a user - high-simulation robot training model M, which is composed of an encoder En and two decoders De1 and De2, M = {En, De1, De2}.

[0080] Step S31, encode the facial expression A of the high-simulation robot and the user photo P0 using the same encoder En to obtain facial data L1, L2 with consistent hidden image features, which is expressed as L1 = En(A), L2 = En(P0).

[0081] Step S32, use two decoders De1 and De2 respectively to restore the obtained facial data L1, L2 with consistent hidden image features to full-view image sequence A' and user photo P0', which is expressed as: A' = De1(L1), P0' = De2(L2).

[0082] Step S33, optimize the encoder En and the decoders De1 and De2 in the gradient descent manner by minimizing the interpolation of the facial expression A of the high-simulation robot and the full-view image A', the user photo P0 and P0', to obtain a user - high-simulation robot training model M.

[0083] Please refer to Figure 8 , Figure 8 The working flow chart of step S3 of a high-simulation robot facial expression control method provided by another embodiment of the present application. The use of a deep neural network in step S3 to train the user photo P0 and the facial expression - full-view image sequence A-P of the high-simulation robot obtains a user - high-simulation robot training model M, which includes:

[0084] Deep fake network training data pairs of a generative adversarial network using unsupervised learning:

[0085] Step S311, input a user photo P0 and a full-view image sequence P of a high-simulation robot into the deep fake network, and train the deep fake network model in a gradient descent manner.

[0086] Step S312, input a user facial expression picture sequence into the deep fake network model, and generate a multi-view image R of a high-simulation robot corresponding to the user facial expression picture sequence, which is expressed by a formula: R = M(P0, P).

[0087] The input and output of the deep neural network are shown in Figure 2 (b) and Figure 2 (c), to generate a corresponding high-simulation robot full-view image sequence-rudder rotation angle data P-∏ database for later use.

[0088] Please refer to Figure 9 , Figure 9 The workflow diagram of step S4 of a high-simulation robot facial expression control method provided in the embodiments of the present application. In step S4, the trained user-high-simulation robot training model M and the facial expression-rudder rotation angle A-∏ of the high-simulation robot are used to output a user facial expression sequence in real time and drive the high-simulation robot, which includes:

[0089] Step S41, shoot a user facial expression picture sequence, i.e., an expression video, to obtain a user facial expression picture sequence P′.

[0090] Step S42, use the user-high-simulation robot training model M to replace the user face with a robot face video sequence R′, which is expressed by a formula: R′ = M(P0, P′).

[0091] Step S43, separate the high-simulation robot face video sequence R′ into t different pictures {r1, r2,..., rt}, and match each separated picture rt with a full-view image sequence P t , i , i , i , to find the closest high-simulation robot facial expression in the full-view image sequence P Specifically, the number of t different pictures is determined by the length of the expression video and the rudder rotation frequency of the high-simulation robot. The high-simulation robot face video sequence R′ is separated into no less than ten different pictures per second. The image matching algorithm of the SIFT algorithm and the SURF algorithm can be used, but is not limited to them, to match each separated picture rt with the full-view image sequence P iFinding the closest facial expression of the high-simulation robot in the database

[0092] Step S44, finding the facial expression in the high-simulation robot full-view image sequence-rudder rotation angle P-Π database The corresponding rudder rotation angle Using the rudder rotation angle Drive the high-simulation robot facial movement to generate the corresponding high-simulation robot facial expression.

[0093] In the actual real-time driving process, steps S41 to S44 are repeatedly performed to realize real-time driving and continuous output of the high-simulation robot expression.

[0094] Please refer to Figure 10 , Figure 10 The structure block diagram of a high-simulation robot facial expression control system provided by the embodiment of the present application. The present application also provides a high-simulation robot facial expression control system, which comprises a collection unit 1, a facial expression generation unit 2, a training unit 3, and a facial expression sequence output unit 4. The collection unit 1 is used to collect user photos P0. The facial expression generation unit 2 is used to generate different facial expressions of the high-simulation robot, and obtain the facial expression-full-view image sequence-rudder rotation angle A-P-Π of the high-simulation robot. The training unit 3 is used to train the user photos P0 and the facial expression-full-view image sequence A-P of the high-simulation robot using a deep neural network, and obtain a user-high-simulation robot training model M. The facial expression sequence output unit 4 is used to output the facial expression sequence of the user in real time and drive the high-simulation robot by using the trained user-high-simulation robot training model M and the facial expression-rudder rotation angle A-Π of the high-simulation robot.

[0095] Please refer to Figure 11 , Figure 11A structural principle block diagram of an electronic device is provided for the embodiments of the present application. The present application further provides an electronic device, which comprises a processor 10 and a memory 20, the memory 20 stores program instructions, and the processor 10 runs the program instructions to realize the high-simulation robot facial expression control method described above. The processor 10 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; the memory 20 can contain a random access memory (RAM), and can also include a non-volatile memory (Non-Volatile Memory), such as at least one disk memory. The memory 20 can also be an internal memory of the random access memory (RAM) type, and the processor 10 and the memory 20 can be integrated into one or more independent circuits or hardware, such as an application specific integrated circuit (ASIC). It should be noted that the computer program in the memory 20 described above can be realized in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application.

[0096] Figure 12 , Figure 12A structural principle block diagram of a computer readable storage medium provided by the embodiment of the present application is provided. The present application further provides a computer readable storage medium 30, which stores computer instructions 40 for enabling the computer to perform the high-simulation robot facial expression control method described above. The computer readable storage medium 30 can be an electronic medium, a magnetic medium, an optical medium, an electromagnetic medium, an infrared medium, or a semiconductor system or a propagation medium. The computer readable storage medium 30 can also include a semiconductor or solid-state memory, a magnetic tape, a removable computer disk, a random access memory (RAM), a read-only memory (ROM), a hard disk, and an optical disk. The optical disk can include a compact disk-read only memory (CD-ROM), a compact disk-read / write (CD-RW), and a DVD.

[0097] In summary, the high-simulation robot facial expression control method of the present application performs 4D data acquisition and training on the high-simulation robot, and performs real-time driving on the face of the high-simulation robot. The present application can significantly improve the realism, universality, and motion editing speed of the high-simulation robot expression. The present application increases the universality of the simulation robot expression driving, solves the phenomenon that the high-simulation robot facial expression is difficult to quickly adjust and control, has high universality, is suitable for various high-simulation robot facial control, and has the potential for wide application.

[0098] The above embodiments only exemplarily illustrate the principles and effects of the present application, and are not used to limit the present application. Any person skilled in the art can modify or change the above embodiments without departing from the spirit and scope of the present application. Therefore, all equivalent modifications or changes completed by those skilled in the art without departing from the spirit and technical thought of the present application should be covered by the claims of the present application.

Claims

1. A method for controlling facial expressions in a highly realistic robot, characterized in that, include: S1, Collect user photos P0; S2. Generate different facial expressions of the highly realistic robot to obtain the facial expression-full-view image sequence-servo motor rotation angle AP-Π of the highly realistic robot; S3. Using a deep neural network, train the user photo P0 and the facial expression-full-view image sequence AP of the highly realistic robot to obtain the user-highly realistic robot training model M, which includes: Training data pairs for autoencoders using unsupervised learning generative adversarial networks: S31. Encode the facial expression A of the highly realistic robot and the user photo P0 using the same encoder En to obtain facial data L1 and L2 with consistent hidden image features. The formula is L1 = En(A) and L2 = En(P0). S32. Using two decoders De1 and De2 respectively, the obtained facial data L1 and L2 with consistent hidden image features are recovered into a full-view image sequence A′ and a user photo P0′ respectively. The formula is expressed as: A′=De(L1), P0′=De2(L2); S33. By minimizing the interpolation of the facial expression A and the full-view image A′ of the high-fidelity robot, and the user photos P0 and P0′, the encoder En and decoder De1 and De2 are optimized along the gradient descent to obtain the user-high-fidelity robot training model M. S4. Using the trained user-high-fidelity robot training model M and the high-fidelity robot's facial expression-servo rotation angle A-Π, the user's facial expression sequence is output in real time and used to drive the high-fidelity robot, including: S41. Capture a sequence of user facial expression images to obtain the facial expression video, resulting in the user facial expression image sequence P′. S42. Using the user-high simulation robot training model M, the user's face is replaced with a robot face video sequence R', which is expressed by the formula: R'=M(P0,P'). S43. Separate the highly realistic robot facial video sequence R' into t different images {r1, r2, ..., r...} t }, separate each image r i With full-view image sequence P i Matching is performed on the full-view image sequence P. i The most realistic facial expressions found in the search results for a highly realistic robot. S44. Find the facial expression in the full-view image sequence-servo rotation angle P-Π database of the highly realistic robot. The corresponding servo rotation angle Using the servo motor rotation angle Drive the facial movements of a highly realistic robot to generate corresponding facial expressions.

2. The method for controlling facial expressions in a highly realistic robot according to claim 1, characterized in that, Step S2 generates different facial expressions of the highly realistic robot, resulting in a high-fidelity robot facial expression-full-view image sequence-servo motor rotation angle AP-Π, including: S21. Using the drive control system of a high-fidelity robot, randomly generate the rotation angles of each servo motor of the high-fidelity robot, and generate different facial expressions of the high-fidelity robot. i Record the rotation angles Π of n servo motors of the highly realistic robot at the current moment. i ={θ1,θ2……θ n Record the corresponding data pairs between the facial expression A of the highly realistic robot and the rotation angle Π of the servo motor; S22. Using a 4D acquisition system, capture a full-view image sequence P of k facial photos of a highly realistic robot from different angles at the current moment. i ={p1,p2,...,p k }, k≥6, to obtain the facial expression-full-view image sequence-servo motor rotation angle AP-Π of the highly realistic robot.

3. The method for controlling facial expressions in a highly realistic robot according to claim 2, characterized in that, Step S3 involves training a deep neural network on the user's photo P0 and the facial expression-full-view image sequence AP of the highly realistic robot to obtain the user-highly realistic robot training model M, which includes: Training data pairs for deep spoofing networks using unsupervised generative adversarial networks: S311. Input a user photo P0 and a full-view image sequence P of a highly realistic robot into the deep forgery network, and train the deep forgery network model by gradient descent. S312. Input the user's facial expression image sequence into the deep fake network model to generate the corresponding high-fidelity robot multi-view image R, which is expressed by the formula: R = M(P0, P).

4. The method for controlling facial expressions in a highly realistic robot according to claim 1, characterized in that: The number of different images separated into t images in step S43 is determined by the length of the facial expression video and the rotation frequency of the servo motor of the high-fidelity robot.

5. The method for controlling facial expressions in a highly realistic robot according to claim 1, characterized in that: In step S43, the highly realistic robot facial video sequence R′ per second is separated into no less than ten different images.

6. The method for controlling facial expressions of a highly realistic robot according to claim 1, characterized in that: In step S43, the feature matching algorithm, including the SIFT algorithm or the SURF algorithm, is used in the full-view image sequence P. i The most realistic facial expressions found in the search results for a highly realistic robot.

7. A highly realistic robot facial expression control system, implemented by any one of the highly realistic robot facial expression control methods according to claims 1 to 6, characterized in that, include: The acquisition unit is used to acquire user photos P0; The facial expression generation unit is used to generate different facial expressions of the highly realistic robot, and obtain the facial expression-full-view image sequence-servo motor rotation angle AP-Π of the highly realistic robot. The training unit is used to train a user photo P0 and a high-fidelity robot facial expression-full-view image sequence AP using a deep neural network to obtain a user-high-fidelity robot training model M. The facial expression sequence output unit is used to output the user's facial expression sequence in real time and drive the high-simulation robot by utilizing the trained user-high-fidelity robot training model M and the facial expression-servo rotation angle A-Π of the high-fidelity robot.

8. An electronic device comprising a processor and a memory, the memory storing program instructions, characterized in that: The processor executes program instructions to implement the highly realistic robot facial expression control method as described in any one of claims 1 to 6.

Images