A method, device, electronic device and storage medium for lip movement recognition

Abstract

The present application relates to a method, device, electronic device and storage medium for lip movement recognition, the method comprising: obtaining a two-dimensional face image and a pre-trained 3D face key point generation model; determining 3D lip key points according to the pre-trained 3D face key point generation model; determining a two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image; estimating 3D face key points according to the pre-trained 3D face key point generation model, and estimating face posture parameters according to the 3D face key points; constructing an adversarial generative network with the two-dimensional image of the lip region, the 3D lip key points and the face posture parameters as input, and obtaining a lip movement recognition model by minimizing the loss of the encoding features of the generated image as the objective function; and inputting the two-dimensional face image into the lip movement recognition model to obtain lip movement information. The present application solves the problem of posture and face direction restrictions during use by users through a solution of obtaining three-dimensional lip movement information through a two-dimensional face.

Description

Technical Field

[0001] The present application relates to the technical field of lip movement recognition, and in particular to a method, device, electronic device and storage medium for lip movement recognition. Background Art

[0002] Currently, lip movement recognition is based on two-dimensional lip shape image information. First, the face position and lip shape position area are obtained from the continuous video stream, and the lip shape information is sent to the lip movement recognition model to identify the corresponding mouth state (open / closed) to determine whether the user is speaking. The camera recording process has great restrictions on human posture and direction. In the current lip movement recognition system, the obtained lip visual feature information is positive, which means that you need to keep facing the device when interacting with it. This limits the usage scenarios of users and products. The two-dimensional lip movement image sequence loses part of the lip movement information, affecting the performance of lip movement recognition. Summary of the invention

[0003] Based on the above problems, the present application provides a method, device, electronic device and storage medium for lip movement recognition.

[0004] In a first aspect, an embodiment of the present application provides a method for lip movement recognition, comprising:

[0005] Obtain a 2D face image and a pre-trained 3D face key point generation model;

[0006] Determine 3D lip key points based on a pre-trained 3D facial key point generation model;

[0007] Determine a two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image;

[0008] Estimate 3D facial key points based on a pre-trained 3D facial key point generation model, and estimate facial posture parameters based on the 3D facial key points;

[0009] The 2D image of the lip area, 3D lip key points and facial posture parameters are used as input to construct a generative adversarial network. The objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model.

[0010] Input the two-dimensional face image into the lip movement recognition model to obtain the lip movement information.

[0011] Furthermore, in the above-mentioned lip movement recognition method, determining 3D lip key points according to a pre-trained 3D face key point generation model includes:

[0012] Estimate 3D facial key points based on a pre-trained 3D facial key point generation model;

[0013] Get 3D lip key points through 3D face key points.

[0014] Furthermore, in the above-mentioned lip movement recognition method, determining a two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image includes:

[0015] Obtain the lip image area of ​​the two-dimensional face image based on the 3D lip key points,

[0016] and expanding outwards in the lip image region to obtain a two-dimensional image of the lip region;

[0017] The outward expansion of the lip image region includes: using the nose tip key point as the upper limit of the expansion, and the key point of the facial contour as the lower limit of the expansion.

[0018] Furthermore, in the above-mentioned lip movement recognition method, estimating 3D facial key points according to a pre-trained 3D facial key point generation model, and estimating facial posture parameters according to the 3D facial key points include:

[0019] Get the key points of the standard 3D face model;

[0020] Calculate the rotation transformation matrix of the key points of the 3D face and the key points of the standard 3D face model;

[0021] Solve the rotation transformation matrix to estimate the facial pose parameters.

[0022] Furthermore, in the above-mentioned lip movement recognition method, minimizing the loss of encoding features of the generated image includes:

[0023] The distance loss between the facial pose parameters of the generated image and the target facial pose parameters is minimized; the difference between the generated image and the target output image is minimized; the distance between the 3D facial key points of the generated image and the target facial key points is minimized.

[0024] Furthermore, in the above-mentioned lip movement recognition method, a two-dimensional image of the lip area, 3D lip key points and facial posture parameters are used as input to construct a generative adversarial network. Before constructing the generative adversarial network, the following steps are included:

[0025] determining encoded features of the two-dimensional image of the lip region;

[0026] Calculate the optical flow parameters of 3D lip key points.

[0027] Furthermore, the above-mentioned method for lip movement recognition includes:

[0028] The adversarial generative network is constructed with the encoded features of the two-dimensional image of the lip area, optical flow parameters, and facial posture parameters as input.

[0029] In a second aspect, an embodiment of the present application further provides a device for lip movement recognition, comprising:

[0030] Acquisition module: used to acquire two-dimensional face images and pre-trained 3D face key point generation model;

[0031] The first determination module is used to determine the 3D lip key points according to the pre-trained 3D face key point generation model;

[0032] A second determination module: used to determine a two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image;

[0033] Estimation module: used to estimate 3D face key points based on the pre-trained 3D face key point generation model, and estimate face posture parameters based on the 3D face key points;

[0034] Construction module: used to construct a generative adversarial network with the 2D image of the lip area, 3D lip key points and facial posture parameters as input. The objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model;

[0035] Input module: used to input the two-dimensional face image into the lip movement recognition model to obtain lip movement information.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that it includes: a processor and a memory;

[0037] The processor is used to execute the above-mentioned lip movement recognition method by calling the program or instruction stored in the memory.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to perform the above-mentioned lip movement recognition method.

[0039] The advantages of the embodiments of the present application are: the present application relates to a method, device, electronic device and storage medium for lip movement recognition, the method comprising: obtaining a two-dimensional face image and a pre-trained 3D face key point generation model; determining 3D lip key points according to the pre-trained 3D face key point generation model; determining a two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image; estimating the 3D face key points according to the pre-trained 3D face key point generation model, and estimating face posture parameters according to the 3D face key points; constructing an adversarial generative network with the two-dimensional image of the lip area, the 3D lip key points and the face posture parameters as input, and the objective function is to minimize the coding feature loss of the generated image to obtain a lip movement recognition model; inputting the two-dimensional face image into the lip movement recognition model to obtain lip movement information. The solution of obtaining three-dimensional lip movement information through two-dimensional face solves the problem of user posture and face direction restrictions during use, and provides a more user-friendly interactive experience; three-dimensional lip movement information is estimated through the user's two-dimensional lip shape, which further improves the accuracy of lip movement recognition compared to the original two-dimensional lip shape lip movement recognition system, and reduces the requirements for image acquisition hardware equipment compared to the lip movement system using 3D camera equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A method for lip movement recognition provided in an embodiment of the present application is shown in FIG. Figure 1 ;

[0042] Figure 2 A method for lip movement recognition provided in an embodiment of the present application is shown in FIG. Figure 2 ;

[0043] Figure 3 A schematic diagram of a lip movement recognition device provided in an embodiment of the present application;

[0044] Figure 4 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0047] Figure 1 A method for lip movement recognition provided in an embodiment of the present application is shown in FIG. Figure 1 .

[0048] In a first aspect, the present application provides a method for lip movement recognition, combining Figure 1 , including six steps from S101 to S106:

[0049] S101: Acquire a two-dimensional face image and a pre-trained 3D face key point generation model.

[0050] Specifically, in an embodiment of the present application, a two-dimensional face image can be obtained by obtaining an image frame sequence through the video stream of the interactive device camera, executing a face detection module, and obtaining a face position area image, thereby obtaining a two-dimensional face image. The pre-trained 3D face key point generation model obtained is a 3D face key point generation model trained by marking the face key points based on existing face data.

[0051] S102: Determine 3D lip key points according to a pre-trained 3D face key point generation model.

[0052] Specifically, in an embodiment of the present application, the 3D lip key points can be determined by a pre-trained 3D face key point generation model.

[0053] S103: Determine a two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image.

[0054] Specifically, in the embodiment of the present application, a two-dimensional image of the lip area is determined based on the 3D lip key points and the two-dimensional face image.

[0055] S104: Estimate 3D facial key points according to a pre-trained 3D facial key point generation model, and estimate facial posture parameters according to the 3D facial key points.

[0056] Specifically, in the embodiment of the present application, the step of estimating facial posture parameters based on 3D facial key points will be described in detail below.

[0057] S105: A two-dimensional image of the lip area, 3D lip key points and facial posture parameters are used as input to construct a generative adversarial network, and the objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model.

[0058] Specifically, in an embodiment of the present application, a two-dimensional image of the lip area, 3D lip key points and facial posture parameters are used as inputs to construct a generative adversarial network to accelerate the convergence of the generative adversarial network. The objective function is to minimize the coding feature loss of the generated image, and the generative adversarial network obtained has high accuracy.

[0059] S106: Input the two-dimensional face image into the lip movement recognition model to obtain lip movement information.

[0060] Specifically, in the embodiments of the present application, three-dimensional lip movement information can be directly obtained through a two-dimensional face image, which solves the problem of posture and face direction restrictions during use, and provides a more user-friendly interactive experience. The three-dimensional lip movement information is estimated through the user's two-dimensional lip shape. Compared with the original two-dimensional lip shape lip movement recognition system, the accuracy of lip movement recognition is further improved, and compared with the lip movement system using 3D camera equipment, the requirements for image acquisition hardware equipment are reduced.

[0061] Furthermore, in the above-mentioned lip movement recognition method, determining 3D lip key points according to a pre-trained 3D face key point generation model includes:

[0062] The 3D facial key points are estimated according to the pre-trained 3D facial key point generation model, and the 3D lip key points are obtained through the 3D facial key points.

[0063] Specifically, in an embodiment of the present application, determining the 3D lip key points according to the pre-trained 3D face key point generation model is firstly to estimate the 3D face key points through the pre-trained 3D face key point generation model, and then to obtain the 3D lip key points through the 3D face key points.

[0064] Furthermore, in the above-mentioned lip movement recognition method, determining a two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image includes:

[0065] Acquire a lip image region of a two-dimensional face image based on the 3D lip key points, and expand the lip image region outward to obtain a two-dimensional image of the lip region;

[0066] The outward expansion of the lip image region includes: using the nose tip key point as the upper limit of the expansion, and the key point of the facial contour as the lower limit of the expansion.

[0067] Specifically, in an embodiment of the present application, 3D lip key points are obtained through 3D face key points, the lip area on the two-dimensional face is obtained based on the 3D lip key points and the two-dimensional face image, and the lip area is expanded outward, using the nose tip key points as the upper limit of the expansion, and the key points of the facial contour are used as the lower limit of the expansion.

[0068] Figure 2 A method for lip movement recognition provided in an embodiment of the present application is shown in FIG. Figure 2 .

[0069] Furthermore, in the above-mentioned method for lip movement recognition, 3D facial key points are estimated according to a pre-trained 3D facial key point generation model, and facial posture parameters are estimated according to the 3D facial key points, including three steps S201 to S203:

[0070] S201: Acquire key points of a standard 3D face model;

[0071] S202: Calculate the rotation transformation matrix of the key points of the 3D face and the key points of the standard 3D face model;

[0072] S203: Estimate facial posture parameters using the rotation transformation matrix.

[0073] Specifically, in the embodiment of the present application, it is assumed that the standard 3D face key point sequence is A = [A_1, A_2, A_3, ..., A_N], N is the number of annotated face key points, the 3D face key point sequence is B = [B_1, B_2, B_3, ..., B_N], N is the current number of face key points, and the rotation transformation matrix R = [R_x, R_y, R_z]; then the calculation relationship between the rotation transformation matrix of the 3D face key points and the key points of the standard 3D face model is B = A*R. Since the number of 3D face key points is greater than the face posture parameters, calculating the face posture parameters through the key points is converted into solving an overdetermined set of equations to calculate the face posture parameters in R.

[0074] Furthermore, in the above-mentioned lip movement recognition method, minimizing the loss of encoding features of the generated image includes:

[0075] The distance loss between the facial pose parameters of the generated image and the target facial pose parameters is minimized; the difference between the generated image and the target output image is minimized; the distance between the 3D facial key points of the generated image and the target facial key points is minimized.

[0076] Specifically, in the embodiment of the present application, the loss function is defined to minimize the loss of the encoding features of the generated image including the following three parts:

[0077] Loss_1: The difference between the generated image output_generator_1 and the target output image output_groundtruth is minimized; Loss_2: The distance between the predicted 3D face key points and the ground_truth 3D face key points is minimized; Loss_3: The distance loss between the predicted face posture parameters and the ground_truth face posture parameters is minimized.

[0078] Furthermore, in the above-mentioned lip movement recognition method, a two-dimensional image of the lip area, 3D lip key points and facial posture parameters are used as input to construct a generative adversarial network. Before constructing the generative adversarial network, the following steps are included:

[0079] determining encoded features of the two-dimensional image of the lip region;

[0080] Calculate the optical flow parameters of 3D lip key points.

[0081] Furthermore, the above-mentioned method for lip movement recognition includes:

[0082] The adversarial generative network is constructed with the encoded features of the two-dimensional image of the lip area, optical flow parameters, and facial posture parameters as input.

[0083] Specifically, in the embodiment of the present application, the input of the generative adversarial network is the original two-dimensional lip area image input_1, the 3D lip key point input_2, and the facial posture parameter input_3, wherein, after input_1 passes through the neural network encoder, the encoded feature is encode_input_1, and input_2 calculates the optical flow parameter flow_input_2 based on the inter-frame transformation. The above features encode_input_1, flow_input_2 and input_3 are used as input to construct the generative adversarial network.

[0084] Figure 3 A schematic diagram of a lip movement recognition device provided in an embodiment of the present application.

[0085] In a second aspect, the present application also provides a lip movement recognition device, combined with Figure 3 ,include:

[0086] Acquisition module 301: used to acquire a two-dimensional face image and a pre-trained 3D face key point generation model.

[0087] Specifically, in the embodiment of the present application, the acquisition module 301 can obtain a two-dimensional face image by obtaining an image frame sequence through the video stream of the interactive device camera, execute the face detection module, obtain the face position area image, and thus obtain a two-dimensional face image. The pre-trained 3D face key point generation model obtained by the acquisition module 301 is a 3D face key point generation model trained by marking the face key points based on existing face data.

[0088] The first determination module 302 is used to determine 3D lip key points according to a pre-trained 3D face key point generation model.

[0089] Specifically, in the embodiment of the present application, the first determination module 302 can determine the 3D lip key points by using a pre-trained 3D face key point generation model.

[0090] The second determination module 303 is used to determine the two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image.

[0091] Specifically, in the embodiment of the present application, the second determination module 303 determines the two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image.

[0092] Estimation module 304: used to estimate 3D facial key points based on a pre-trained 3D facial key point generation model, and estimate facial posture parameters based on the 3D facial key points.

[0093] Specifically, in the embodiment of the present application, the step of estimating the facial posture parameters according to the 3D facial key points by the estimation module 304 has been described in detail above.

[0094] Construction module 305: used to construct a generative adversarial network using the two-dimensional image of the lip area, 3D lip key points and facial posture parameters as input, and the objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model.

[0095] Specifically, in the embodiment of the present application, the construction module 305 takes the two-dimensional image of the lip area, the 3D lip key points and the facial posture parameters as inputs to construct an adversarial generative network to accelerate the convergence of the adversarial generative network. The objective function is to minimize the coding feature loss of the generated image, and the accuracy of the adversarial generative network obtained is high.

[0096] Input module 306: used to input the two-dimensional face image into the lip movement recognition model to obtain lip movement information.

[0097] Specifically, in the embodiment of the present application, the input module 306 can directly obtain three-dimensional lip movement information by inputting a two-dimensional face image into a lip movement recognition model, thereby solving the problem of posture and face direction restrictions during use of the user, and providing a more user-friendly interactive experience. The three-dimensional lip movement information is estimated through the user's two-dimensional lip shape, which further improves the accuracy of lip movement recognition compared to the original two-dimensional lip shape lip movement recognition system, and reduces the requirements for image acquisition hardware equipment compared to the lip movement system using 3D camera equipment.

[0098] In a third aspect, an embodiment of the present application further provides an electronic device, characterized in that it includes: a processor and a memory;

[0099] The processor is used to execute the above-mentioned lip movement recognition method by calling the program or instruction stored in the memory.

[0100] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a program or instruction, and the program or instruction enables the computer to perform the above-mentioned lip movement recognition method.

[0101] Figure 4 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure.

[0102] like Figure 4 As shown, the electronic device includes: at least one processor 401, at least one memory 402 and at least one communication interface 403. The various components in the electronic device are coupled together through a bus system 404. The communication interface 403 is used to transmit information between external devices. It can be understood that the bus system 404 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 404 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 4 Various buses are labeled as bus system 404 .

[0103] It can be understood that the memory 402 in this embodiment can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories.

[0104] In some implementations, the memory 402 stores the following elements, executable units or data structures, or a subset thereof, or an extended set thereof: an operating system and application programs.

[0105] The operating system includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application includes various application programs, such as a media player (Media Player), a browser (Browser), etc., which are used to implement various application services. A program that implements any of the methods of lip movement recognition provided in the embodiments of the present application can be included in the application.

[0106] In the embodiment of the present application, the processor 401 calls the program or instructions stored in the memory 402, specifically, the program or instructions stored in the application, and the processor 401 is used to execute the steps of each embodiment of the method for lip movement recognition provided in the embodiment of the present application.

[0107] Obtain a 2D face image and a pre-trained 3D face key point generation model;

[0108] Determine 3D lip key points based on a pre-trained 3D facial key point generation model;

[0109] Determine a two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image;

[0110] Estimate 3D facial key points based on a pre-trained 3D facial key point generation model, and estimate facial posture parameters based on the 3D facial key points;

[0111] The 2D image of the lip area, 3D lip key points and facial posture parameters are used as input to construct a generative adversarial network. The objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model.

[0112] Input the two-dimensional face image into the lip movement recognition model to obtain the lip movement information.

[0113] Any of the methods of lip movement recognition provided in the embodiments of the present application can be applied to the processor 401, or implemented by the processor 401. The processor 401 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 401 or instructions in the form of software. The above processor 401 can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0114] The steps of any method in the method for lip movement recognition provided in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software units in the decoding processor. The software unit can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 402, and the processor 401 reads the information in the memory 402 and completes the steps of a method for lip movement recognition in combination with its hardware.

[0115] Those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0116] Those skilled in the art will appreciate that the description of each embodiment has its own emphasis, and for parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for lip movement recognition, characterized in that: include: Get a 2D face image and a pre-trained 3D face key point generation model; Determining 3D lip key points according to the pre-trained 3D facial key point generation model; Determine a two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image; estimating 3D facial key points according to the pre-trained 3D facial key point generation model, and estimating facial posture parameters according to the 3D facial key points; determining encoded features of the two-dimensional image of the lip region; Calculating optical flow parameters of the 3D lip key points; A generative adversarial network is constructed using the two-dimensional image of the lip area, the 3D lip key points and the facial posture parameters as input, and the objective function is to minimize the encoding feature loss of the generated image to obtain a lip movement recognition model; Constructing a generative adversarial network using the encoded features of the two-dimensional image of the lip region, the optical flow parameters, and the facial posture parameters as input; The two-dimensional face image is input into the lip movement recognition model to obtain lip movement information.

2. The method for lip movement recognition according to claim 1, characterized in that: The determining of 3D lip key points according to the pre-trained 3D face key point generation model includes: estimating 3D facial key points according to the pre-trained 3D facial key point generation model; The 3D lip key points are obtained through the 3D face key points.

3. The method for lip movement recognition according to claim 1, characterized in that: The determining of the two-dimensional image of the lip region according to the 3D lip key points and the two-dimensional face image includes: Acquire the lip image region of the two-dimensional face image based on the 3D lip key points, and expanding outwards in the lip image region to obtain a two-dimensional image of the lip region; The outward expansion of the lip image region includes: using the nose tip key point as the upper limit of the expansion, and the key point of the facial contour as the lower limit of the expansion.

4. The method for lip movement recognition according to claim 1, characterized in that: The method of estimating 3D facial key points according to the pre-trained 3D facial key point generation model and estimating facial posture parameters according to the 3D facial key points includes: Get the key points of the standard 3D face model; Calculating the rotation transformation matrix of the 3D face key points and the key points of the standard 3D face model; Solve the rotation transformation matrix to estimate facial posture parameters.

5. The method for lip movement recognition according to claim 1, characterized in that: The encoding feature loss of the generated image is minimized, including: The distance loss between the facial pose parameters of the generated image and the target facial pose parameters is minimized; the difference between the generated image and the target output image is minimized; the distance between the 3D facial key points of the generated image and the target facial key points is minimized.

6. A lip movement recognition device, characterized in that: include: Acquisition module: used to acquire two-dimensional face images and pre-trained 3D face key point generation model; A first determination module: used to determine 3D lip key points according to the pre-trained 3D face key point generation model; A second determination module: used to determine a two-dimensional image of the lip area according to the 3D lip key points and the two-dimensional face image; Estimation module: used to estimate 3D face key points according to the pre-trained 3D face key point generation model, and estimate face posture parameters according to the 3D face key points; determining encoded features of the two-dimensional image of the lip region; Calculating optical flow parameters of the 3D lip key points; Construction module: used to construct a generative adversarial network using the two-dimensional image of the lip area, the 3D lip key points and the facial posture parameters as input, and the objective function is to minimize the coding feature loss of the generated image to obtain a lip movement recognition model; Constructing a generative adversarial network using the encoded features of the two-dimensional image of the lip region, the optical flow parameters, and the facial posture parameters as input; Input module: used to input the two-dimensional face image into the lip movement recognition model to obtain lip movement information.

7. An electronic device, characterized in that: include: Processor and memory; The processor is used to execute the lip movement recognition method as described in any one of claims 1 to 5 by calling the program or instruction stored in the memory.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and the program or instruction enables a computer to execute the lip movement recognition method as claimed in any one of claims 1 to 5.

Images