Expression editing method and device and storage medium

By obtaining the images and voice text to be edited, key points are extracted and fusion processing are performed, and the conditions are used to generate the expression images of characters constrained by voice content are used to solve the problems of facial expression editing in the prior art. Improved image quality and user experience.

CN118968576AInactive Publication Date: 2024-11-15HISENSE ELECTRONIC TECH (WUHAN) CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202310533785.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-09
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing facial expression editing methods can easily lead to deformity and artifacts on the facial features of the characters in the image when changing the facial expressions of the characters in the image, affecting the quality of the generated image.

Method used

By obtaining the image to be edited and the voice text, key points are extracted and the face key points are fused with the image, and the conditions are used to generate the expression images of characters constrained by the voice content.

Benefits of technology

It avoids facial features deformation and artifact problems caused by static information, improves the quality of generated characters' expression images, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968576A_ABST
    Figure CN118968576A_ABST
Patent Text Reader

Abstract

The invention provides an expression editing method and device and a storage medium, and relates to the technical field of face attribute editing. The expression editing method comprises the steps of obtaining a to-be-edited image and a voice text; wherein the voice text is a text obtained by performing text conversion on the voice content of the constraint to-be-edited image; the to-be-edited image is a figure image; performing key point extraction processing on the to-be-edited image according to the voice text to obtain face key points corresponding to the to-be-edited image; wherein the number of the extracted face key points is greater than a preset threshold value; performing fusion processing on the face key points and the to-be-edited image to obtain a character expression image; the character expression in the character expression image is constrained by the voice content. The method is used for solving the problems of deformation and artifacts of five sense organs of an image face in a current facial expression editing method.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of face attribute editing, and in particular to an expression editing method, device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence, face attribute editing applications have frequently appeared in front of the public and have aroused widespread discussion and interest. Face attribute editing is a technology that changes specific attributes of the face in an image. For example, through face attribute editing, the face in the image can be made older or younger, a male face can be changed to a female face, or a celebrity face. As a branch of face attribute editing, facial expression editing can change the facial expression of a person without changing the environmental features and appearance of the person in the image, thereby showing the emotional state of the person in the image. It is used in many technical fields such as human-computer interaction, game development, and emotional data set supplementation.

[0003] The current facial expression editing method usually involves learning a generative model for manually annotated and cropped facial images, facial expression distribution, and facial identity information, then inputting the facial image to be edited and the expression control vector into the generator, and outputting a facial image controlled by the expression control vector. The facial images obtained in this way usually have problems with deformation of the facial features and artifacts. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an expression editing method, device and storage medium.

[0005] In a first aspect, the present application provides an expression editing device, comprising: a processor, configured to obtain an image to be edited and a voice text; wherein the voice text is a text obtained by converting the voice content that constrains the image to be edited into text; the image to be edited is a person image; the processor is also configured to perform key point extraction processing on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited; wherein the number of extracted facial key points is greater than a preset threshold; the processor is also configured to fuse the facial key points and the image to be edited to obtain a person expression image; the person expression in the person expression image is constrained by the voice content.

[0006] In the second aspect, the present application provides an expression editing method, including: obtaining an image to be edited and a voice text; wherein the voice text is a text obtained by converting the voice content that constrains the image to be edited into text; the image to be edited is a person image; performing key point extraction processing on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited; wherein the number of extracted facial key points is greater than a preset threshold; fusing the facial key points and the image to be edited to obtain a person expression image; the person expression in the person expression image is constrained by the voice content.

[0007] In a third aspect, the present application provides an electronic device, comprising: a memory and a processor, the memory being used to store a computer program; the processor being used to enable the electronic device to implement any expression editing method provided in the second aspect when executing the computer program.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium, comprising: a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a computing device, the computing device executes any expression editing method provided in the second aspect.

[0009] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute any expression editing method provided in the second aspect.

[0010] It should be noted that the above computer instructions may be stored in whole or in part on a first computer-readable storage medium, wherein the first computer-readable storage medium may be packaged together with the processor of the server, or may be packaged separately from the processor of the server, which is not limited in this application.

[0011] The description of the second, third, fourth and fifth aspects of the present application can refer to the detailed description of the first aspect; and the beneficial effects of the description of the second, third, fourth and fifth aspects can refer to the beneficial effect analysis of the first aspect, which will not be repeated here.

[0012] In this application, the name of the above server does not limit the device or functional module itself. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they fall within the scope of the claims of this application and their equivalent technologies.

[0013] These and other aspects of the present application will become more apparent from the following description.

[0014] Compared with the prior art, the technical solution provided by this application has the following advantages:

[0015] In the above scheme, the expression editing device first obtains the image to be edited and the voice text information, and performs key point extraction processing on the image to be edited according to the voice text information to obtain the facial key points of the image to be edited. After that, the facial key points and the image to be edited are fused to obtain the character expression image constrained by the voice content. In this way, the expression state of the user when speaking is added to the facial expression editing process, avoiding the problem of deformation and artifacts of facial features in the image when facial expression editing is performed based on static information such as facial images and identity information alone, improving the quality of generated character expression images, and thus improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 A schematic diagram of a scenario of an expression editing method provided in an embodiment of the present application;

[0019] Figure 2 One of the structural schematic diagrams of the expression editing device provided in the embodiment of the present application;

[0020] Figure 3 The second structural diagram of the expression editing device provided in the embodiment of the present application;

[0021] Figure 4 One of the flowcharts of the expression editing method provided in the embodiment of the present application;

[0022] Figure 5 A schematic diagram of facial key point distribution provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of a method flow chart for training a first generation model provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of a method flow chart for training a second generation model provided in an embodiment of the present application;

[0025] Figure 8 The second flowchart of the expression editing method provided in the embodiment of the present application;

[0026] Fig. 9The third flowchart of the expression editing method provided in the embodiment of the present application;

[0027] Fig.10 A fourth flowchart of the expression editing method provided in an embodiment of the present application;

[0028] Fig.11 A fifth flowchart of the expression editing method provided in the embodiment of the present application;

[0029] Fig.12 Flow chart of the expression editing method provided in the embodiment of the present application is as follows:

[0030] Fig.13 The third structural diagram of the expression editing device provided in the embodiment of the present application;

[0031] Fig.14 A schematic diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0033] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0034] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0035] Figure 1 Schematic diagram of the scene architecture of the expression editing method provided in the embodiment of the present application. Figure 1 As shown, the scenario architecture provided by the embodiment of the present application includes: a server 100 and an expression editing device 200.

[0036] The expression editing device 200 provided in the embodiment of the present application can have various implementation forms, for example, it can be a mobile phone, a personal computer (PC), a smart TV, a laser projection device, a TV, a monitor, a wearable device, a vehicle-mounted device, an electronic table, etc.

[0037] In some embodiments, when receiving the expression editing instruction, the expression editing device 200 may communicate data with the server 100. The expression editing device 200 may be allowed to communicate with the server 100 via a local area network (LAN) or a wireless local area network (WLAN).

[0038] The server 100 may be a server that provides various services, such as a server that supports image data and voice data acquired by the expression editing device 200. The server may analyze and process the received image and voice data, and feed back the processing results (such as a character expression image) to the expression editing device 200. The server 100 may be a server cluster or multiple server clusters, and may include one or more types of servers.

[0039] The expression editing device 200 can be hardware or software. When the expression editing device 200 is hardware, it can be various electronic devices with sound collection functions, including but not limited to smart phones, televisions, tablet computers, smart watches, computers, AI devices, robots, smart vehicles, etc. When the expression editing device 200 is software, it can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, to provide voice collection services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0040] It should be noted that the expression editing method provided in the embodiment of the present application can be executed by the server 100, can be executed by the expression editing device 200, or can be executed jointly by the server 100 and the expression editing device 200, and the present application does not limit this.

[0041] For example, when the expression editing method provided in the embodiment of the present application is executed by the expression editing device 200, the process is as follows: after receiving the image to be edited and the voice content, the expression editing device 200 first converts the voice content into voice text. Then, the key point extraction process is performed on the image to be edited according to the voice text to obtain the facial key points corresponding to the image to be edited, and then the facial key points and the image to be edited are fused to obtain the character expression image, and the character expression image is displayed. In this way, the expression state of the user when speaking is added to the facial expression editing process, avoiding the problem of deformation and artifacts of the facial features of the image when the facial expression is edited based on static information such as facial images and identity information alone, improving the quality of the generated character expression image, and thus improving the user experience.

[0042] Figure 2 FIG. 2 shows a hardware configuration block diagram of the expression editing device 200 according to an exemplary embodiment. Figure 2 The expression editing device 200 shown includes at least one of a tuner 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, a RAM, a ROM, and a first interface to an nth interface for input / output. The display 260 can be a display with a touch function, such as a touch display. The tuner 210 receives broadcast television signals by wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as and EPG data signals. The detector 230 is used to collect signals of the external environment or interaction with the outside. The controller 250 and the tuner 210 can be located in different split devices, that is, the tuner 210 can also be in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0043] In some embodiments, the controller 250 controls the operation of the expression editing device 200 and responds to the user's operation through various software control programs stored in the memory. The controller 250 controls the overall operation of the expression editing device 200.

[0044] In some examples, the expression editing device 200 of one or more embodiments is a mobile phone 1, and the operating system of the mobile phone 1 is an Android system. Figure 3As shown, the mobile phone 1 can be logically divided into an application layer (Applications) (referred to as "application layer") 21, an application framework layer (Application Framework) (referred to as "framework layer") 22, an Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer") 23, and a kernel layer 24.

[0045] Among them, the application layer 21 includes one or more applications. The application can be a system application or a third-party application. For example, the application layer 21 includes a first application, which can provide an expression editing function. The framework layer 22 provides an application programming interface (API) and a programming framework for the application of the application layer 21. The system runtime layer 23 provides support for the upper layer, namely the framework layer 22. When the framework layer 22 is used, the Android operating system will run the C / C++ library contained in the system runtime layer 23 to implement the functions to be implemented by the framework layer 22. The kernel layer 24 serves as a software middleware between the hardware layer and the application layer 21, and is used to manage and control hardware and software resources.

[0046] In some examples, after the mobile phone 1 is turned on, if the user needs to edit the expression of a certain image during the process of using the expression editing device 200, the user can select the control corresponding to the first application provided by the expression editing device 200, and select the image to be edited and the corresponding voice content. The processing unit 201 of the expression editing device 200 responds to the selection operation of the target control, starts the expression editing configuration, and creates a voice conversion module, an extraction processing module and a fusion processing module in the local framework (native layer) of the application layer 21. Among them, the voice conversion module is used to convert the voice content selected by the user into text to obtain voice text; the extraction processing module is used to extract key points of the image to be edited selected by the user according to the voice text to obtain the facial key points corresponding to the image to be edited; the fusion processing module is used to fuse the facial key points and the image to be edited to obtain the human expression image constrained by the voice content. In this way, the processing unit 201 of the expression editing device 200 adds the facial expression state of the user when speaking to the facial expression editing process, avoiding the problems of deformation of facial features and artifacts in the image when facial expressions are edited based solely on static information such as facial images and identity information, improving the quality of generated character expression images, and thereby enhancing the user experience.

[0047] Specifically, the storage unit 203 of the mobile phone 1 is used to store the application program of the first application, and data such as the operating system of the mobile phone 1 .

[0048] In the following embodiments, the method of the embodiment of the present application is described by taking the mobile phone 1 as an example in which the execution subject of the expression editing method provided by the embodiment of the present disclosure is executed.

[0049] The present application embodiment provides an expression editing method, such as Figure 4 As shown, the expression editing method may include S11-S13.

[0050] S11. Obtain the image and speech text to be edited.

[0051] The speech text is a text obtained by converting the speech content constraining the image to be edited into text; and the image to be edited is a person image.

[0052] In some embodiments, when receiving an expression editing instruction, the expression editing device first obtains the image to be edited and its corresponding voice content input by the user, and then converts the voice content into text to obtain voice text.

[0053] In some other embodiments, after the expression editing device receives the image to be edited and the voice text, it determines that the user needs to perform expression editing on the image to be edited, that is, the expression editing device directly receives the image to be edited and the voice text, without the need for a voice-to-text conversion process.

[0054] Of course, the expression editing device may also receive image data and voice data in other forms, which can convert the image data into the image to be edited and convert the voice data into voice text in the expression editing device. This application does not limit this.

[0055] S12, extracting key points of the image to be edited according to the speech text to obtain key points of the face corresponding to the image to be edited.

[0056] Facial key points usually refer to the key points of the face, such as eyebrows, eyes, nose, mouth, and points in the facial contour area, which can be used to describe the shape of the facial features. Usually, the number of points for extracting facial key points is 5 points, 68 points, 96 points, 98 points, 106 points, 186 points, etc. Figure 5 The figure shows a 68-point facial key point, including 17 points 1-17 at the facial contour, 10 points 18-27 at the eyebrows, 9 points 28-36 at the nose, 12 points 37-48 at the eyes, and 20 points 49-68 at the mouth. These 68 points together constitute the facial key points.

[0057] In some embodiments, the number of facial key points extracted by the present application is greater than a preset threshold value, wherein the value of the preset threshold value may be a default value or a manually marked value, for example, the preset threshold value is 68 points.

[0058] In some embodiments, key point extraction processing is performed on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited. The image to be edited and the voice text are input into a first generation model to obtain facial key points in the image to be edited.

[0059] First, before using the first generation model for the first time, it is necessary to train and generate the first generation model according to a preset video set.

[0060] In some embodiments, an initial frame of a preset video and a speech text corresponding to the speech content of the preset video are taken as input, and a cutoff frame of the preset video is taken as output, and a conditional generative adversarial network (cGAN) is used to train a first generative model.

[0061] Specifically, because traditional generative adversarial networks (GANs) are mainly composed of a generator and a discriminator, they can input a set of noises of fixed dimensions to generate images close to the real world. Therefore, the process of generating images by traditional generative adversarial networks is random and uncertain, making it difficult to generate images with specified attributes.

[0062] In response to this shortcoming, Mehdi Mirza et al. proposed a conditional generative adversarial network. Like the generative adversarial network, the conditional generative adversarial network also contains a generator and a discriminator. The difference is that in addition to the input noise, the generator also inputs conditional information to constrain the category or attribute of the generated image. The input of the discriminator is the generated image (such as the face key points generated by the generator), the true value image (such as the annotated face key points) and the conditional information (such as speech text), which is responsible for distinguishing whether the image is real. Therefore, the core idea of ​​the conditional generative adversarial network is to integrate the attribute information into the generator and the discriminator together with the input. The attribute information can be any label, such as object category, color, facial expression, etc., to constrain the image generation process.

[0063] Therefore, in order to make the facial key points extracted by the first generation model more accurate, the present application adopts a conditional generative adversarial network and uses the speech content as a condition for training to generate the first generation model, so that the first generation model can generate facial key points constrained by the speech content.

[0064] More specifically, Figure 6 As shown, the specific process of generating the first generation model according to the preset video set training includes the following steps:

[0065] S001. Obtain a preset video set, and extract a first training sample from the preset video set.

[0066] Specifically, the initial frame, the end frame and the speech text corresponding to the speech content in each video in the video set are extracted as the first training sample. A first training sample includes the initial frame, the end frame and the speech text in the same video.

[0067] S002. Initialize network parameters of the first generation model.

[0068] S003. Use the initial frame in the first training sample as the input of the first generation model, the speech text as the conditional input, and the facial key points corresponding to the cutoff frame as the target output, and train the first generation model n times until the loss function of the first generation model converges.

[0069] Specifically, a first training sample is extracted from the first training sample, and the initial frame in the first training sample is used as the input of the first generation model, the speech text is used as the conditional input, the first generation sample is generated using the noise distribution defined in the first generation model, and the loss function of the first generation model is calculated based on the first generation sample and the cutoff frame in the first training sample and the corresponding facial key points. When it is determined that the loss function of the first generation model has not converged, the network parameters of the first generation model are corrected according to the loss function of the first generation model. After correcting the network parameters of the first generation model, the first training sample is continuously extracted to train the first generation model until the loss function of the first generation model converges, and finally the first generation model is obtained.

[0070] Among them, the loss function of the first generation model is: L G1 =L1+αL2. Among them, L G1 Used to represent the loss function of the first generation model, L1 is used to represent the pixel loss, and L2 is used to represent the generative adversarial loss between the first identification model and the first generation model; the pixel loss is the pixel loss between the face key points generated by the first generation model and the face key points corresponding to the cutoff frame.

[0071] In some embodiments, the expression editing method further includes training and generating a first identification model based on the facial key points generated by the first generation model and the facial key points corresponding to the cutoff frame, specifically including the following steps:

[0072] S004. Use the facial key points generated by the first generation model and the facial key points corresponding to the cutoff frame as inputs of the first identification model, and perform adversarial loss training m times until the loss function of the first identification model converges.

[0073] Specifically, the face key points generated by the first generation model (at this time, the first generation model has been trained to converge the loss function) and the face key points corresponding to the cutoff frame are obtained, and the loss function of the first identification model is calculated based on the face key points generated by the first generation model and the face key points corresponding to the cutoff frame. More specifically, in each round of iteration of training the first identification model, the network parameters of the first generation model are first fixed, the face key points generated by the first generation model and the face key points corresponding to the cutoff frame are input into the first identification model, and the first identification model is required to output true when the face key points corresponding to the cutoff frame are used as input, and output false when the face key points generated by the first generation model are used as input, and the identification loss of the first identification model is calculated. Then, the network parameters of the first identification model are fixed again, and the face key points generated by the first generation model and the face key points corresponding to the cutoff frame are input into the first identification model again, and the first identification model is required to output the opposite corresponding result of the previous step (that is, the first identification model outputs false when the face key points corresponding to the cutoff frame are used as input, and outputs true when the face key points generated by the first generation model are used as input), and the generation loss of the first generation model is calculated. Then the loss function of the first identification model is the weighted sum of the identification loss of the first identification model and the generation loss of the first generation model. After that, the facial key points generated by the first generation model and the facial key points corresponding to the cutoff frame are used as the input of the first identification model, and the first identification model is trained with adversarial loss m times, and the network parameters of the first generation model and the first identification model are corrected until the loss function of the first identification model converges to obtain the first identification model.

[0074] In this way, after the first generation model is trained, the image to be edited and the speech text are input into the first generation model, and the key points of the face in the image to be edited can be obtained.

[0075] In some embodiments, the key points of the image to be edited are extracted according to the speech text to obtain the key points of the face corresponding to the image to be edited, or a face key point extraction algorithm is used to extract the key points of the image to be edited according to the speech text to obtain the key points of the face corresponding to the image to be edited, for example, the Dlib algorithm in OpenCV, the deep alignment network (DAN) algorithm, etc.

[0076] S13, fusing the key points of the face with the image to be edited to obtain a character expression image.

[0077] Among them, the facial expressions of the characters in the facial expression images are constrained by the speech content.

[0078] In some embodiments, the facial key points and the image to be edited are fused to obtain the character expression image, which can be achieved by inputting the facial key points and the image to be edited into a second generation model to obtain the character expression image.

[0079] First, before using the second generation model for the first time, it is necessary to train and generate the second generation model based on a preset video set.

[0080] In some embodiments, the initial frame of the preset video and the first output result are used as input, and the cutoff frame of the preset video is used as output, and the conditional generative adversarial network is used to train the second generative model. The first output result is the output result of the preset voice text and the initial frame of the preset video in the first generative model.

[0081] Specifically, Figure 7 As shown, the specific process of generating the second generation model according to the preset video set training includes the following steps:

[0082] S011. Obtain a preset video set, and extract a second training sample from the preset video set.

[0083] Specifically, the initial frame and the cutoff frame of each video in the video set are extracted as the second training sample, wherein a second training sample includes the initial frame and the cutoff frame in the same video.

[0084] S012. Initialize network parameters of the second generation model.

[0085] S013. Use the initial frame in the second training sample as the input of the second generation model, the first output result as the conditional input, and the cutoff frame in the second training sample as the target output, and train the second generation model k times until the loss function of the second generation model converges.

[0086] Specifically, a second training sample is extracted from the second training sample, and the initial frame in the second training sample is used as the input of the second generation model, the first output result (i.e., the generation result of the initial frame in the second training sample in the first generator) is used as the conditional input, the second generation sample is generated using the noise distribution defined in the second generation model, and the loss function of the second generation model is calculated based on the second generation sample and the cutoff frame in the second training sample. When it is determined that the loss function of the second generation model has not converged, the network parameters of the second generation model are corrected according to the loss function of the second generation model. After correcting the network parameters of the second generation model, the second training sample and the first output result are continued to be extracted to train the second generation model until the loss function of the second generation model converges, and finally the second generation model is obtained.

[0087] Among them, the loss function of the second generation model is: L G2=L3+αL4+βL5. Among them, L G2 Used to represent the loss function of the second generation model, L3 is used to represent the pixel loss between the character expression image generated by the second generation model and the cutoff frame, L4 is used to represent the generative adversarial loss between the second identification model and the second generation model, and L5 is used to represent the cross entropy loss of the expression identification model to the second generation model.

[0088] In some embodiments, the expression editing method further includes training and generating a second identification model based on the character expression image generated by the second generation model and the face image corresponding to the cutoff frame, specifically including the following steps:

[0089] S014. Use the facial expression image generated by the second generation model and the face image corresponding to the cutoff frame as input of the second identification model, and perform adversarial loss training p times until the loss function of the second identification model converges.

[0090] Specifically, the facial expression image of the person generated by the second generation model (at this time, the second generation model has been trained to converge the loss function) and the facial image corresponding to the cutoff frame are obtained, and the loss function of the second identification model is calculated based on the facial expression image of the person generated by the second generation model and the facial image corresponding to the cutoff frame. Similarly, in each round of iteration of training the second identification model, the network parameters of the second generation model are first fixed, the facial expression image of the person generated by the second generation model and the facial image corresponding to the cutoff frame are input into the second identification model, and the second identification model is required to output true when the facial image corresponding to the cutoff frame is used as input, and output false when the facial expression image of the person generated by the second generation model is used as input, and the identification loss of the second identification model is calculated. Subsequently, the network parameters of the second identification model are fixed again, and the facial expression image of the person generated by the second generation model and the facial image point corresponding to the cutoff frame are input into the second identification model again, and the second identification model is required to output the corresponding result opposite to the previous step (that is, the second identification model outputs false when the facial image corresponding to the cutoff frame is used as input, and outputs true when the facial expression image of the person generated by the second generation model is used as input), and the generation loss of the second generation model is calculated. Then the loss function of the second identification model is the weighted sum of the identification loss of the second identification model and the generation loss of the second generation model. After that, the facial expression image generated by the second generation model and the face image corresponding to the cutoff frame are used as the input of the second identification model, and the second identification model is trained with adversarial loss p times, and the network parameters of the second generation model and the second identification model are corrected until the loss function of the second identification model converges to obtain the second identification model.

[0091] In some embodiments, the expression editing method further includes training and generating an expression identification model based on the face image corresponding to the cutoff frame and the expression label corresponding to the face image, specifically including the following steps:

[0092] S015. Use the facial image corresponding to the cutoff frame as the input of the expression recognition model, and the expression label as the target output of the expression recognition model, and perform i prediction training until the loss function of the expression recognition model converges.

[0093] Specifically, the expression label corresponding to the face image is preset, for example, it is manually annotated by relevant personnel according to the expression of the face image corresponding to the cutoff frame, or the expression of the face image corresponding to the cutoff frame is obtained by one-hot encoding. Afterwards, the face image corresponding to the cutoff frame is used as the input of the expression identification model, the expression of the face image is predicted, and the predicted expression label is obtained, and then the cross entropy loss between the predicted expression label and the expression label corresponding to the face image is calculated, and the network parameters of the expression identification model are corrected according to the cross entropy loss. Afterwards, the face image corresponding to the cutoff frame continues to be used as the input of the expression identification model, and the expression identification model is trained for prediction until the loss function of the expression identification model converges to obtain the expression identification model.

[0094] In this way, after the second generation model is trained, the facial key points and the image to be edited are input into the second generation model to obtain the character expression image.

[0095] In some embodiments, the key points of the face and the image to be edited are fused to obtain the expression image of the person, or an image fusion algorithm is used to fuse the key points of the face and the image to be edited to obtain the expression image of the person, such as a weighted average fusion algorithm, a pyramid fusion algorithm, a gradient domain fusion algorithm, a wavelet transform algorithm, a structural deformation image fusion algorithm, etc.

[0096] In the above scheme, the expression editing device first obtains the image to be edited and the voice text information, and performs key point extraction processing on the image to be edited according to the voice text information to obtain the facial key points of the image to be edited. After that, the facial key points and the image to be edited are fused to obtain the character expression image constrained by the voice content. In this way, the expression state of the user when speaking is added to the facial expression editing process, avoiding the problem of deformation and artifacts of facial features in the image when facial expression editing is performed based on static information such as facial images and identity information alone, improving the quality of generated character expression images, and thus improving the user experience.

[0097] In some embodiments, after obtaining the facial key points corresponding to the image to be edited, Figure 8 As shown, the expression editing method also includes the following steps:

[0098] S121. Input the key points of the face into a first identification model to obtain a first identification result.

[0099] The first identification model is trained by the first output result and the facial key points corresponding to the cutoff frame of the preset video.

[0100] S122: Feedback the first identification result to the first generation model to correct the first generation model.

[0101] Specifically, when the first identification model receives the facial key points generated by the first generation model, it determines the identification result, and when it determines that the first identification result is false, it feeds back to the expression editing device that the first identification result is false. When the expression editing device receives the first identification result, it adjusts the network parameters of the first generation model according to the currently received first identification result to achieve the correction of the first generation model.

[0102] In some embodiments, after obtaining the character expression image, such as Fig. 9 As shown, the expression editing method also includes the following steps:

[0103] S1311. Input the facial expression image of the person into a second identification model to obtain a second identification result.

[0104] Among them, the second identification model is trained by the second output result and the cutoff frame of the preset video; the second output result is the output result of the first output result and the initial frame of the preset video in the second generation model.

[0105] S1311. Feedback the second identification result to the second generation model to correct the second generation model.

[0106] Specifically, when the second identification model receives the human expression image generated by the second generation model, it determines the identification result, and when it determines that the second identification result is false, it feeds back to the expression editing device that the second identification result is false. When the expression editing device receives the second identification result, it adjusts the network parameters of the second generation model according to the currently received second identification result to achieve the correction of the second generation model.

[0107] In the above scheme, the expression editing device can correct the first generation model according to the first identification result of the first identification model, correct the second generation model according to the second identification result of the second identification model, and correct the second generation model according to the expression identification result of the expression identification model, further ensuring that the final generated character expression image is consistent with the user's current emotional state and improving user satisfaction.

[0108] In some embodiments, the expression editing method further includes: inputting the facial key points corresponding to the image to be edited, or the character expression image, into the identification model to obtain an identification result, and determining that the identification result is true. Specifically, at least one embodiment is included:

[0109] In some embodiments, Fig.10As shown, after step S12, the expression editing method also includes the following steps S1: inputting facial key points into a first identification model to obtain a first identification result, and determining whether the first identification result is true, executing step S13 when it is determined that the first identification result is true, and executing step S12 when it is determined that the first identification result is false.

[0110] In some embodiments, Fig.11 As shown, after step S13, the expression editing method further includes the following step S2: inputting the character expression image into the second identification model to obtain a second identification result, and judging whether the second identification result is true, and executing step S14 when the second identification result is determined to be true, and executing step S13 when the second identification result is determined to be false. Step S14: outputting the character expression image.

[0111] In some embodiments, Fig.12 As shown, after step S13, the expression editing method further includes the following step S3: inputting the character expression image into the expression identification model, obtaining the expression identification result, and judging whether the expression identification result is true, executing step S14 when the expression identification result is determined to be true, and executing step S13 when the expression identification result is determined to be false. Step S14: outputting the character expression image.

[0112] In the above scheme, after the first generation model generates the character key points and the second generation model generates the character expression image, the expression editing device can use the first identification model, the second identification model and the expression identification model to identify the generated images respectively, and execute the next step when the identification model output is true, thereby avoiding the problems of deformation of facial features and artifacts in the image, ensuring the accuracy of the generated task expression image, thereby improving the quality of the generated character expression image and enhancing the user experience.

[0113] The embodiment of the present application can divide the functional modules of the expression editing device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation.

[0114] like Fig.13 As shown, an embodiment of the present application provides a structural schematic diagram of an expression editing device 200 , which includes a processor 101 and a communicator 102 .

[0115] The processor 101 is configured to obtain an image to be edited and a voice text from the communicator 102; wherein the voice text is a text obtained by converting the voice content that constrains the image to be edited into text; the image to be edited is a person image; the processor 101 is also configured to perform key point extraction processing on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited; wherein the number of extracted facial key points is greater than a preset threshold; the processor 101 is also configured to fuse the facial key points and the image to be edited to obtain a person expression image; the person expression in the person expression image is constrained by the voice content.

[0116] In some feasible examples, the processor 101 is further configured to: input the image to be edited and the voice text into the first generation model to obtain the facial key points in the image to be edited; the first generation model is trained with the preset voice text and the initial frame of the preset video as input and the cutoff frame of the preset video as output; the voice content corresponding to the preset voice text is the voice content of the preset video.

[0117] In some feasible examples, the processor 101 is further configured to: input the facial key points and the image to be edited into the second generation model to obtain a character expression image; the second generation model is trained with the initial frame of the preset video and the first output result as input and the cutoff frame of the preset video as output; the first output result is the output result of the preset voice text and the initial frame of the preset video in the first generation model.

[0118] In some feasible examples, the processor 101 is further configured to: input facial key points into a first identification model to obtain a first identification result; the first identification model is trained by the facial key points corresponding to the first output result and a cutoff frame of a preset video; and the first identification result is fed back to the first generation model to correct the first generation model.

[0119] In some feasible examples, the processor 101 is further configured to: input the character expression image into the second identification model to obtain a second identification result; the second identification model is trained by the second output result and the cutoff frame of the preset video; the second output result is the output result of the first output result and the initial frame of the preset video in the second generation model; and the second identification result is fed back to the second generation model to correct the second generation model.

[0120] In some feasible examples, the processor 101 is further configured to: input the facial key points corresponding to the image to be edited, or the character expression image into the identification model, obtain the identification result, and determine that the identification result is true.

[0121] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and its role will not be repeated here.

[0122] Of course, the expression editing device 200 provided in the embodiment of the present application includes but is not limited to the above modules, for example, the expression editing device 200 may also include a memory 103. The memory 103 may be used to store the program code of the expression editing device 200, and may also be used to store data generated by the expression editing device 200 during operation, such as data in a write request, etc.

[0123] As an example, combining Figure 3 The function implemented by the receiving unit 202 in the expression editing device 200 is the same as the function of the communicator 102, the function implemented by the processing unit 201 is the same as the function of the processor 101, and the function implemented by the storage unit 203 is the same as the function of the memory 103.

[0124] like Fig.14 As shown, an embodiment of the present application also provides a chip system, which can be applied to the expression editing device 200 in the aforementioned embodiment. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 can be the processor in the above-mentioned expression editing device 200. The processor 1501 and the interface circuit 1502 can be interconnected through a line. The processor 1501 can receive and execute computer instructions from the memory of the above-mentioned expression editing device 200 through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the expression editing device 200 can execute the various steps executed by the expression editing device 200 in the above-mentioned embodiment. Of course, the chip system can also include other discrete devices, which are not specifically limited in the embodiment of the present application.

[0125] The embodiment of the present application also provides a computer-readable storage medium for storing computer instructions executed by the above-mentioned expression editing device 200.

[0126] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An expression editing device, characterized in that: include: The processor is configured to obtain an image to be edited and a voice text; wherein the voice text is a text obtained by performing text conversion on the voice content constraining the image to be edited; and the image to be edited is a person image; The processor is further configured to perform key point extraction processing on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited; wherein the number of the extracted facial key points is greater than a preset threshold; The processor is further configured to fuse the facial key points with the image to be edited to obtain a character expression image; the character expression in the character expression image is constrained by the voice content.

2. The expression editing device according to claim 1, characterized in that: The processor is specifically configured to: Inputting the image to be edited and the speech text into a first generation model to obtain key points of the face in the image to be edited; the first generation model is trained by taking a preset speech text and an initial frame of a preset video as input and taking a cutoff frame of the preset video as output; The voice content corresponding to the preset voice text is the voice content of the preset video.

3. The expression editing device according to claim 1, characterized in that: The processor is specifically configured to: Inputting the facial key points and the image to be edited into a second generation model to obtain a character expression image; the second generation model is trained by taking the initial frame of a preset video and the first output result as input and the cutoff frame of the preset video as output; The first output result is the output result of the preset voice text and the initial frame of the preset video in the first generation model.

4. The expression editing device according to claim 2, characterized in that: The processor is further configured to: Inputting the facial key points into a first identification model to obtain a first identification result; the first identification model is trained by the first output result and the facial key points corresponding to the cutoff frame of the preset video; The first identification result is fed back to the first generation model to correct the first generation model.

5. The expression editing device according to claim 3, characterized in that: The processor is further configured to: Inputting the character expression image into a second identification model to obtain a second identification result; the second identification model is obtained by training the second output result and the cutoff frame of the preset video; The second output result is the output result of the first output result and the initial frame of the preset video in the second generation model; The second identification result is fed back to the second generation model to correct the second generation model.

6. The expression editing device according to any one of claims 1 to 5, characterized in that: The processor is further configured to: The key points of the human face corresponding to the image to be edited, or the human expression image, are input into the identification model to obtain an identification result, and the identification result is determined to be true.

7. A method for editing facial expressions, characterized in that: Acquire an image to be edited and a voice text; wherein the voice text is a text obtained by converting the voice content constraining the image to be edited; and the image to be edited is a person image; Performing key point extraction processing on the image to be edited according to the voice text to obtain facial key points corresponding to the image to be edited; wherein the number of the extracted facial key points is greater than a preset threshold; The facial key points and the image to be edited are fused to obtain a character expression image; the character expression in the character expression image is constrained by the voice content.

8. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the expression editing method described in claim 7 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the expression editing method described in claim 7.

10. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to implement the expression editing method described in claim 7.

Citation Information

Cited By

  • Expression editing method based on facial component, electronic equipment and readable storage medium

    CN119152082A

  • Expression editing method based on facial components, electronic device and readable storage medium

    CN119152082B