Virtual image driving method, driving device and readable medium
By using a closed-mouth image generation model to adjust the differences in face and lip shape of virtual avatars under silent audio, the problem of mismatch between face and lips of virtual avatars when silent is solved, and a more natural virtual avatar performance is achieved.
Patent Information
- Application Number
- CN202411046412.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-11-19
AI Technical Summary
In existing technologies, when the audio switches from speaking to silence during the driving process, the face and lips of a virtual avatar do not match, resulting in unnatural movement.
By acquiring the driving audio sequence, a closed-mouth image generation model is used to generate closed-mouth images under silent conditions. The differences in face shape and lip shape are adjusted to ensure that the virtual image presents a natural closed-mouth state under silent audio.
This achieved a natural, silent state for the virtual avatar under muted audio, improving the realism and watchability of the video and enhancing audience acceptance.
Smart Images

Figure CN118982606A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of virtual image driving, and more specifically, to a virtual image driving method, a driving device and a readable medium. Background Art
[0002] In the process of driving the virtual image by driving audio, the face of the virtual image may be in a state corresponding to speaking, and when the driving audio is a silent audio corresponding to not speaking, the related technology only drives the lips to close, resulting in a mismatch between the face and the lips. Summary of the invention
[0003] The present application is provided to solve the above defects in the prior art. A virtual image driving method, a driving device and a readable medium are needed to solve the problem that the face still moves in the closed-mouth video portion during the virtual image driving process.
[0004] According to the first scheme of the present application, a method for driving a virtual image is provided, and the method comprises the following steps. A driving audio sequence is obtained, wherein the driving audio sequence comprises a plurality of driving audios. One of the plurality of driving audios is sequentially used as a target driving audio, and a target virtual image is driven by the target driving audio; the target virtual image presents a state corresponding to the target driving audio under the drive of the target driving audio; wherein, when the target driving audio satisfies a mute condition, a closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
[0005] In some embodiments, calling the closed-mouth image generation model to obtain the closed-mouth image includes:
[0006] Obtaining a reference closed-mouth image, silent audio, and a target image, wherein the target image is a face image after the mouth image is eliminated;
[0007] The reference closed-mouth image, the silent audio and the target image are input into the closed-mouth image generation model to obtain the closed-mouth image output by the closed-mouth image generation model.
[0008] In some embodiments, the closed mouth image generation model is obtained by:
[0009] Determine facial feature difference information based on the reference closed-mouth image and the initial closed-mouth image output by the initial model;
[0010] Obtaining image quality feature information based on the initial closed-mouth image;
[0011] Obtaining lip shape difference information based on the initial closed-mouth image and the silent audio;
[0012] The initial model is adjusted based on the facial feature difference information, the image quality feature information and the lip shape difference information to obtain the closed-mouth image generation model.
[0013] In some embodiments, the initial model outputs an initial closed-mouth image, including: inputting a reference closed-mouth image, a first image, and silent audio into the initial model to obtain the initial closed-mouth image output by the initial model, wherein the first image is a face image after eliminating the mouth image.
[0014] In some embodiments, inputting the reference closed-mouth image, the first image, and the silent audio into the initial model to obtain the initial closed-mouth image includes:
[0015] Determine encoding information corresponding to the reference closed-mouth image, the first image, and the silent audio respectively;
[0016] Based on the coding information corresponding to the reference closed-mouth image, the first image and the silent audio, decoding is performed to obtain an initial closed-mouth image.
[0017] In some embodiments, determining facial feature difference information based on the reference closed-mouth image and the initial closed-mouth image output by the initial model includes:
[0018] Determine first face shape information corresponding to the initial closed-mouth image and second face shape information corresponding to the reference closed-mouth image;
[0019] Determining semantic information corresponding to the initial closed-mouth image;
[0020] Based on the first face shape information, the second face shape information and the semantic information, face shape feature difference information is determined.
[0021] In some embodiments, the method further comprises:
[0022] The face shape feature difference information is determined using a face shape discrimination model, wherein the face shape discrimination model is obtained in the following manner:
[0023] Based on the initial face shape feature difference information obtained by the initial face shape discrimination model and the initial closed-mouth image, pixel adjustment is performed on the initial closed-mouth image to obtain an adjusted target closed-mouth image;
[0024] The face shape discrimination initial model is adjusted based on the initial closed-mouth image and the adjusted target closed-mouth image to obtain an adjusted face shape discrimination model.
[0025] In some embodiments, adjusting the initial model based on the face shape feature difference information, the image quality feature information, and the lip shape difference information includes:
[0026] The weights of the lip shape difference information, the image quality feature information, and the face shape feature difference information on the initial model adjustment decrease in sequence.
[0027] According to the second scheme of the present application, a virtual image driving device is provided, and the generating device includes an acquisition module and a processing module. The acquisition module is used to acquire a driving audio sequence and a target virtual image. The processing module sequentially uses one of the multiple driving audios as a target driving audio, and drives the target virtual image through the target driving audio; the target virtual image presents a state corresponding to the target driving audio under the drive of the target driving audio; wherein, when the target driving audio satisfies the mute condition, a closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
[0028] According to a third solution of the present application, a computer-readable medium is provided, on which instructions are stored. When executed by a processor, the instructions execute the steps of the virtual image driving method as described in any embodiment of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In the drawings, which are not necessarily drawn to scale, the same reference numerals may describe similar parts in different views. The same reference numerals with letter suffixes or different letter suffixes may represent different instances of similar parts. The accompanying drawings generally illustrate various embodiments by way of example and not limitation, and together with the specification and claims, are used to illustrate the embodiments applied for. When appropriate, the same reference numerals are used throughout the drawings to refer to the same or similar parts. Such embodiments are illustrative and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.
[0030] Figure 1 A flowchart showing a method for driving a virtual image according to an embodiment of the present application is shown;
[0031] Figure 2 According to the embodiment of the present application Figure 1 A flowchart of an embodiment of step S300;
[0032] Figure 3 The embodiment according to the present application is shown Figure 1 A flowchart of another embodiment of step S300;
[0033] Figure 4 The embodiment according to the present application is shown Figure 3 A flowchart of an embodiment of step S3001;
[0034] Figure 5 A structural block diagram of a virtual image driving device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0035] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described in detail below in conjunction with the accompanying drawings and specific implementation examples. The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and specific implementation examples, but are not intended to limit the present application.
[0036] The words "first", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish. The words "include" or "comprise" and similar words mean that the elements before the word include the elements listed after the word, and do not exclude the possibility of including other elements.
[0037] To solve the problems in the background technology, such as Figure 1 As shown, an embodiment of the present application provides a virtual image driving method, which includes the following steps.
[0038] S100: Obtain a driving audio sequence, where the driving audio sequence includes a plurality of driving audios.
[0039] Exemplarily, the driving audio sequence may be a driving audio sequence pre-stored in a certain storage location, and the driving audio sequence may be obtained from the storage location by calling a function or the like. The driving audio sequence includes a plurality of driving audios, and the plurality of driving audios may include audible driving audios and silent driving audios, and the silent driving audios may also be referred to as silent driving audios. For example, the driving audio sequence may include one silent driving audio, or may include more than two silent driving audios. There is no specific limit to the number of audible driving audios and silent driving audios. The plurality of driving audios are obtained by tokenizing and segmenting the original driving audios, that is, each driving audio is an audio token.
[0040] For example, when a person is giving a speech, he may not speak all the time, but may need to take a break or pause for a period of time due to special reasons. The audio sequence generated during the entire speech can be understood as the above-mentioned driving audio sequence; the audio generated during the speech of the speaker can be understood as the above-mentioned audible driving audio; and the audio generated during the rest or pause can be understood as the above-mentioned silent driving audio.
[0041] S200, sequentially taking one of the multiple driving audios as a target driving audio, and driving a target virtual image through the target driving audio; the target virtual image presents a state corresponding to the target driving audio under the drive of the target driving audio. The target virtual image can be obtained through one of the following: a virtual character image, a real character photo, a real character video, a virtual animal image, a real animal photo, and a real animal video. The state corresponding to the target driving audio includes the target virtual image presenting a target action corresponding to the target driving audio, and the target action includes at least one of the following: facial expression action, lip action, and body action.
[0042] Exemplarily, multiple driving audios in the driving audio sequence may be numbered, and the multiple driving audios may be numbered according to, but not limited to, the continuity of the driving audio sequence. Continuing with the above embodiment, in order to ensure the smoothness of the audible driving audio or silent driving audio generated during the speech, the multiple driving audios generated during the speech may be numbered in the order of precedence, and the corresponding driving audio (one driving audio) may be used as the target driving audio in the order of the numbering. Of course, it is understandable that the order of the multiple driving audios generated during the speech may also be adjusted, the driving audios may be numbered in the order of the adjustment, and then one of the numbered driving audios may be used as the target driving audio in turn.
[0043] The target virtual image is driven by the target driving audio, where the target virtual image can be determined according to the type of driving audio. For example, if the sender of the sound in the driving audio is a person, the target virtual image can be a person; of course, the target virtual image can be set accordingly according to the person's gender, age and other characteristics, so that the target virtual image can more accurately reflect the driving audio characteristics of the person, so that the audience watching the generated video can have a better viewing experience and feeling. If the sender of the sound in the driving audio simulates an animal image speaking, such as simulating a lion speaking, the target virtual image can be a corresponding animal image, such as a lion.
[0044] The target virtual image is driven by the target driving audio, so that the target virtual image presents a dynamic or static video. Taking the virtual image of a human face as an example, when the target driving audio is an audible audio, an audible video is generated, and when the target driving audio is a silent audio, a silent video is generated.
[0045] For audio with sound, the target virtual image can be driven to produce state changes based on the audio with sound, such as opening of the mouth and changes in the face, etc. In some embodiments, for audio with sound, the target virtual image is driven to present a state corresponding to the audio with sound based on the target driving model, such as SadTalker, Wav2lip model, etc.
[0046] The target virtual image may be an image in a target video, and driving the target virtual image through the target driving audio includes: according to the sequence correspondence between the video frame and the plurality of driving audios in the target video or the playback progress correspondence, driving the image contained in the corresponding target frame in the target video through the target driving audio. There is at least one target frame corresponding to each target driving audio, and at least one target driving audio corresponding to each target frame. The target video may be a video corresponding to a character, and the image in the target video is the character image. The "virtual" in the target virtual image means that the image is a drivable image, which can present a state that the original real image does not have after being driven by the audio. The "virtual" in the target virtual image does not mean that the image is an animation-synthesized image.
[0047] For silent audio, calling the closed-mouth image generation model can generate a closed-mouth image based on the target virtual image. The closed-mouth image makes the target virtual image appear to have its mouth closed instead of showing its face still moving, making the video more realistic.
[0048] The target virtual image can be presented in a state corresponding to the target driving audio under the driving effect of the target driving audio. For example, in combination with the above-mentioned speaker embodiment, the target virtual image is a male teenager. During the speech, there may be some special emotions, such as surprise or sadness. The target virtual image can present a corresponding expression of surprise or sadness under the driving of the target driving audio corresponding to the above emotions; and when the speaker speaks normally, the corresponding audible driving audio drives the face and lips of the target virtual image to make corresponding movements, keeping the target virtual image presenting the facial and lip movements corresponding to the audible driving audio, so that the target virtual image is more in line with the target driving audio, so that the generated video can more accurately reflect the target driving audio, improve the viewing quality of the video, and also improve the audience's acceptance.
[0049] S300, wherein, when the target driving audio satisfies a mute condition, a closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
[0050] The goal-driven model is different from the closed-mouth image generation model.
[0051] Exemplarily, in combination with the above embodiment, the multiple driving audios included in the driving audio sequence generated during the speech process may include silent driving audio. Here, in the process of generating the driving audio sequence, the driving audio generated during the time period when the speaker does not make a speech can be divided into silent driving audio. That is, it is avoided that a piece of audio includes both the audio in the corresponding time period when the speaker makes a speech and the audio in the corresponding time period when the speaker does not make a speech. Of course, it should be specially noted here that the driving audio may include the pauses made during the normal speech of the speaker.
[0052] When the target driving audio satisfies the mute condition, that is, when the target driving audio is the corresponding mute driving audio, the closed-mouth image generation model can be called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image appear in a closed-mouth state. Here, the closed-mouth image obtained by the closed-mouth image generation model corresponds to the target virtual image. The target driving audio that satisfies the mute condition does not include speech sounds. The mute driving audio is audio that does not include speech sounds.
[0053] For example, if the target virtual image is a male teenager, the closed-mouth image here is also the image corresponding to the male teenager with his mouth closed. In the process of driving the target virtual image with the corresponding silent driving audio, the closed-mouth image is the target virtual phenomenon presenting a closed-mouth state, or it can be understood that the presentation form of the target virtual image is the closed-mouth image.
[0054] By directly calling and presenting the closed-mouth image, the state displayed by the corresponding target virtual image can be adjusted to the closed-mouth state. The face and lip movements of the target virtual image do not need to be adjusted, but are directly replaced by the corresponding static face and lip states in the closed-mouth image, thereby realizing the rapid switching of the face and lip movements of the virtual image when the mouth is closed and muted. At the same time, the accuracy and smoothness of the corresponding video generated under the drive of the driving audio sequence including the audible driving audio and the silent driving audio are guaranteed.
[0055] In some embodiments, Figure 2 As shown, the calling of the closed-mouth image generation model to obtain the closed-mouth image includes:
[0056] S301, obtaining a reference closed-mouth image, silent audio, and a target image, wherein the target image is a face image after the mouth image is eliminated.
[0057] Exemplarily, some images are stored in an image library, and an image can be called from the image library as a reference closed-mouth image, and the reference closed-mouth image corresponds to the target virtual image. Silent audio can also be pre-stored in a certain storage space, and the silent audio can be directly called from the storage space. The method of obtaining the target image can be similar to the method of obtaining the reference closed-mouth image, that is, some images are stored in the target image library, and an image can be called from the target image library as the target image. When the target virtual image is a person, the target image is a face image after eliminating the mouth image. The face image corresponds to the target virtual image. Of course, the reference closed-mouth image, silent audio and target image can also be obtained in other ways. This is only used as an example and is not enough to limit the scope of protection of the claims.
[0058] The reference closed-mouth image is an image of a closed mouth that is close to a natural state. The target image is a face image after the mouth image is eliminated. In one embodiment, the pixel value of the mouth of the target image can be set to 0, or the pixel value of the mouth of the target image can be set to 255, or the pixel value of the mouth of the target image can be set to a pixel value close to 0, such as 1, 2, etc., or the pixel value of the mouth of the target image can be set to a pixel value close to 255, such as 250, 253, etc. The original pixel value of the mouth on the target image is removed to facilitate the generation of a new pixel value of the mouth.
[0059] The closed-mouth image generation model uses the reference closed-mouth image as a reference for generating the closed-mouth image, so that the generated closed-mouth image can be closer to a natural state, and the problem of the face still moving will not occur in the closed-mouth image.
[0060] S302, inputting the reference closed-mouth image, the silent audio and the target image into the closed-mouth image generation model to obtain the closed-mouth image output by the closed-mouth image generation model.
[0061] Exemplarily, the reference closed mouth image, the silent audio and the target image are input into the closed mouth image generation model as input parameters of the closed mouth image generation model, and the closed mouth image generation model generates a corresponding output based on at least the reference closed mouth image, the silent audio and the target image, and the output is a closed mouth image. The closed mouth image may include a part of the target image and a part of the reference closed mouth image, for example, a part of the target image is the part above the mouth of the target image, that is, the part of the facial image that has not been eliminated. A part of the reference closed mouth image is used to fill in the missing part of the target image, that is, the mouth and the part below the mouth, so that the closed mouth image is a complete facial image. The closed mouth image corresponds to the silent audio part, that is, the closed mouth image output by the closed mouth image generation model does not emit any sound.
[0062] In some embodiments, Figure 3As shown, the closed mouth image generation model is obtained by:
[0063] S3001, determining facial feature difference information based on a reference closed-mouth image and an initial closed-mouth image output by an initial model.
[0064] Exemplarily, the initial model may first output an initial closed mouth image, and there may be some differences between the initial closed mouth image and the reference closed mouth image. Therefore, the reference closed mouth image and the initial closed mouth image may be compared in face shape to determine the face shape feature difference information between the reference closed mouth image and the initial closed mouth image. For example, the face shape of the reference closed mouth image is round, while the initial closed mouth image is long, and the two face shapes are different. The determined face shape difference information may be recorded, for example, the face shape information of the reference closed mouth image is recorded as: round face; the face shape information of the initial closed mouth image is recorded as: long face, so as to be used in the subsequent adjustment of the initial model. Alternatively, it can be understood that the initial closed mouth image output by the initial model may have a certain gap in face shape with the reference closed mouth image, so the image information of the reference closed mouth image and the initial closed mouth image can be compared to obtain face shape feature difference information. Face shape features include feature information such as facial contours.
[0065] S3002: Obtain image quality feature information based on the initial closed-mouth image.
[0066] Exemplarily, the generated initial closed-mouth image includes but is not limited to image quality feature information such as resolution, color saturation, and signal-to-noise ratio. The above image quality feature information of the initial closed-mouth image can be extracted and the obtained image quality feature information can be recorded for use in subsequent adjustments to the initial model.
[0067] S3003: Obtain lip shape difference information based on the initial closed-mouth image and the silent audio.
[0068] For example, the initial closed-mouth image obtained by the initial model may have a lip shape that is different from the lip shape of the closed-mouth image expected in the closed-mouth state. Therefore, corresponding to the silent audio, the lip shape difference information between the lip shape of the initial closed-mouth image and the closed-mouth image expected in the closed-mouth state is obtained, for example, the upper lip is offset upward and the lower lip is offset downward. The lip shape difference information is recorded for use in subsequent adjustments to the initial model.
[0069] It should be noted that the silent audio here may be a non-silent driving audio, the initial closed-mouth image may be an image corresponding to the silent audio, and the initial closed-mouth image and the silent audio are used for training a closed-mouth image generation model.
[0070] S3004, adjusting the initial model based on the facial feature difference information, the image quality feature information and the lip shape difference information to obtain the closed mouth image generation model.
[0071] Exemplarily, the initial model is adjusted based on facial feature difference information, image quality feature information and lip shape difference information. For example, parameters related to the face shape, image quality and lip shape of the output closed mouth image in the initial model are adjusted according to the above information to obtain a closed mouth image generation model that can output more accurate closed mouth images.
[0072] That is, the corresponding face shape loss function, image quality loss function and lip shape loss function can be determined based on the face shape feature difference information, image quality feature information and lip shape difference information respectively, and then the initial model is adjusted using each loss function to obtain the closed mouth image generation model, so that the adjusted initial model can output a closed mouth image that is closer to the natural state, and avoid the facial state in the generated closed mouth image still showing a moving state.
[0073] For example, in combination with the above embodiment, the face shape information of the reference closed mouth image is recorded as: round face; the face shape information of the initial closed mouth image is recorded as: long face; the facial contours are different. In the above case, the parameters related to the face shape of the output closed mouth image in the initial model can be adjusted so that the parameters can adjust the face shape of the closed mouth image to be more rounded when the closed mouth image is output, thereby enabling the obtained closed mouth image generation model to output a more accurate closed mouth image.
[0074] In one embodiment, the initial model outputs an initial closed-mouth image, including:
[0075] The reference closed-mouth image, the first image and the silent audio are input into the initial model to obtain the initial closed-mouth image output by the initial model, wherein the first image is the face image after the mouth image is eliminated.
[0076] Exemplarily, the first image is a face image after the mouth image is eliminated. Here, the first image is mainly used for training the initial model. Although the first image and the target image represent basically the same meaning, the two are respectively applied in the initial model and the closed-mouth image generation model generated after training. The two are not the same image.
[0077] In one embodiment, the first image may be obtained by: setting the pixel value of the mouth of the face image to 0, or setting the pixel value of the mouth of the face image to 255, or setting the pixel value of the mouth of the face image to a pixel value close to 0, such as 1, 2, etc., or setting the pixel value of the mouth of the face image to a pixel value close to 255, such as 250, 253, etc. The original pixel value of the mouth on the face image is removed to facilitate the generation of a new pixel value of the mouth to fill the removed portion of the first image.
[0078] In some embodiments, Figure 4As shown, the reference closed-mouth image, the first image and the silent audio are input into the initial model to obtain the initial closed-mouth image, including:
[0079] S30011, respectively determine the coding information corresponding to the reference closed-mouth image, the first image and the silent audio.
[0080] Exemplarily, a reference closed-mouth image can be encoded by a first encoder to determine first encoding information corresponding to the reference closed-mouth image; a first image can be encoded by a second encoder to determine second encoding information corresponding to the first image; and silent audio can be encoded by a third encoder to determine third encoding information corresponding to the silent audio.
[0081] S30012, based on the coding information corresponding to the reference closed-mouth image, the first image and the silent audio, decoding is performed to obtain an initial closed-mouth image.
[0082] Exemplarily, the first coding information of the reference closed mouth image, the second coding information of the first image, and the third coding information of the silent audio are summarized to form total coding information including the first coding information, the second coding information, and the third coding information. The total coding information can be decoded by a decoder to obtain an initial closed mouth image, so that a closed mouth image close to a natural state can be combined. This is only used as an example and does not constitute a limitation on the scope of protection of the claims. For example, the first coding information of the reference closed mouth image, the second coding information of the first image, and the third coding information of the silent audio can be decoded by three decoders respectively, and then the decoded images are integrated to obtain the initial closed mouth image.
[0083] In some embodiments, determining facial feature difference information based on the reference closed-mouth image and the initial closed-mouth image output by the initial model includes:
[0084] S30013, determining first face shape information corresponding to the initial closed-mouth image and second face shape information corresponding to the reference closed-mouth image.
[0085] Exemplarily, the facial contour of the initial closed mouth image may be drawn, and the drawing result may reflect the face shape information of the initial closed mouth image, and the face shape information is the first face shape information corresponding to the initial closed mouth image. Similarly, the facial contour of the reference closed mouth image may be drawn, and the drawing result may reflect the face shape information of the reference closed mouth image, and the face shape information is the second face shape information corresponding to the reference closed mouth image.
[0086] S30014, determining semantic information corresponding to the initial closed-mouth image.
[0087] For example, the initial closed mouth image can be processed and analyzed by tools such as OpenCV (Open Source Computer Vision Library), TensorFlow, PyTorch, etc., so as to extract the corresponding semantic information from the initial closed mouth image. The semantic information extraction process can include steps such as image segmentation, object recognition, and feature extraction. The semantic information extraction method can be implemented by conventional semantic extraction technology, which will not be described in detail here.
[0088] For example, semantic information is the meaning or information conveyed by the initial closed-mouth image that can be understood or interpreted by humans, and may include specific closed-mouth postures, emotional states (such as contemplation, anger, calmness, etc.), and the like.
[0089] S30015. Determine facial feature difference information based on the first facial shape information, the second facial shape information and the semantic information.
[0090] Exemplarily, the first face shape information and the second face shape information are compared to determine the first face shape feature difference information between the reference closed mouth image and the initial closed mouth image. Of course, it is understandable that the first face shape feature difference information between the closed mouth image and the initial closed mouth image can also be determined based on the first face shape information, the second face shape information, and information such as the closed mouth posture in the semantic information.
[0091] Based on the semantic information and the calm and emotionless semantic information corresponding to the reference closed-mouth image, second facial feature difference information between the reference closed-mouth image and the initial closed-mouth image is determined.
[0092] The first face shape feature difference information and the second face shape feature difference information together constitute the face shape feature difference information.
[0093] Of course, the facial shape feature difference information may also be determined directly through the first facial shape information, the second facial shape information and the semantic information.
[0094] In some embodiments, the method further comprises:
[0095] The face shape feature difference information is determined using a face shape discrimination model, wherein the face shape discrimination model is obtained in the following manner:
[0096] Based on the initial face shape feature difference information obtained by the initial face shape discrimination model and the initial closed-mouth image, pixel adjustment is performed on the initial closed-mouth image to obtain an adjusted target closed-mouth image.
[0097] Exemplarily, initial face shape feature difference information can be obtained based on the face shape discrimination initial model, and then pixels of the initial closed mouth image are adjusted based on the initial face shape feature difference information and the initial closed mouth image to obtain an adjusted target closed mouth image.
[0098] For example, the face shape discrimination initial model can preliminarily judge the face shape of a person based on the measurement and observation results of the person. In one embodiment, the measurement and observation results of the person are: the widest part of the face is the cheekbones, the length of the face is slightly larger than the width, the forehead is of moderate width, and the chin is round. Based on these features, it can be preliminarily judged that the face shape of the person is an oval face.
[0099] The face shape discrimination initial model is adjusted based on the initial closed-mouth image and the adjusted target closed-mouth image to obtain an adjusted face shape discrimination model.
[0100] Exemplarily, the initial face shape discrimination model is adjusted according to the initial closed-mouth image and the adjusted target closed-mouth image, so that the adjusted face shape discrimination model can more accurately determine the face shape feature difference information.
[0101] In some embodiments, adjusting the initial model based on the face shape feature difference information, the image quality feature information, and the lip shape difference information includes:
[0102] The weights of the lip shape difference information, the image quality feature information, and the face shape feature difference information on the initial model adjustment decrease in sequence.
[0103] Exemplarily, the lip shape difference information has the greatest impact on the accuracy of the initial closed mouth image generated by the initial model, followed by the image quality feature information and the face shape feature difference information. Therefore, when adjusting the initial model, the weight of the lip shape difference information on the initial model adjustment can be appropriately increased, and the weight of the image quality feature information and the face shape feature difference information on the initial model adjustment can be appropriately reduced. In one embodiment, the weight of the lip shape difference information on the initial model adjustment can be 50%, the weight of the image quality feature information on the initial model adjustment can be 30%, and the weight of the face shape feature difference information on the initial model adjustment can be 20%. Of course, it can be understood that the weight ratio of the lip shape difference information, the image quality feature information, and the face shape feature difference information on the initial model adjustment can be adjusted according to specific needs. This is only used as an example and does not constitute a limitation on the scope of protection of the claims.
[0104] In addition, the weight ratio of the lip shape difference information, the image quality feature information, and the face shape feature difference information to the initial model adjustment may be set by other weight division methods.
[0105] According to an embodiment of the present application, a virtual image driving device is also provided, such as Figure 5As shown, the generating device includes an acquisition module and a processing module. The acquisition module is used to acquire a driving audio sequence and a target virtual image. The processing module sequentially uses one of the multiple driving audios as a target driving audio, and drives the target virtual image through the target driving audio; the target virtual image presents a state corresponding to the target driving audio under the drive of the target driving audio; wherein, when the target driving audio meets the mute condition, the closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
[0106] In some embodiments, the processing module includes an encoding module, and the encoding module includes a convolutional neural network (CNN), an autoencoder, a recurrent neural network, etc.
[0107] In some embodiments, the processing module is further configured to:
[0108] A reference closed-mouth image, silent audio, and a target image are obtained, wherein the target image is a face image after the mouth image is eliminated.
[0109] The reference closed-mouth image, the silent audio and the target image are input into the closed-mouth image generation model to obtain the closed-mouth image output by the closed-mouth image generation model.
[0110] In some embodiments, the processing module is further configured to:
[0111] Based on the reference closed-mouth image and the initial closed-mouth image output by the initial model, facial feature difference information is determined.
[0112] Image quality feature information is obtained based on the initial closed-mouth image.
[0113] Lip shape difference information is obtained based on the initial closed-mouth image and the silent audio.
[0114] The initial model is adjusted based on the facial feature difference information, the image quality feature information and the lip shape difference information to obtain the closed-mouth image generation model.
[0115] In some embodiments, the processing module is further configured to:
[0116] The reference closed-mouth image, the first image and the silent audio are input into the initial model to obtain the initial closed-mouth image output by the initial model, wherein the first image is the face image after the mouth image is eliminated.
[0117] In some embodiments, the processing module is further configured to:
[0118] The coding information corresponding to the reference closed-mouth image, the first image and the silent audio is determined respectively.
[0119] Based on the coding information corresponding to the reference closed-mouth image, the first image and the silent audio, decoding is performed to obtain an initial closed-mouth image.
[0120] Determine first face shape information corresponding to the initial closed-mouth image and second face shape information corresponding to the reference closed-mouth image.
[0121] Determine semantic information corresponding to the initial closed-mouth image.
[0122] Based on the first face shape information, the second face shape information and the semantic information, face shape feature difference information is determined.
[0123] In some embodiments, the processing module is further configured to:
[0124] The face shape feature difference information is determined using a face shape discrimination model, wherein the face shape discrimination model is obtained in the following manner:
[0125] Based on the initial face shape feature difference information obtained by the initial face shape discrimination model and the initial closed-mouth image, pixel adjustment is performed on the initial closed-mouth image to obtain an adjusted target closed-mouth image.
[0126] The face shape discrimination initial model is adjusted based on the initial closed-mouth image and the adjusted target closed-mouth image to obtain an adjusted face shape discrimination model.
[0127] According to an embodiment of the present application, a computer-readable medium is also provided, on which instructions are stored. When executed by a processor, the instructions execute the steps of the virtual image driving method as described in any embodiment of the present application.
[0128] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, electronic equipment, computer-readable storage media or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code. When implemented by software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0129] The processor may be a general purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. The general purpose processor may be a microprocessor or any conventional processor.
[0130] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.
[0131] The above-mentioned readable storage medium can be a disk, an optical disk, a DVD, a USB, a read-only memory (ROM) or a random access memory (RAM), etc. This application does not limit the specific form of the storage medium.
[0132] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the implementation of the application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.
[0133] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the application. This should not be interpreted as a feature of an application that does not require protection being necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of the embodiments of a specific application. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the full scope of the equivalent forms of the attached claims and these claims.
[0134] The above embodiments are only exemplary embodiments of the present application and are not intended to limit the present invention. The protection scope of the present invention is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.
Claims
1. A method for driving a virtual image, the method comprising: include: Obtaining a driving audio sequence, wherein the driving audio sequence includes a plurality of driving audios; sequentially using one of the plurality of driving audios as a target driving audio, and driving a target virtual image through the target driving audio; The target virtual image presents a state corresponding to the target driving audio under the driving of the target driving audio; Wherein, when the target driving audio satisfies the mute condition, a closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
2. The method according to claim 1, wherein the closed-mouth image generation model is called to obtain the closed-mouth image, include: Obtaining a reference closed-mouth image, silent audio, and a target image, wherein the target image is a face image after the mouth image is eliminated; The reference closed-mouth image, the silent audio and the target image are input into the closed-mouth image generation model to obtain the closed-mouth image output by the closed-mouth image generation model.
3. According to the method of claim 1, the closed mouth image generation model is obtained by: Determine facial feature difference information based on the reference closed-mouth image and the initial closed-mouth image output by the initial model; Obtaining image quality feature information based on the initial closed-mouth image; Obtaining lip shape difference information based on the initial closed-mouth image and the silent audio; The initial model is adjusted based on the facial feature difference information, the image quality feature information and the lip shape difference information to obtain the closed-mouth image generation model.
4. The method according to claim 3, wherein the initial model outputs an initial closed-mouth image, include: The reference closed-mouth image, the first image and the silent audio are input into the initial model to obtain the initial closed-mouth image output by the initial model, wherein the first image is the face image after the mouth image is eliminated.
5. The method according to claim 4, inputting the reference closed-mouth image, the first image and the silent audio into the initial model to obtain an initial closed-mouth image, include: Determine encoding information corresponding to the reference closed-mouth image, the first image, and the silent audio respectively; Based on the coding information corresponding to the reference closed-mouth image, the first image and the silent audio, decoding is performed to obtain an initial closed-mouth image.
6. The method according to claim 3, determining facial feature difference information based on the reference closed-mouth image and the initial closed-mouth image output by the initial model, include: Determine first face shape information corresponding to the initial closed-mouth image and second face shape information corresponding to the reference closed-mouth image; Determining semantic information corresponding to the initial closed-mouth image; Based on the first face shape information, the second face shape information and the semantic information, face shape feature difference information is determined.
7. The method according to claim 3, further comprising: include: The face shape feature difference information is determined using a face shape discrimination model, wherein the face shape discrimination model is obtained in the following manner: Based on the initial face shape feature difference information obtained by the initial face shape discrimination model and the initial closed-mouth image, pixel adjustment is performed on the initial closed-mouth image to obtain an adjusted target closed-mouth image; The face shape discrimination initial model is adjusted based on the initial closed-mouth image and the adjusted target closed-mouth image to obtain an adjusted face shape discrimination model.
8. The method according to claim 3, adjusting the initial model based on the face shape feature difference information, the image quality feature information and the lip shape difference information, include: The weights of the lip shape difference information, the image quality feature information, and the face shape feature difference information on the initial model adjustment decrease in sequence.
9. A virtual image driving device, the driving device include: An acquisition module, used for acquiring a driving audio sequence and a target virtual image; A processing module, sequentially using one of the plurality of driving audios as a target driving audio, and driving a target virtual image through the target driving audio; The target virtual image presents a state corresponding to the target driving audio when driven by the target driving audio; wherein, when the target driving audio satisfies the mute condition, a closed-mouth image generation model is called to obtain a closed-mouth image, and the closed-mouth image is used to make the target virtual image present a closed-mouth state.
10. A computer-readable medium having instructions stored thereon, which, when executed by a processor, execute the steps of the virtual image driving method according to any one of claims 1 to 8.
Citation Information
Cited By
Video generation method and device based on reference video, computing equipment and medium
CN119697433A
Video generation method, device, computing equipment and medium based on reference video
CN119697433B
