Method, device, equipment, medium and product for generating virtual image video
By adjusting the character phoneme pronunciation characteristics of the target audio to generate facial driving parameters of virtual avatar video, the problems of poor virtual avatar video generation and limited application scenarios are solved, and more natural and intelligent video generation is achieved, reducing hardware dependence and cost.
Patent Information
- Application Number
- CN202210573718.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-05-24
AI Technical Summary
The existing virtual image video generation method has poor generation effect in some scenarios, and the application scenarios are limited, making it difficult to meet the diverse interaction needs of users.
By adjusting the pronunciation characteristics of the character phonemes in the target audio based on the melody characteristics of the target audio, the facial driving parameters of the virtual image model are determined, and these parameters are used to generate virtual image videos, thereby improving the naturalness and intelligence of the video.
It effectively improves the naturalness and intelligence of virtual image videos, reduces hardware equipment requirements, reduces generation costs, expands application scenarios, and meets users' diverse interaction needs.
Smart Images

Figure CN114882151B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of artificial intelligence technology, specifically the field of computer vision technology, and can be applied to scenes such as 3D vision, augmented reality, virtual reality, etc., and specifically relates to a method for generating a virtual image video and its device, equipment, medium and product. Background Art
[0002] The generation of virtual image videos is widely used in 3D vision, augmented reality, virtual reality and other scenarios. However, in some scenarios, the generation methods of virtual image videos have the phenomenon of poor generation effect and limited application scenarios. Summary of the invention
[0003] The present invention provides a method for generating a virtual image video, a method for training a target network model, and a device, equipment, medium, and product.
[0004] According to one aspect of the present disclosure, a method for generating a virtual image video is provided, comprising: adjusting a pronunciation feature of at least one character phoneme in a second target audio according to a melody feature of a first target audio to obtain an adjusted second target audio; determining facial drive parameters for a virtual image model according to the adjusted second target audio; and applying the facial drive parameters to the virtual image model to obtain a virtual image video based on the adjusted second target audio, wherein the second target audio is a reference audio obtained based on a target text.
[0005] According to another aspect of the present disclosure, a device for generating a virtual image video is provided, comprising: a first processing module, used to adjust the pronunciation characteristics of at least one character phoneme in a second target audio according to the melody characteristics of the first target audio, so as to obtain an adjusted second target audio; a second processing module, used to determine facial driving parameters for a virtual image model according to the adjusted second target audio; and a third processing module, used to apply the facial driving parameters to the virtual image model, so as to obtain a virtual image video based on the adjusted second target audio, wherein the second target audio is a reference audio obtained based on a target text.
[0006] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory in communication with the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating a virtual image video.
[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned method for generating a virtual image video.
[0008] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above-mentioned method for generating a virtual image video when executed by a processor.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0011] Figure 1 The system architecture of a method and device for generating a virtual image video according to an embodiment of the present disclosure is schematically shown;
[0012] Figure 2 A flowchart of a method for generating a virtual image video according to an embodiment of the present disclosure is schematically shown;
[0013] Figure 3 A flowchart of a method for generating a virtual image video according to another embodiment of the present disclosure is schematically shown;
[0014] Figure 4 A schematic diagram schematically shows a process of generating a virtual image video according to an embodiment of the present disclosure;
[0015] Figure 5 A block diagram schematically shows a device for generating a virtual image video according to an embodiment of the present disclosure;
[0016] Figure 6 A block diagram of an electronic device for generating a virtual image video according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.
[0018] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0019] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0020] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).
[0021] The embodiment of the present disclosure provides a method for generating a virtual image video. The method of this embodiment includes: adjusting the pronunciation characteristics of at least one character phoneme in the second target audio according to the melody characteristics of the first target audio to obtain the adjusted second target audio, determining the facial drive parameters for the virtual image model according to the adjusted second target audio, and applying the facial drive parameters to the virtual image model to obtain a virtual image video based on the adjusted second target audio, wherein the second target audio is a reference audio obtained based on the target text.
[0022] Figure 1 The system architecture of the method and device for generating a virtual image video according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0023] The system architecture 100 according to this embodiment may include a request terminal 101, a network 102, and a server 103. The network 102 is used to provide a medium for a communication link between the request terminal 101 and the server 103. The network 102 may include various connection types, such as wired or wireless communication links or optical fiber cables, etc. The server 103 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud computing, network services, and middleware services.
[0024] The request terminal 101 communicates with the server 103 via the network 102 to receive or send data, etc. The request terminal 101 is used, for example, to initiate a request to generate a virtual image video to the server 103, and the request terminal 101 is also used to send a first target audio or target text for generating a virtual image video to the server 103.
[0025] The server 103 may be a server that provides various services, for example, may be a background processing server that performs interactive processing according to a generation request initiated by the request terminal 101 (only as an example).
[0026] For example, in response to a request for generating a virtual image video from the requesting terminal 101, the server 103 adjusts the pronunciation features of at least one character phoneme in the second target audio according to the melodic features of the first target audio to obtain an adjusted second target audio, determines facial drive parameters for the virtual image model based on the adjusted second target audio, and applies the facial drive parameters to the virtual image model to obtain a virtual image video based on the adjusted second target audio, where the second target audio is a baseline audio obtained based on the target text.
[0027] It should be noted that the method for generating a virtual image video provided in the embodiment of the present disclosure can be executed by the server 103. Accordingly, the device for generating a virtual image video provided in the embodiment of the present disclosure can be set in the server 103. The method for generating a virtual image video provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 103 and can communicate with the request terminal 101 and / or the server 103. Accordingly, the device for generating a virtual image video provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 103 and can communicate with the request terminal 101 and / or the server 103.
[0028] It should be understood that Figure 1 The number of request terminals, networks and servers in the embodiment is only for illustration. Any number of request terminals, networks and servers may be provided according to implementation requirements.
[0029] The present disclosure provides a method for generating a virtual image video. Figure 1 The system architecture of Figure 2 to Figure 4 The method for generating a virtual image video according to an exemplary embodiment of the present disclosure is described below. The method for generating a virtual image video according to an exemplary embodiment of the present disclosure can be, for example, Figure 1 The server 103 shown is used for execution.
[0030] Figure 2 The flowchart of a method for generating a virtual image video according to an embodiment of the present disclosure is schematically shown.
[0031] like Figure 2 As shown, the method 200 for generating a virtual image video according to an embodiment of the present disclosure may include, for example, operations S210 to S230.
[0032] In operation S210, a pronunciation feature of at least one character phoneme in the second target audio is adjusted according to the melody feature of the first target audio to obtain an adjusted second target audio.
[0033] In operation S220, a facial driving parameter for the avatar model is determined according to the adjusted second target audio.
[0034] In operation S230, the facial driving parameters are applied to the avatar model to obtain an avatar video based on the adjusted second target audio, where the second target audio is a reference audio obtained based on the target text.
[0035] The following is an example of the operation process of the method for generating a virtual avatar video of this embodiment.
[0036] Exemplarily, in response to the request to generate the virtual avatar video, the pronunciation feature of at least one character phoneme in the second target audio can be adjusted according to the melody feature of the first target audio to obtain the adjusted second target audio. In an exemplary manner, the first target audio can be determined according to the request to generate the virtual avatar video, or the target text for generating the second target audio can be determined.
[0037] For example, according to the request for generating the avatar video, a target music audio can be selected from a preset music library as the first target audio. Alternatively, the first target audio can be determined according to the music audio segment specified by the generation request. In addition, according to the text segment specified by the generation request, a target text for generating the second target audio can be determined.
[0038] The first target audio may be, for example, a target music audio, which has corresponding melody features, which may include, for example, melody time domain features and melody spectrum features. The melody time domain features may, for example, indicate the pronunciation timestamp of at least one melody phoneme in the first target audio. The melody phoneme may, for example, be a note in the first target audio, which may include, for example, a perfect note, a half note, a quarter note, etc. The melody spectrum features may, for example, indicate the pitch and volume features of the first target audio.
[0039] The second target audio may be a reference audio obtained based on a target text. The target text may be, for example, a target lyrics text, and the target text includes at least one character phoneme, which may be the smallest pronunciation unit in the target text. The character phoneme may be the smallest pronunciation unit based on multiple languages, such as Chinese characters based on Chinese, or may be English words.
[0040] The pronunciation features of at least one character phoneme may include, for example, pronunciation time domain features and pronunciation spectrum features. The pronunciation time domain features may indicate, for example, the pronunciation timestamp of the corresponding character phoneme, and the pronunciation spectrum features may indicate, for example, the pronunciation pitch and pronunciation volume features of the corresponding character phoneme.
[0041] The pronunciation characteristics of at least one character phoneme in the second target audio may be adjusted according to the melodic characteristics of the first target audio to obtain the adjusted second target audio. In an example, the pronunciation time domain characteristics and pronunciation spectrum characteristics of at least one character phoneme in the second target audio may be adjusted according to the melodic characteristics of the first target audio to obtain the adjusted second target audio.
[0042] According to the adjusted second target audio, the facial drive parameters for the avatar model are determined. The facial drive parameters are applied to the avatar model to obtain a avatar video based on the adjusted second target audio. Exemplarily, the limb drive parameters for the avatar model can also be determined according to the adjusted second target audio to achieve the avatar video by singing the adjusted second target audio through the avatar model.
[0043] Through the disclosed embodiments, the pronunciation features of at least one character phoneme in the second target audio are adjusted according to the melody features of the first target audio to obtain the adjusted second target audio, and the facial drive parameters for the virtual image model are determined according to the adjusted second target audio, and based on the facial drive parameters, the virtual image model is controlled to play the adjusted second target audio to obtain the virtual image video. The naturalness and intelligence of the virtual image video can be effectively improved, which is conducive to meeting the diverse interaction needs of users. The second target audio is a reference audio obtained based on the target text, and driving the virtual image model to play the adjusted second target audio can effectively solve the application scenario limitations of the virtual image video generation. Determining the facial drive parameters for the virtual image model can effectively reduce the hardware equipment requirements for the virtual image video generation, which is conducive to reducing the cost consumption of the virtual image video generation.
[0044] Figure 3 The flowchart of a method for generating a virtual image video according to another embodiment of the present disclosure is schematically shown.
[0045] like Figure 3As shown, the method 300 for generating a virtual avatar video according to an embodiment of the present disclosure may include, for example, operation S310 and operations S220 to S230.
[0046] In operation S310, according to the melody feature of the first target audio, the pronunciation time domain feature and the pronunciation spectrum feature of at least one character phoneme in the second target audio are adjusted to obtain the adjusted second target audio.
[0047] In operation S220, a facial driving parameter for the avatar model is determined according to the adjusted second target audio.
[0048] In operation S230, the facial driving parameters are applied to the avatar model to obtain an avatar video based on the adjusted second target audio, where the second target audio is a reference audio obtained based on the target text.
[0049] The following is an example flow of each operation of the method for generating a virtual avatar video of this embodiment.
[0050] Exemplarily, the melody features of the first target audio may include melody time domain features and melody spectrum features. The melody time domain features may, for example, include pronunciation time domain features of at least one melody phoneme in the first target audio, and the pronunciation time domain features may, for example, indicate the pronunciation timestamp of the corresponding melody phoneme. The melody spectrum features may, for example, indicate the pitch and volume features of the corresponding melody phoneme. For example, the melody features of the first target audio may be determined based on the MFCC (Mel-scale Frequency Cepstral Coefficients) information of the first target audio. MFCC information may be more consistent with the auditory characteristics of the human ear, have better robustness, and may have better recognition performance when the signal-to-noise ratio of the audio data is reduced.
[0051] In an exemplary manner, a target melody contour of a first target audio may be determined, and the target melody contour may be subjected to phoneme encoding processing to obtain at least one melody phoneme in the first target audio. For example, the target melody contour may be subjected to phoneme encoding processing using an encoder, and the encoder may include, for example, a convolutional layer, a downsampling layer, and a recursive layer based on a gated recurrent unit. The melody features of the first target audio may be extracted using the convolutional layer, and the melody features may be subjected to sampling processing using the downsampling layer to obtain melody sampling features. The melody sampling features may be subjected to regularization processing using the recursive layer to obtain a melody feature matrix. The melody feature matrix may be converted into phonemes to obtain at least one melody phoneme in the first target audio.
[0052] According to the melody feature of the first target audio, the pronunciation time domain feature and the pronunciation spectrum feature of at least one character phoneme are adjusted to obtain the adjusted second target audio. In an example, the pronunciation time domain feature of at least one character phoneme can be adjusted according to the melody time domain feature of the first target audio. And, according to the melody spectrum feature of the first target audio, the pronunciation spectrum feature of at least one character phoneme after the pronunciation time domain feature is adjusted is adjusted to obtain the adjusted second target audio.
[0053] By adjusting the pronunciation time domain features and pronunciation spectrum features of at least one character phoneme, an adjusted second target audio is obtained, which can effectively ensure the naturalness of virtual image video generation, help improve the generalization and universality of virtual image video generation, and help meet users' diverse digital human application needs.
[0054] The character phoneme is the smallest pronunciation unit in the target text, for example, it can be a Chinese character in a Chinese lyrics text, or it can be a word in an English lyrics text. In an exemplary manner, the target text can be segmented based on the smallest pronunciation unit to obtain a segmentation result. According to the segmentation result, the pronunciation timestamp of each character phoneme in at least one character phoneme is determined as a pronunciation time domain feature.
[0055] Segmenting the target text based on the smallest pronunciation unit and determining the pronunciation timestamp of at least one character phoneme according to the segmentation result can effectively generate virtual image videos based on target texts in different languages, which is conducive to fully considering the differences in pronunciation logic of different languages and can effectively solve the application scenario limitations of virtual image video generation.
[0056] The pronunciation time domain feature indicates the pronunciation timestamp of the corresponding phoneme. Exemplarily, the pronunciation timestamp of at least one character phoneme in the second target audio can be adjusted according to the pronunciation timestamp of the melody phoneme in the first target audio. According to the melody spectrum feature of the first target audio, the pronunciation pitch and frequency of at least one character phoneme after the pronunciation timestamp is adjusted are adjusted to obtain the adjusted second target audio.
[0057] Exemplarily, a phase vocoder can be used to change the phase information of at least one character phoneme to compress or stretch the pronunciation time window of at least one character phoneme in the time domain. For example, the pronunciation timestamp of the vowel of at least one character phoneme can be adjusted to obtain the second target audio with the pronunciation timestamp adjusted.
[0058] According to the adjusted second target audio, the facial drive parameters for the avatar model are determined. In an exemplary manner, the spatial domain features and / or time domain features are extracted from the adjusted second target audio to obtain initial audio features. Context features are extracted from the initial audio features to obtain autocorrelation audio features, and the facial drive parameters for the avatar model are determined based on the autocorrelation audio features.
[0059] For example, the adjusted second target audio can be converted into a spectrum feature map, and the spectrum feature map can be extracted in the time domain using a fully convolutional neural network to obtain initial audio features. LSTM (Long Short Term Memory) can be used to extract context features from the initial audio features to obtain autocorrelation audio features.
[0060] By extracting rich audio information from the adjusted second target audio, the generalization effect for a variety of timbres can be effectively improved, as well as the naturalness and intelligence of the virtual image video, which is conducive to meeting the diverse interaction needs of users.
[0061] Exemplarily, the adjusted second target audio may be time-framed. Feature extraction may be performed on the adjusted second target audio by frame to obtain autocorrelation audio features associated with each frame of audio. According to the autocorrelation audio features associated with each frame of audio, facial drive parameters for the avatar model are determined. The autocorrelation audio features may, for example, include at least one of the amplitude, frequency, and waveform features of the adjusted second target audio.
[0062] For example, the trained target network model can be used to output facial drive parameters for the virtual image model according to the adjusted autocorrelation audio features of the second target audio. The target network model can be, for example, trained using the autocorrelation audio features and character encoding of the sample audio as training samples, and the facial drive parameters for the virtual image model as label data. The character encoding can include, for example, word encoding or word encoding, and the autocorrelation audio features and character encoding can be fused, and model training can be performed based on the fused features to obtain a trained target network model. The target network model can be implemented, for example, using a convolutional neural network (CNN), which is not limited in this embodiment.
[0063] The facial drive parameters may include a pose parameter of at least one facial feature point. In an example manner, the pose parameter of at least one facial feature point is weighted to obtain a lip-type drive parameter for the avatar model. The lip-type drive parameter is applied to the avatar model so that the adjusted second target audio is played through the avatar model to obtain the avatar video.
[0064] The mouth shape driving parameters can be characterized by, for example, the pose parameters of the lip bone feature points. The mouth shape driving parameters can indicate the states of the upper and lower lips and the upper and lower jaws during the pronunciation of phonemes that match the adjusted second target audio. For example, they can indicate information such as the longitudinal change value of the lips, the lateral change value of the lips, and the change value of the opening and closing degree between the upper and lower jaws and the teeth.
[0065] By determining the mouth shape driving parameters of the virtual avatar model, the dependence on hardware such as image acquisition devices and facial mouth shape capture devices can be effectively reduced, the cost consumption for generating virtual avatar videos can be effectively reduced, which is conducive to realizing a digital human driving solution with low cost and strong generalization effect.
[0066] Adjusting the pronunciation time-domain features and pronunciation spectrum features of at least one character phoneme in the second target audio, and generating a virtual avatar video based on the adjusted second target audio can effectively ensure the strong generalization effect of virtual avatar video generation, which is conducive to improving the naturalness and intelligence of virtual avatar video generation, and is conducive to meeting the diverse interaction needs of users. Determining the facial driving parameters for the virtual avatar model according to the adjusted second target audio can effectively reduce the dependence on hardware such as image acquisition devices, which is conducive to reducing the cost consumption of virtual avatar video generation, and is conducive to solving the application scenario limitations of digital human driving.
[0067] Figure 4 A schematic diagram schematically shows the generation process of a virtual avatar video according to an embodiment of the present disclosure.
[0068] As Figure 4 shown, exemplarily, the first target audio 401 includes multiple melody phonemes. For example, it can include melody phonemes 401A, 401B, 401C, 401D, and 401E. Taking the target text "Hello" as an example, the second target audio 402 based on the target text can include character phonemes "你 (nǐ)", "好 (hǎo)", and "呀 (ya)".
[0069] According to the melody characteristics of the first target audio 401, the pronunciation characteristics of at least one character phoneme in the second target audio 402 are adjusted. In one example way, according to the melody time-domain characteristics of the first target audio 401, the pronunciation time-domain characteristics of at least one character phoneme in the second target audio 402 are adjusted to obtain the second target audio 403 with adjusted pronunciation time-domain characteristics. The melody time-domain characteristics can indicate, for example, the pronunciation timestamps of the melody phonemes in the first target audio 401, and the pronunciation time-domain characteristics of at least one character phoneme can indicate the pronunciation timestamps of the corresponding character phonemes.
[0070] According to the melody spectrum feature of the first target audio 401, the pronunciation spectrum feature of at least one character phoneme in the second target audio 403 after the pronunciation time domain feature is adjusted is adjusted to obtain the adjusted second target audio 404. The melody spectrum feature may indicate, for example, the pitch and volume features of the first target audio.
[0071] According to the adjusted second target audio 404, the lip-driven parameters for the avatar model 405 are determined. The lip-driven parameters are applied to the avatar model 405, so that the adjusted second target audio 404 is played through the avatar model 405 to obtain the avatar video.
[0072] It can effectively ensure the strong generalization effect of virtual image video generation, which is conducive to improving the naturalness and intelligence of virtual image video generation and meeting the diverse interaction needs of users.
[0073] Figure 5 A block diagram schematically shows a device for generating a virtual image video according to an embodiment of the present disclosure.
[0074] like Figure 5 As shown, the device 500 for generating a virtual image video according to an embodiment of the present disclosure includes, for example, a first processing module 510 , a second processing module 520 and a third processing module 530 .
[0075] The first processing module 510 is used to adjust the pronunciation characteristics of at least one character phoneme in the second target audio according to the melodic characteristics of the first target audio to obtain the adjusted second target audio; the second processing module 520 is used to determine the facial driving parameters for the virtual image model according to the adjusted second target audio; and the third processing module 530 is used to apply the facial driving parameters to the virtual image model to obtain a virtual image video based on the adjusted second target audio, where the second target audio is a reference audio obtained based on the target text.
[0076] Through the disclosed embodiments, the pronunciation features of at least one character phoneme in the second target audio are adjusted according to the melody features of the first target audio to obtain the adjusted second target audio, and the facial drive parameters for the virtual image model are determined according to the adjusted second target audio, and based on the facial drive parameters, the virtual image model is controlled to play the adjusted second target audio to obtain the virtual image video. The naturalness and intelligence of the virtual image video can be effectively improved, which is conducive to meeting the diverse interaction needs of users. The second target audio is a reference audio obtained based on the target text, and driving the virtual image model to play the adjusted second target audio can effectively solve the application scenario limitations of the virtual image video generation. Determining the facial drive parameters for the virtual image model can effectively reduce the hardware equipment requirements for the virtual image video generation, which is conducive to reducing the cost consumption of the virtual image video generation.
[0077] According to an embodiment of the present disclosure, the first processing module includes: a first processing sub-module, which is used to adjust the pronunciation time domain characteristics and pronunciation spectrum characteristics of at least one character phoneme according to the melody characteristics of the first target audio to obtain an adjusted second target audio.
[0078] According to an embodiment of the present disclosure, the melody feature includes a melody time domain feature and a melody spectrum feature, and the melody time domain feature indicates the pronunciation time domain feature of the melody phoneme in the first target audio; the first processing submodule includes: a first processing unit, used to adjust the pronunciation time domain feature of at least one character phoneme according to the melody time domain feature of the first target audio; and a second processing unit, used to adjust the pronunciation spectrum feature of at least one character phoneme after the pronunciation time domain feature is adjusted according to the melody spectrum feature of the first target audio, to obtain the adjusted second target audio.
[0079] According to an embodiment of the present disclosure, the pronunciation time domain feature indicates the pronunciation timestamp of the corresponding phoneme; the first processing unit includes: a first processing subunit, which is used to adjust the pronunciation timestamp of at least one character phoneme according to the pronunciation timestamp of the melody phoneme in the first target audio.
[0080] According to an embodiment of the present disclosure, a character phoneme is the smallest pronunciation unit in a target text; the device also includes a fourth processing module, which is used to: perform word segmentation processing on the target text based on the smallest pronunciation unit to obtain a word segmentation result; and determine, based on the word segmentation result, a pronunciation timestamp of each character phoneme in at least one character phoneme as a pronunciation time domain feature.
[0081] According to an embodiment of the present disclosure, the second processing module includes: a second processing sub-module, used to extract spatial domain features and / or time domain features of the adjusted second target audio to obtain initial audio features; a third processing sub-module, used to extract contextual features of the initial audio features to obtain autocorrelation audio features; and a fourth processing sub-module, used to determine facial drive parameters for the virtual image model based on the autocorrelation audio features.
[0082] According to an embodiment of the present disclosure, the facial drive parameters include posture parameters of at least one facial feature point; the third processing module includes: a fifth processing sub-module, used to weight the posture parameters of at least one facial feature point to obtain lip-type drive parameters for the virtual image model; and a sixth processing sub-module, used to apply the lip-type drive parameters to the virtual image model so as to play the adjusted second target audio through the virtual image model to obtain the virtual image video.
[0083] It should be noted that the information collection, storage, use, processing, transmission, provision and disclosure involved in the technical solution of the present disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0084] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0085] Figure 6 A block diagram of an electronic device for generating a virtual image video according to an embodiment of the present disclosure is schematically shown.
[0086] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device 600 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0087] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0088] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0089] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running deep learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the method for generating a virtual image video. For example, in some embodiments, the method for generating a virtual image video may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the method for generating a virtual image video described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the method for generating a virtual image video in any other appropriate manner (eg, by means of firmware).
[0090] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0091] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable model training device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.
[0092] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0093] To provide interaction with an object, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the object; and a keyboard and pointing device (e.g., a mouse or trackball) through which the object can provide input to the computer. Other types of devices can also be used to provide interaction with an object; for example, the feedback provided to the object can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the object can be received in any form (including acoustic input, voice input, or tactile input).
[0094] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., an object computer having a graphical object interface or a web browser through which the object can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0095] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating a virtual image video, include: According to the melody feature of the first target audio, adjusting the pronunciation feature of at least one character phoneme in the second target audio to obtain an adjusted second target audio; Determining facial drive parameters for the avatar model according to the adjusted second target audio; as well as applying the facial drive parameters to the avatar model to obtain an avatar video based on the adjusted second target audio, Wherein, the second target audio is a reference audio obtained based on the target text; The melody feature includes a melody time domain feature and a melody spectrum feature, and the melody time domain feature indicates a pronunciation time domain feature of a melody phoneme in the first target audio; The step of adjusting the pronunciation feature of at least one character phoneme in the second target audio according to the melody feature of the first target audio to obtain the adjusted second target audio includes: According to the melody time-domain feature of the first target audio, adjusting the pronunciation time-domain feature of the at least one character phoneme; and According to the melody spectrum feature of the first target audio, the pronunciation spectrum feature of at least one character phoneme after the pronunciation time domain feature is adjusted to obtain the adjusted second target audio.
2. The method according to claim 1, in, The pronunciation time domain feature indicates the pronunciation timestamp of the corresponding phoneme; The step of adjusting the pronunciation time domain feature of the at least one character phoneme according to the melody time domain feature of the first target audio comprises: The pronunciation timestamp of the at least one character phoneme is adjusted according to the pronunciation timestamp of the melody phoneme in the first target audio.
3. The method according to claim 1, in, The character phoneme is the smallest pronunciation unit in the target text; the method further comprises: Performing word segmentation processing on the target text based on the minimum pronunciation unit to obtain a word segmentation result; and According to the word segmentation result, a pronunciation timestamp of each character phoneme in the at least one character phoneme is determined as the pronunciation time domain feature.
4. The method according to claim 1, in, The step of determining facial drive parameters for the avatar model according to the adjusted second target audio comprises: Extracting spatial domain features and / or time domain features from the adjusted second target audio to obtain initial audio features; Performing context feature extraction on the initial audio features to obtain autocorrelation audio features; and Based on the autocorrelation audio features, facial drive parameters for the avatar model are determined.
5. The method according to any one of claims 1 to 4, in, The facial driving parameters include a pose parameter of at least one facial feature point; The step of applying the facial drive parameters to the avatar model to obtain an avatar video based on the adjusted second target audio comprises: Weighting the position and posture parameters of the at least one facial feature point to obtain lip-driven parameters for the avatar model; as well as The lip-driven parameters are applied to the avatar model so as to play the adjusted second target audio through the avatar model to obtain the avatar video.
6. A device for generating a virtual image video, include: A first processing module, configured to adjust a pronunciation feature of at least one character phoneme in a second target audio according to a melody feature of the first target audio, to obtain an adjusted second target audio; A second processing module, configured to determine a facial driving parameter for the avatar model according to the adjusted second target audio; as well as A third processing module is configured to apply the facial driving parameters to the avatar model to obtain an avatar video based on the adjusted second target audio. The second target audio is a reference audio obtained based on the target text; the melody feature includes a melody time domain feature and a melody spectrum feature, and the melody time domain feature indicates the pronunciation time domain feature of the melody phoneme in the first target audio; the first processing module includes a first processing submodule, including: A first processing unit, configured to adjust the pronunciation time-domain feature of the at least one character phoneme according to the melody time-domain feature of the first target audio; and The second processing unit is used to adjust the pronunciation spectrum feature of at least one character phoneme after the pronunciation time domain feature is adjusted according to the melody spectrum feature of the first target audio, so as to obtain the adjusted second target audio.
7. The device according to claim 6, in, The pronunciation time domain feature indicates the pronunciation timestamp of the corresponding phoneme; the first processing unit includes: The first processing subunit is used to adjust the pronunciation timestamp of the at least one character phoneme according to the pronunciation timestamp of the melody phoneme in the first target audio.
8. The device according to claim 6, in, The character phoneme is the smallest pronunciation unit in the target text; the device also includes a fourth processing module, which is used to: Performing word segmentation processing on the target text based on the minimum pronunciation unit to obtain a word segmentation result; as well as According to the word segmentation result, a pronunciation timestamp of each character phoneme in the at least one character phoneme is determined as the pronunciation time domain feature.
9. The device according to claim 6, in, The second processing module comprises: A second processing submodule is used to extract spatial domain features and / or time domain features of the adjusted second target audio to obtain initial audio features; A third processing submodule is used to extract context features from the initial audio features to obtain autocorrelation audio features; and The fourth processing submodule is used to determine facial driving parameters for the virtual image model based on the autocorrelation audio features.
10. The device according to any one of claims 6 to 9, in, The facial driving parameters include the posture parameters of at least one facial feature point; the third processing module includes: a fifth processing submodule, configured to weight the position and posture parameters of the at least one facial feature point to obtain lip-driven parameters for the avatar model; and The sixth processing submodule is used to apply the lip-driven parameters to the virtual image model so as to play the adjusted second target audio through the virtual image model to obtain the virtual image video.
11. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating a virtual image video according to any one of claims 1 to 5.
12. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method for generating a virtual avatar video according to any one of claims 1 to 5.
13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method for generating a virtual image video according to any one of claims 1 to 5.
Citation Information
Patent Citations
Facial animation generation method, device and equipment and readable storage medium
CN109523616A
Virtual object display method and device, equipment and medium
CN112541959A
Virtual image generation method and device, storage medium and electronic equipment
CN112650831A
Animation synthesis method and device, electronic equipment and storage medium
CN113744370A
Phoneme detection method and device, training method and device, equipment and medium
CN114267376A
