Virtual avatar lip shape driving method and device, medium and electronic device

Abstract

This disclosure relates to a method, apparatus, medium, and electronic device for driving the lip movements of a virtual avatar. The method includes: determining a target IPA sequence corresponding to a target text and the pronunciation duration of each phoneme in the sequence; extending the target IPA sequence based on the pronunciation duration to obtain an extended sequence; for each phoneme in the extended sequence, extracting an N-gram phoneme string containing that phoneme from the extended sequence to obtain a target phoneme sequence; determining lip shape parameters matching the target phoneme sequence based on the correspondence between a reference phoneme sequence and lip shape parameters; and rendering the lip movements of the target virtual avatar based on the lip shape parameters corresponding to the phoneme. By considering both the target phoneme itself and its contextual information when obtaining the lip shape parameters for each target phoneme in the extended sequence, the lip shape parameters under different IPA combinations can better match real facial features, making the lip movements of the target virtual avatar as realistic as possible and improving the animation effect.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision technology, and more specifically, to a method, apparatus, medium, and electronic device for driving the lip movements of a virtual character. Background Technology

[0002] With the rapid development of artificial intelligence (AI) and big data technologies, AI has permeated all aspects of life. Virtual object technology is an important subfield of AI technology. It can use AI technology to construct a virtual image and drive the facial expressions of this virtual image to simulate human speech.

[0003] Facial expression-driven applications include using text input to drive the lip movements of virtual avatars. Among these, making the virtual avatar's lips as close to those of a real person as possible is crucial for improving the animation effect. Summary of the Invention

[0004] This section is provided to briefly introduce the concepts, which will be described in detail in the Detailed Description section later. This section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0005] In a first aspect, this disclosure provides a virtual avatar lip-driving method, including:

[0006] Determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence;

[0007] Based on the duration of each target pronunciation, the target IPA sequence is duration-extended to obtain an extended sequence;

[0008] For each target phonetic symbol in the extended sequence, an N-gram phonetic symbol string containing the target phonetic symbol is extracted from the extended sequence to obtain a target phonetic symbol sequence, where N≥2, and the target phonetic symbol is located at a preset position in the target phonetic symbol sequence; based on the pre-established correspondence between reference phonetic symbol sequences and lip shape parameters, lip shape parameters matching the target phonetic symbol sequence are determined, wherein the lip shape parameters corresponding to the reference phonetic symbol sequence are used to characterize the lip shape parameters corresponding to the phonetic symbol at the preset position in the reference phonetic symbol sequence; based on the lip shape parameters corresponding to the target phonetic symbol, lip shape rendering is performed on the target virtual image.

[0009] Secondly, this disclosure provides a virtual avatar lip-syncing device, comprising:

[0010] The first determining module is used to determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phonetic symbol in the target IPA sequence;

[0011] The first extension module is used to extend the duration of the target IPA sequence according to the duration of each target pronunciation to obtain an extended sequence;

[0012] The first extraction module extracts an N-gram phonetic string containing the target phonetic symbol from the extended sequence for each target phonetic symbol in the extended sequence to obtain a target phonetic symbol sequence, where N≥2, and the target phonetic symbol is located at a preset position in the target phonetic symbol sequence; the second determination module determines the lip shape parameters matching the target phonetic symbol sequence based on a pre-established correspondence between a reference phonetic symbol sequence and lip shape parameters, wherein the lip shape parameters corresponding to the reference phonetic symbol sequence are used to characterize the lip shape parameters corresponding to the phonetic symbol at the preset position in the reference phonetic symbol sequence; the rendering module renders the lip shape of the target virtual image based on the lip shape parameters corresponding to the target phonetic symbol.

[0013] Thirdly, this disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the virtual avatar lip-driving method provided in the first aspect of this disclosure.

[0014] Fourthly, this disclosure provides an electronic device, comprising:

[0015] A storage device having at least one computer program stored thereon;

[0016] At least one processing device is configured to execute the at least one computer program in the storage device to implement the steps of the virtual avatar lip-driving method provided in the first aspect of this disclosure.

[0017] In the above technical solution, after obtaining the extended sequence corresponding to the target text, when obtaining the corresponding lip shape parameters for each target phonetic symbol in the extended sequence, not only the target phonetic symbol itself is considered, but also its contextual information. In this way, the lip shape parameters under different IPA combinations can be more in line with the real facial situation, and good results can be obtained without deliberately adjusting the smoothing filter parameters. This makes the lip shape of the target virtual image as close to that of a real person as possible, thus improving the animation effect.

[0018] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0019] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale. In the drawings:

[0020] Figure 1 This is a flowchart illustrating a virtual avatar lip-syncing method according to an exemplary embodiment.

[0021] Figure 2 This is a schematic diagram illustrating a process of duration extension of a target IPA sequence according to an exemplary embodiment.

[0022] Figure 3 This is a flowchart illustrating a method for constructing a correspondence between a reference phonetic sequence and lip shape parameters according to an exemplary embodiment.

[0023] Figure 4 This is a schematic diagram illustrating a process for determining reference lip shape parameters from the first candidate parameter sequence according to an exemplary embodiment.

[0024] Figure 5 This is a flowchart illustrating a method for constructing a correspondence between a reference phonetic sequence and lip shape parameters according to another exemplary embodiment.

[0025] Figure 6 This is a block diagram illustrating a virtual avatar lip-syncing device according to an exemplary embodiment.

[0026] Figure 7 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0027] As discussed in the background section, making the lip movements of a virtual avatar as close to those of a real person as possible is crucial for improving animation quality. Currently, virtual avatar lip-motion driving schemes typically begin by obtaining the International Phonetic Alphabet (IPA) sequence corresponding to the input text and the pronunciation duration of each phoneme within the IPA sequence. Then, based on the pronunciation duration of each phoneme, the IPA sequence is duration-extended. Finally, for each phoneme in the duration-extended IPA sequence, the corresponding lip-motion parameters are obtained according to a pre-established correspondence between phonemes and lip-motion parameters. Based on these parameters, the virtual avatar's lip movements are rendered. To make the virtual avatar's lip movements smoother, filtering algorithms, such as mean filtering, are generally used to smooth the lip-motion parameters corresponding to each phoneme.

[0028] The lip-shape driving scheme described above only considers the parameter values ​​corresponding to each IPA, neglecting their contextual information. This leads to the use of the same set of parameters when the same IPA encounters different contexts. Furthermore, because the filtering used for curve smoothing is global, the lip-shape parameters for some IPA combinations do not conform to reality. For example, regarding the "b" in the IPA sequences "aobu" and "hubu," when it is preceded by "o," the mouth transitions from a wide-open state to a closed state. However, when it is preceded by "u," the mouth transitions from a slightly open state to a closed state. Since the lip-shape driving scheme described above only has one set of lip-shape parameters for "b," the lip-shape parameters for the two "b"s in these two different IPA sequences are the same. Therefore, in the subsequent smoothing stage, it becomes difficult to find a suitable filtering algorithm to make the transitions between these two states appear natural.

[0029] In view of this, the present disclosure provides a method, apparatus, medium and electronic device for driving the lip movements of a virtual image.

[0030] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0033] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0034] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0035] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0036] All actions involving the acquisition of signals, information, or data in this disclosure are carried out in accordance with the relevant data protection laws and policies of the country where the location is situated, and with the authorization granted by the owner of the relevant device.

[0037] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0038] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0039] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0040] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0041] Meanwhile, it is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0042] Figure 1 This is a flowchart illustrating a virtual avatar lip-syncing method according to an exemplary embodiment. Figure 1 As shown, the method may include the following steps S101 to S105.

[0043] In S101, the target IPA sequence corresponding to the target text and the target pronunciation duration of each phonetic symbol in the target IPA sequence are determined.

[0044] The purpose of this disclosure is to generate a target virtual avatar that speaks through lip-syncing, so that the target virtual avatar can perform its role functions. The target virtual avatar can be a cartoon character or a non-cartoon character, and its role can be customer service, host, idol, tour guide, etc., without specific limitations in this disclosure.

[0045] The target text can be any text content, including Chinese text, English text, etc. This publication does not specifically limit the language of the target text.

[0046] In S102, the target IPA sequence is extended in duration according to the pronunciation duration of each target to obtain an extended sequence.

[0047] In S103, for each target phonetic symbol in the extended sequence, an N-gram phonetic symbol string containing that target phonetic symbol is extracted from the extended sequence to obtain the target phonetic symbol sequence.

[0048] Here, N-gram is the Chinese language model, and N is a positive integer greater than or equal to 2.

[0049] An N-gram phonetic string containing the target phonetic symbol can be composed of the target phonetic symbol and the following N-1 consecutive phonetic symbols. In this case, the preset position is the first position of the corresponding phonetic symbol sequence. Alternatively, an N-gram phonetic string containing the target phonetic symbol can be composed of the preceding N-1 consecutive phonetic symbols of the target phonetic symbol and the target phonetic symbol. In this case, the preset position is the last position of the corresponding phonetic symbol sequence. Furthermore, an N-gram phonetic string containing the target phonetic symbol can be composed of the preceding L consecutive phonetic symbols of the target phonetic symbol, the target phonetic symbol, and the following D consecutive phonetic symbols of the target phonetic symbol. In this case, the preset position is the L+1th position of the corresponding phonetic symbol sequence, where L≥1, D≥1, L+D+1=N, and N≥3.

[0050] Each target phonetic symbol sequence has a length of N. The length of the N-gram phonetic symbol string corresponding to the target phonetic symbol at the beginning or end of the extended sequence may be less than N. To ensure that the length of the target phonetic symbol sequence corresponding to each target phonetic symbol is N, the N-gram phonetic symbol string with a length less than N can be extended using a sequence start symbol or sequence end symbol to obtain a target phonetic symbol sequence of length N.

[0051] Specifically, if the length of the N-gram phonetic symbol string containing the target phonetic symbol is less than N, and the target phonetic symbol is located at the beginning of the extended sequence, a sequence start symbol can be added to the beginning of the corresponding N-gram phonetic symbol string to obtain a target phonetic symbol sequence of length N; if the length of the N-gram phonetic symbol string containing the target phonetic symbol is less than N, and the target phonetic symbol is located at the end of the extended sequence, a sequence start symbol can be added to the end of the corresponding N-gram phonetic symbol string to obtain a target phonetic symbol sequence of length N.

[0052] For example, an N-gram phonetic string containing the target phonetic symbol consists of the target phonetic symbol and its adjacent phonetic symbols following it, i.e., N=2, such as... Figure 2 The extended sequence "t, a, a, shown" is shown in the figure. i,i,a,a,x,a,a,a, For the phonetic symbol "t" in the extended sequence, its corresponding target phonetic symbol sequence is "t, a". The target phonetic symbol sequence corresponding to the second phonetic symbol "a" in the extended sequence is "a, a". The target phonetic symbol sequence corresponding to the third phonetic symbol "a" in the extended sequence is "a, a". "Phonetic symbols in this extended sequence" The corresponding target phonetic sequence is " The target phonetic sequence for the fifth phonetic symbol "i" in this extended sequence is "i, i". The target phonetic sequence for the sixth phonetic symbol "i" in this extended sequence is "i, a". The target phonetic sequence for the seventh phonetic symbol "a" in this extended sequence is "a, a". The target phonetic sequence for the eighth phonetic symbol "a" in this extended sequence is "a, x". The target phonetic sequence for the phonetic symbol "x" in this extended sequence is "x, a". The target phonetic sequence for the tenth phonetic symbol "a" in this extended sequence is "a, a". The target phonetic sequence for the eleventh phonetic symbol "a" in this extended sequence is "a, a". The target phonetic sequence for the twelfth phonetic symbol "a" in this sequence is "a, a". The thirteenth phonetic symbol in this extended sequence The corresponding target phonetic symbol sequence is The fourteenth phonetic symbol in this extended sequence The corresponding target phonetic symbol sequence is The fifteenth phonetic symbol in this sequence The corresponding target phonetic sequence is " "[eos]" where "[eos]" is the sequence terminator.

[0053] In S104, the lip shape parameters that match the target phonetic sequence are determined based on the pre-established correspondence between the reference phonetic sequence and the lip shape parameters.

[0054] In this disclosure, the lip shape parameter corresponding to the reference phonetic symbol sequence is used to characterize the lip shape parameter corresponding to the phonetic symbol at a preset position in the reference phonetic symbol sequence. Thus, the lip shape parameter that matches the target phonetic symbol sequence is the lip shape parameter corresponding to the phonetic symbol at the preset position in the target phonetic symbol sequence, i.e., the lip shape parameter corresponding to the target phonetic symbol.

[0055] By referencing the correspondence between phonetic symbols and lip shape parameters, we can consider not only the phonetic symbols at preset positions but also the contextual information of those phonetic symbols. This allows us to better describe the relationship between phonetic symbols and lip shape parameters, making the lip shape parameters corresponding to the target phonetic symbols determined based on this correspondence more consistent with the actual situation.

[0056] Among them, the lip shape parameters can include parameters for characterizing the degree of mouth opening and parameters for characterizing the smile amplitude of the mouth. That is, the lip shape parameters matched with the target phonetic sequence include a set of parameters for characterizing lip features.

[0057] In S105, the lip shape of the target virtual image is rendered based on the lip shape parameters corresponding to the target phonetic symbol.

[0058] In the above technical solution, after obtaining the extended sequence corresponding to the target text, when obtaining the corresponding lip shape parameters for each target phonetic symbol in the extended sequence, not only the target phonetic symbol itself is considered, but also its contextual information. In this way, the lip shape parameters under different IPA combinations can be more in line with the real facial situation, and good results can be obtained without deliberately adjusting the smoothing filter parameters. This makes the lip shape of the target virtual image as close to that of a real person as possible, thus improving the animation effect.

[0059] The following is a detailed description of the specific implementation method for determining the phoneme sequence corresponding to the target text and the pronunciation duration of each phoneme in the phoneme sequence in S101 above. Specifically, it can be achieved through the following steps 1) to 4):

[0060] Step 1): Determine the phoneme sequence corresponding to the target text and the pronunciation duration of each phoneme in the phoneme sequence.

[0061] Specifically, based on the target text, a text-to-speech (TTS) front-end engine can be used to obtain the phoneme sequence corresponding to the target text and the pronunciation duration of each phoneme in the phoneme sequence.

[0062] Step 2): Based on the pre-established mapping relationship between the phonemes of the target text's language and IPA, map the phoneme sequence corresponding to the target text to the target IPA sequence.

[0063] Step 3): For each phoneme in the phoneme sequence corresponding to the target text, determine the target pronunciation duration of each IPA corresponding to the phoneme based on the number of IPAs corresponding to the phoneme and the pronunciation duration of the phoneme.

[0064] In this disclosure, the mapping relationship between phonemes and IPAs may be one-to-one or one-to-many, for example, one phoneme corresponds to two IPAs. Specifically, when a phoneme corresponds to one IPA, the pronunciation duration of that phoneme can be determined as the target pronunciation duration of the IPA corresponding to that phoneme; when a phoneme corresponds to multiple IPAs, the pronunciation duration of that phoneme can be evenly distributed among the multiple IPAs corresponding to that phoneme, i.e., the target pronunciation duration of each IPA corresponding to that phoneme = pronunciation duration of that phoneme / M, where M is the number of IPAs corresponding to that phoneme.

[0065] The following is a detailed description of the specific implementation method for duration extension of the target IPA sequence based on the duration of each target pronunciation in S102 above. Specifically, it can be achieved through the following steps ① and ②:

[0066] Step ①: For each phonetic symbol in the target IPA sequence, determine the number of video frames corresponding to that phonetic symbol based on the pronunciation duration of that phonetic symbol and the duration of a video frame.

[0067] The duration of a video frame is the length of time it occupies, i.e., the playback duration.

[0068] For example, if the pronunciation duration of a phoneme is 100ms and the duration of a video frame is 10ms, then the number of video frames corresponding to that phoneme is 10.

[0069] For example, if the duration of a phoneme's pronunciation is 103ms and the duration of a video frame is 10ms, then the number of video frames corresponding to that phoneme is: That is, the last frame that is less than 10ms is treated as one frame.

[0070] Step 2: Expand the phonetic symbol in the target IPA sequence into K consecutive phonetic symbols, where K is the number of video frames corresponding to the phonetic symbol.

[0071] For example, such as Figure 2As shown, the target text is "Hello everyone". Through the TTS FrontEnd Engine, the phoneme sequence corresponding to the target text "Hello everyone" is "d, a, j, ia, h, ao". Among them, the pronunciation duration of the phoneme "d" is 60 ms, the pronunciation duration of the phoneme "a" is 120 ms, the pronunciation duration of the phoneme "j" is 60 ms, the pronunciation duration of the phoneme "ia" is 240 ms, the pronunciation duration of the phoneme "h" is 60 ms, and the pronunciation duration of the phoneme "ao" is 360 ms; in the pre-established mapping relationship between Chinese phonemes and IPA, the IPA corresponding to the phoneme "d" is "t", the IPA corresponding to the phoneme "a" is "a", and the IPA corresponding to the phoneme "j" is The IPA corresponding to the phoneme "ia" is "i" and "a", the IPA corresponding to the phoneme "h" is "x", and the IPA corresponding to the phoneme "ao" is "a" and Therefore, the phoneme sequence "d, a, j, ia, h, ao" can be mapped to the target IPA sequence "t, a, i, a, x, a, "; According to the pronunciation duration of each phoneme in the phoneme sequence "d, a, j, ia, h, ao", the target pronunciation durations of each phonetic symbol in the target IPA sequence "t, a, i, a, x, a, " are 60 ms, 120 ms, 60 ms, 120 ms, 120 ms, 60 ms, 180 ms, and 180 ms in sequence; the time length of a video frame is 60 ms, then the number of video frames corresponding to each phonetic symbol in the target IPA sequence "t, a, i, a, x, a, " are 1, 2, 1, 2, 2, 1, 3, and 3 in sequence. After that, according to the number of video frames corresponding to each phonetic symbol in the target IPA sequence "t, a, i, a, x, a, ", the target IPA sequence "t, a, i, a, x, a, " is extended in duration to obtain the extended sequence "t, a, a, i, i, a, a, x, a, a, a, ".

[0072] Next, the specific implementation method of lip rendering for the target virtual image based on the lip parameters corresponding to the target phonetic symbol in S105 above will be described in detail. Specifically, it can be implemented through various implementation methods. In one implementation method, the lip parameters corresponding to the target phonetic symbol can be directly input into a preset rendering engine (Unreal Engine) to perform lip rendering on the target virtual image (as Figure 2 shown).

[0073] In another implementation, to improve the smoothness of the lip movements of the target virtual avatar and thus enhance the animation effect, after determining the lip shape parameters corresponding to the target phonetic symbol, these parameters can be filtered first. Then, based on the filtered lip shape parameters corresponding to the target phonetic symbol, the target virtual avatar is rendered. Specifically, the lip shape parameters corresponding to the target phonetic symbol can be filtered (e.g., mean filtering) based on the lip shape parameters corresponding to neighboring phonetic symbols. Then, the filtered lip shape parameters corresponding to the target phonetic symbol are input into a preset rendering engine to render the lip shape of the target virtual avatar.

[0074] In this disclosure, a neighboring phonetic symbol is a phonetic symbol in the extended sequence whose positional distance from the target phonetic symbol is less than a preset distance threshold. For example, the two phonetic symbols before and the two phonetic symbols after the target phonetic symbol in the extended sequence, i.e., the preset distance threshold is 2, wherein the positional distance between the adjacent phonetic symbols of the target phonetic symbol and the target phonetic symbol is 1.

[0075] To more clearly describe the specific implementation of S104 above, before describing the specific implementation method of S104, the specific construction method of the correspondence between the reference phonetic sequence and the lip shape parameters will be described in detail. Specifically, it can be achieved through... Figure 3 The S201 to S207 shown are used to construct the correspondence between the reference phonetic sequence and the lip shape parameters.

[0076] In S201, multiple facial videos are acquired.

[0077] In this disclosure, the facial video includes audio and facial lip shape image information.

[0078] In S202, for each facial video, lip shape parameters are extracted from each video frame of that facial video.

[0079] In S203, the sample IPA sequence corresponding to the audio in the facial video and the sample pronunciation duration of each sample phonetic symbol in the sample IPA sequence are determined.

[0080] Specifically, speech recognition can be performed on the audio in the facial video to obtain the reference text corresponding to the audio. Then, the IPA sequence corresponding to the reference text and the pronunciation duration of each phonetic symbol in the IPA sequence are determined, that is, the sample IPA sequence corresponding to the audio in the facial video and the sample pronunciation duration of each sample phonetic symbol in the sample IPA sequence are obtained.

[0081] The IPA sequence corresponding to the reference text and the pronunciation duration of each phoneme in the IPA sequence can be determined in a manner similar to that used in S101 above to determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence. This disclosure will not elaborate further.

[0082] In S204, based on the sample IPA sequence and the pronunciation duration of each sample, the video segment corresponding to each sample phonetic symbol is determined from the facial video.

[0083] Specifically, we can first determine the number of video frames corresponding to each sample phonetic symbol in the sample IPA sequence based on the sample pronunciation duration of the sample phonetic symbol and the duration of a video frame in the facial video; then, based on the order of each sample phonetic symbol in the sample IPA sequence and the number of video frames corresponding to each sample phonetic symbol, we can determine the video frames corresponding to each sample phonetic symbol from the facial video corresponding to the sample IPA sequence, that is, the video segments corresponding to each sample phonetic symbol.

[0084] For example, the sample phonetic symbol sequence includes 3 sample phonetic symbols. The number of video frames corresponding to the first sample phonetic symbol is 3, the number of video frames corresponding to the second sample phonetic symbol is 5, and the number of video frames corresponding to the third sample phonetic symbol is 1. Then, the video segment corresponding to the first sample phonetic symbol is the 1st to 3rd video frames in the facial video corresponding to the sample phonetic symbol sequence, the video segment corresponding to the second sample phonetic symbol is the 4th to 8th video frames in the facial video, and the video segment corresponding to the third sample phonetic symbol is the 9th video frame in the facial video.

[0085] In S205, for each sample phonetic symbol, an N-gram phonetic symbol string containing that sample phonetic symbol is extracted from the sample IPA sequence to obtain a reference phonetic symbol sequence.

[0086] In this disclosure, an N-gram phonetic string containing the sample phonetic symbol can be extracted from the sample IPA sequence using a rule similar to S103 described above. Specifically, in S103, where the N-gram phonetic string containing the target phonetic symbol consists of the target phonetic symbol and the following N-1 consecutive phonetic symbols of the target phonetic symbol, the N-gram phonetic string containing the sample phonetic symbol consists of the sample phonetic symbol and the following N-1 consecutive phonetic symbols of the sample phonetic symbol; in S103, where the N-gram phonetic string containing the target phonetic symbol consists of the first N... In the case where the sample phonetic symbol consists of -1 consecutive phonetic symbols and the target phonetic symbol, the N-gram phonetic symbol string containing the sample phonetic symbol is composed of the sample phonetic symbol, the first N-1 consecutive phonetic symbols of the sample phonetic symbol, and the sample phonetic symbol; in the case where the N-gram phonetic symbol string containing the target phonetic symbol consists of the first L consecutive phonetic symbols of the target phonetic symbol, the target phonetic symbol, and the last D consecutive phonetic symbols of the target phonetic symbol in S103 above, the N-gram phonetic symbol string containing the sample phonetic symbol consists of the first L consecutive phonetic symbols of the sample phonetic symbol, the sample phonetic symbol, and the last D consecutive phonetic symbols of the sample phonetic symbol.

[0087] In the correspondence between the reference phonetic symbols and the lip shape parameters, the length of each reference phonetic symbol sequence is N. Each sample phonetic symbol is in the same position within its corresponding reference phonetic symbol sequence, that is, it is located at a preset position within its corresponding reference phonetic symbol sequence.

[0088] In S206, the parameter sequence formed by the lip-shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol is determined as the first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

[0089] In this disclosure, the lip shape parameters of each video frame corresponding to the sample phonetic symbol can be arranged according to the chronological order of the corresponding video frames in the facial video to obtain the parameter sequence corresponding to the sample phonetic symbol, that is, to obtain the first candidate parameter sequence corresponding to the reference phonetic symbol sequence in which the sample phonetic symbol is located. Each lip shape parameter in the first candidate parameter sequence includes a set of parameters used to characterize lip features. Thus, the first candidate parameter sequence corresponding to the reference phonetic symbol sequence includes the parameter value sequence corresponding to each parameter in the aforementioned set of parameters.

[0090] In S207, for each reference phonetic symbol sequence, the lip shape parameter corresponding to the reference phonetic symbol sequence is determined based on at least one first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

[0091] In this disclosure, a facial video may contain the same reference phonetic sequence, and different facial videos may also contain the same reference phonetic sequence. Therefore, after obtaining the lip shape parameters corresponding to the reference phonetic sequence contained in each facial video, the lip shape parameters corresponding to the same reference phonetic sequence can be summarized. In this way, a reference phonetic sequence may correspond to multiple first candidate parameter sequences.

[0092] The following is a detailed description of the specific implementation method for determining the lip shape parameter corresponding to the reference phonetic symbol sequence based on at least one first candidate parameter sequence in step S207 above. Specifically, this can be achieved through various implementation methods. In one implementation method, the mean or median of the lip shape parameters in all first candidate parameter sequences corresponding to the reference phonetic symbol sequence can be directly determined as the lip shape parameter corresponding to the reference phonetic symbol sequence. Since the lip shape parameters include a set of parameters used to characterize lip features, determining the mean or median of the lip shape parameters in all first candidate parameter sequences means, for each parameter in the aforementioned set of parameters, calculating the mean or median of the parameter values ​​in the parameter value sequence corresponding to that parameter.

[0093] In another implementation, outliers in the lip shape parameters of all first candidate parameter sequences corresponding to the reference phonetic sequence can be removed first. Then, the mean or median of the resulting first candidate parameter sequences after outlier removal can be used as the lip shape parameter corresponding to the reference phonetic sequence. This improves the accuracy of the lip shape parameter corresponding to the reference phonetic sequence, making it more consistent with actual facial features.

[0094] Specifically, for each parameter in the aforementioned set of parameters, the mean and variance of each parameter value in the corresponding parameter value sequence can be calculated, and then outliers can be removed from the corresponding parameter value sequence based on the mean and variance.

[0095] For example, parameter values ​​that do not fall within the range of [mean - T * variance, mean + T * variance] can be identified as outliers, where T is a constant greater than 1, for example, T = 2.

[0096] In another implementation, the lip shape parameters corresponding to the reference phonetic sequence can be determined by the following steps (a) and (b):

[0097] Step (a): For each first candidate parameter sequence corresponding to the reference phonetic symbol sequence, determine the reference lip shape parameter from the first candidate parameter sequence.

[0098] Step (b): Determine the mean or median of each reference lip shape parameter as the lip shape parameter corresponding to the reference phonetic sequence.

[0099] Specifically, for each parameter in the aforementioned set of parameters, a reference lip shape parameter can be determined from the parameter value sequence corresponding to that parameter, resulting in multiple reference lip shape parameters corresponding to that parameter. The reference lip shape parameter is the parameter value in the parameter value sequence that best represents the lip shape feature of that parameter in the current reference phonetic symbol sequence. Then, the mean or median of the multiple reference lip shape parameters corresponding to that parameter is determined as the target parameter value. Thus, the lip shape parameter corresponding to the reference phonetic symbol sequence can be obtained.

[0100] In another implementation, to improve the accuracy of the lip shape parameters corresponding to the reference phonetic sequence and better reflect the actual facial features, after determining the reference lip shape parameters in each first candidate parameter sequence, they are not directly used to determine the lip shape parameters corresponding to the reference phonetic sequence. Instead, outlier removal is performed on the reference lip shape parameters, and then the lip shape parameters obtained after outlier removal are used to determine the lip shape parameters corresponding to the reference phonetic sequence. Specifically, the lip shape parameters corresponding to the reference phonetic sequence can be determined through the following steps (a) to (c):

[0101] Step (a): For each first candidate parameter sequence corresponding to the reference phonetic symbol sequence, determine the reference lip shape parameter from the first candidate parameter sequence.

[0102] Step (b): Remove outlier data from all reference lip shape parameters corresponding to the reference phonetic sequence.

[0103] In this disclosure, outlier data can be removed from all reference lip-shape parameters corresponding to the reference phonetic sequence using a method similar to the outlier removal method described above for all first candidate parameter sequences corresponding to the reference phonetic sequence. This disclosure will not elaborate further.

[0104] Step (c): The mean or median of the reference lip shape parameters obtained after removing outlier data is determined as the lip shape parameters corresponding to the reference phonetic sequence.

[0105] The following is a detailed description of the specific implementation method for determining the reference lip shape parameter from the first candidate parameter sequence in step a) above.

[0106] Specifically, the absolute value of the difference between adjacent lip parameters in the first candidate parameter sequence can be calculated first to obtain the gradient sequence. Then, the lip parameter at the Nth position in the first candidate parameter sequence is determined as the reference lip parameter. The maximum value in the gradient sequence is located at the Nth position of the gradient sequence. The position in the first candidate parameter sequence corresponding to the position of the maximum value in the gradient sequence is determined as the reference lip parameter.

[0107] For example, lip shape parameters include jawOpen (the degree to which the mouth opens) and mouthSmileLeft (the degree to which the left side of the mouth is turned into a smile), such as... Figure 4 As shown, the reference phonetic sequence for the sample phonetic symbol "a" in the sample IPA sequence corresponding to the audio in a facial video is "ab". The lip shape parameters corresponding to the reference phonetic sequence "ab" include: the parameter value sequence "0.1, 0.2, 0.3, 0.1, 0.0, 0.3" for jawOpen and the parameter value sequence "0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1" for mouthSmileLeft. In this sample video segment, the number of video frames corresponding to the phonetic symbol "a" is 5, and the number of video frames corresponding to the phonetic symbol "b" is 1. Therefore, the jawOpen values ​​for each video frame corresponding to the phonetic symbol "a" are 0.1, 0.2, 0.3, 0.1, and 0.0, respectively, and the mouthSmileLeft values ​​are 0.1, 0.1, 0.1, 0.1, 0.1, and 0.1, respectively. The jawOpen value for the video frame corresponding to the phonetic symbol "b" is 0.3, and the mouthSmileLeft value is 0.1. The parameter sequence formed by the lip-shape parameters of each video frame in the sample video segment corresponding to the phonetic symbol "a" includes the parameter value sequence for jawOpen: "0.1, 0.2, 0.3, 0.1, 0.0", and the parameter value sequence for mouthSmileLeft: "0.1, 0.1, 0.1, 0.1". The first candidate parameter sequence corresponding to the reference phonetic sequence "ab" includes: the parameter value sequence "0.1, 0.2, 0.3, 0.1, 0.0" for jawOpen and the parameter value sequence "0.1, 0.1, 0.1, 0.1, 0.1" for mouthSmileLeft. For the parameter value sequence "0.1, 0.2, 0.3, 0.1, 0.0" for jawOpen, the absolute value of the difference between adjacent jawOpen parameter values ​​in the sequence is calculated to obtain the gradient sequence "0.1, 0.1, 0.2, 0.1". The maximum value in the gradient sequence is located at the 3rd position of the gradient sequence. Therefore, the jawOpen parameter value "0.3" at the 3rd position in the parameter value sequence "0.1, 0.2, 0.3, 0.1, 0.0" can be determined as the reference lip shape parameter for this parameter value sequence.

[0108] In the above implementation, since the parameters that change drastically before and after are the most representative of the changes in lip shape, the position corresponding to the position of the maximum value in the gradient sequence in the first candidate parameter sequence is determined as the reference lip shape parameter, thereby improving the accuracy of lip shape driving.

[0109] The following is a detailed explanation of the specific implementation method for determining the lip shape parameters that match the target phonetic sequence based on the pre-established correspondence between the reference phonetic sequence and the lip shape parameters in S104 above. Specifically, this can be achieved through various implementation methods, including the following steps (1) to (3):

[0110] Step (1): Determine whether there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence between the reference phonetic sequence and the lip shape parameters.

[0111] In this disclosure, if there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence between the reference phonetic sequence and the lip shape parameter, then step (2) is performed; otherwise, step (3) is performed.

[0112] Step (2): Determine the lip shape parameters corresponding to the perfectly matching reference phonetic sequence as the lip shape parameters that match the target phonetic sequence.

[0113] For example, such as Figure 2 As shown, the target phonetic sequence is "t, a". The correspondence between the reference phonetic sequence and the lip-shaped reference phonetic sequence and the lip-shaped parameters includes the correspondence between the reference phonetic sequence "t, a" and the lip-shaped parameters. That is, the jawOpen corresponding to the reference phonetic sequence "t, a" is 0.3. Therefore, the jawOpen matching the target phonetic sequence "t, a" is 0.3.

[0114] Step (3): Based on the pre-established correspondence between IPA phonetic symbols and lip shape parameters, determine the lip shape parameters corresponding to the target phonetic symbols, that is, obtain the lip shape parameters that match the target phonetic symbol sequence.

[0115] When constructing the correspondence between reference phonetic sequences and lip shape parameters, the limited number of facial videos may not cover all IPA combinations of length N. Therefore, this can be compensated for by using the correspondence between IPA combinations of length less than N and lip shape parameters. Specifically, for example... Figure 5 As shown, the above-mentioned correspondence can also be constructed in the following ways: S208 to S215.

[0116] In S208, let k = 1.

[0117] In S209, for each sample phonetic symbol, the (Nk)-gram phonetic symbol string containing that sample phonetic symbol is extracted from the sample IPA sequence to obtain the sample phonetic symbol sequence.

[0118] In this disclosure, a (Nk)-gram phonetic string containing the sample phonetic symbol can be extracted from the sample IPA sequence using a rule similar to S205 described above, the difference being the length of the phonetic string. Specifically, in S205 described above, when the N-gram phonetic string containing the sample phonetic symbol consists of the sample phonetic symbol and the last N-1 consecutive phonetic symbols of the sample phonetic symbol, the (Nk)-gram phonetic string containing the sample phonetic symbol consists of the sample phonetic symbol and the last N-2 consecutive phonetic symbols of the sample phonetic symbol; in S205 described above, when the N-gram phonetic string containing the sample phonetic symbol consists of the first N-1 consecutive phonetic symbols of the sample phonetic symbol and the sample phonetic symbol, the (Nk)-gram phonetic string containing the sample phonetic symbol... The phonetic string consists of the sample phonetic symbol, the first N-2 consecutive phonetic symbols of the sample phonetic symbol, and the sample phonetic symbol. In the case of the above S205, where the N-gram phonetic string containing the sample phonetic symbol consists of the first L consecutive phonetic symbols of the sample phonetic symbol, the sample phonetic symbol, and the last D consecutive phonetic symbols of the sample phonetic symbol, the (Nk)-gram phonetic string containing the sample phonetic symbol consists of the first H consecutive phonetic symbols of the sample phonetic symbol, the sample phonetic symbol, and the last F consecutive phonetic symbols of the sample phonetic symbol, where L≥H, D≥F, and H+F+1=Nk.

[0119] In S210, the parameter sequence formed by the lip-shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol is determined as the second candidate parameter sequence corresponding to the sample phonetic symbol sequence.

[0120] In this disclosure, the lip shape parameters of each video frame corresponding to the sample phonetic symbol can be arranged according to the chronological order of the corresponding video frames in the facial video to obtain the parameter sequence corresponding to the sample phonetic symbol, that is, to obtain the second candidate parameter sequence corresponding to the sample phonetic symbol sequence. Each lip shape parameter in the second candidate parameter sequence includes a set of parameters used to characterize lip features. Thus, the second candidate parameter sequence corresponding to the sample phonetic symbol sequence includes a sequence of parameter values ​​corresponding to each parameter in the aforementioned set of parameters.

[0121] In S211, for each sample phonetic symbol sequence, the lip shape parameter corresponding to the sample phonetic symbol sequence is determined based on at least one second candidate parameter sequence corresponding to the sample phonetic symbol sequence.

[0122] In this disclosure, a similar method to that used in S207 above to determine the lip shape parameter corresponding to the reference phonetic sequence based on at least one first candidate parameter sequence can be used to determine the lip shape parameter corresponding to the sample phonetic sequence based on at least one second candidate parameter sequence. This disclosure will not elaborate further.

[0123] In S212, the sample phonetic symbol sequence is expanded into an N-gram phonetic symbol string by adding preset characters to the sample phonetic symbol sequence, thus obtaining a reference phonetic symbol sequence.

[0124] In this disclosure, the default character can be, for example, "_default_" (e.g., Figure 2 As shown in the diagram, when the (Nk)-gram phonetic string containing the sample phonetic symbol consists of the sample phonetic symbol and the following N-2 consecutive phonetic symbols, k consecutive preset characters can be added to the end of the sample phonetic symbol sequence to expand it into an N-gram phonetic string; when the (Nk)-gram phonetic string containing the sample phonetic symbol consists of the sample phonetic symbol, the preceding N-2 consecutive phonetic symbols of the sample phonetic symbol, and the sample phonetic symbol, k consecutive preset characters can be added to the beginning of the sample phonetic symbol sequence to expand it into an N-gram phonetic string; when the (Nk)-gram phonetic string containing the sample phonetic symbol consists of the preceding H consecutive phonetic symbols of the sample phonetic symbol, the sample phonetic symbol, and the following F consecutive phonetic symbols of the sample phonetic symbol, (L-H) consecutive preset characters can be added to the beginning of the sample phonetic symbol sequence, and (D-F) consecutive preset characters can be added after it to expand it into an N-gram phonetic string.

[0125] In S213, the lip shape parameters corresponding to the sample phonetic symbol sequence are determined as the lip shape parameters corresponding to the expanded reference phonetic symbol sequence.

[0126] In S214, determine whether Nk is equal to 1.

[0127] If Nk is not equal to 1, then execute S214 below, and then return to S209 above until Nk = 1; if Nk = 1, then the correspondence is completed, and at this time, we can end.

[0128] In S215, let k be incremented by 1.

[0129] For example, N=3. After constructing the correspondence between the reference phonetic sequence and the lip shape parameter with a length of 3 and without preset characters through the above S201 to S205, the correspondence between the reference phonetic sequence and the lip shape parameter can be further improved by the correspondence between the IPA combination with a length of 2 and the lip shape parameter, and the correspondence between the IPA with a length of 1 and the lip shape parameter. Specifically, first, let k=1. Then, for each sample phonetic symbol, extract a 2-gram (i.e., bigram) phonetic symbol string containing that sample phonetic symbol from the sample IPA sequence to obtain a sample phonetic symbol sequence of length 2. Then, determine the parameter sequence composed of the lip-shape parameters of each video frame in the video segment corresponding to that sample phonetic symbol as the second candidate parameter sequence corresponding to that sample phonetic symbol sequence. Next, for each sample phonetic symbol sequence of length 2, determine the lip-shape parameter corresponding to that sample phonetic symbol sequence based on at least one second candidate parameter sequence corresponding to that sample phonetic symbol sequence. Thus, the correspondence between the sample phonetic symbol sequence of length 2 and the lip-shape parameter can be obtained. Then, by adding one preset character to the sample phonetic symbol sequence, the sample phonetic symbol sequence is expanded into a 3-gram phonetic symbol string to obtain a reference phonetic symbol sequence of length 3. After that, the lip-shape parameter corresponding to the sample phonetic symbol sequence is determined as the lip-shape parameter corresponding to the expanded reference phonetic symbol sequence (length 3, containing one preset character). Thus, the correspondence between the reference phonetic symbol sequence of length 3 containing one preset character and the lip-shape parameter can be obtained.

[0130] At this point, N-1 is greater than 1. Then, let k = 2, and for each sample phonetic symbol, extract the 1-gram (i.e., unigram) phonetic symbol string containing that sample phonetic symbol from the sample IPA sequence to obtain a sample phonetic symbol sequence of length 1. Then, determine the parameter sequence formed by the lip-shape parameters of each video frame in the video segment corresponding to that sample phonetic symbol as the second candidate parameter sequence corresponding to that sample phonetic symbol sequence. Next, for each sample phonetic symbol sequence of length 1, determine the pair of sample phonetic symbols based on at least one second candidate parameter sequence corresponding to that sample phonetic symbol sequence. The corresponding lip shape parameters are obtained, thus the correspondence between the sample phonetic sequence of length 1 and the lip shape parameters can be obtained; then, by adding 2 preset characters to the sample phonetic sequence, the sample phonetic sequence is expanded into a 3-gram phonetic string, resulting in a reference phonetic sequence of length 3; then, the lip shape parameters corresponding to the sample phonetic sequence are determined as the lip shape parameters corresponding to the expanded reference phonetic sequence (length 3, containing 2 preset characters), thus the correspondence between the reference phonetic sequence of length 3 containing 2 preset characters and the lip shape parameters can be obtained.

[0131] At this point, S104 can determine the lip shape parameters that match the target phonetic sequence through the following steps [1] to [4]:

[0132] Step [1]: Determine whether there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence between the reference phonetic sequence and the lip shape parameters.

[0133] In this disclosure, if there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence between the reference phonetic sequence and the lip shape parameter, then the following step [2] is performed; otherwise, the following steps [3] and [4] are performed.

[0134] Step [2]: Determine the lip shape parameters corresponding to the perfectly matching reference phonetic sequence as the lip shape parameters that match the target phonetic sequence.

[0135] Step [3]: From the reference phonetic symbol sequence containing preset characters, determine the candidate phonetic symbol sequence that matches the target phonetic symbol sequence the most.

[0136] In this disclosure, the preset character is not an International Phonetic Alphabet (IPA) symbol and is located in a position other than the preset position in the reference phonetic symbol sequence in which it is located. The phonetic symbols in the candidate phonetic symbol sequence at other positions besides the position of the preset character are the same as the phonetic symbols at the corresponding positions in the target phonetic symbol sequence.

[0137] In addition, the candidate phonetic sequence containing the preset character that has the highest matching degree with the target phonetic sequence may be zero, that is, there is no reference phonetic sequence containing the preset character that matches the target phonetic sequence, or there may be one or more.

[0138] Step [4]: ​​Determine the lip shape parameters that match the target lip shape sequence based on the lip shape parameters corresponding to the candidate lip shape sequence.

[0139] Specifically, if the candidate phonetic sequence containing the preset character that has the highest matching degree with the target phonetic sequence determined in the above step [3] is zero, then the lip shape parameter corresponding to the target phonetic symbol is determined according to the pre-established correspondence between IPA phonetic symbols and lip shape parameters, that is, the lip shape parameter that matches the target phonetic sequence is obtained.

[0140] If the candidate phonetic sequence containing the preset character that has the highest matching degree with the target phonetic sequence is determined in the above step [3], then the lip shape parameter corresponding to the candidate phonetic sequence is determined as the lip shape parameter that matches the target phonetic sequence.

[0141] If the candidate phonetic sequence containing the preset character that has the highest matching degree with the target phonetic sequence is determined in the above step [3], then the mean or median of the lip shape parameter corresponding to each candidate phonetic sequence is determined as the lip shape parameter that matches the target phonetic sequence.

[0142] For example, such as Figure 2 As shown, the target phonetic symbol sequence is "a, The default position is the first position of the corresponding phonetic symbol sequence. The correspondence between the reference phonetic symbol sequence and the lip shape parameters does not include the reference phonetic symbol sequence "a". The correspondence between the target phonetic sequence "a" and the lip shape parameters, but including the correspondence between the reference phonetic sequence "a" and "_default_", where "_default_" is a preset character, and in the reference phonetic sequence containing the preset character, the correspondence between the target phonetic sequence "a" and the lip shape parameters. The candidate phonetic sequence with the highest matching degree is "a、_default_". At this point, the jawOpen value corresponding to the reference phonetic sequence "a、_default_" is 0.3. Therefore, the target phonetic sequence "a、_default_" is the correct match. "The matching jawOpen is 0.3."

[0143] Figure 6 This is a block diagram illustrating a virtual avatar lip-syncing device according to an exemplary embodiment. Figure 6 As shown, the device 300 includes: a first determining module 301, used to determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence; a first expanding module 302, used to extend the target IPA sequence according to each target pronunciation duration to obtain an expanded sequence; a first extraction module 303, used to extract an N-gram phonetic string containing the target phoneme from the expanded sequence for each target phoneme in the expanded sequence to obtain a target phonetic sequence, wherein N≥2, and the target phoneme is located at a preset position in the target phonetic sequence; a second determining module 304, used to determine the lip shape parameter matching the target phonetic sequence according to a pre-established correspondence between a reference phonetic sequence and lip shape parameters, wherein the lip shape parameter corresponding to the reference phonetic sequence is used to characterize the lip shape parameter corresponding to the phoneme at the preset position in the reference phonetic sequence; and a rendering module 305, used to perform lip shape rendering on the target virtual image based on the lip shape parameter corresponding to the target phoneme.

[0144] In the above technical solution, after obtaining the extended sequence corresponding to the target text, when obtaining the corresponding lip shape parameters for each target phonetic symbol in the extended sequence, not only the target phonetic symbol itself is considered, but also its contextual information. In this way, the lip shape parameters under different IPA combinations can be more in line with the real facial situation, and good results can be obtained without deliberately adjusting the smoothing filter parameters. This makes the lip shape of the target virtual image as close to that of a real person as possible, thus improving the animation effect.

[0145] Optionally, the second determining module 304 includes:

[0146] The first determining submodule is used to determine the lip shape parameter corresponding to the completely matching reference phonetic sequence as the lip shape parameter matching the target phonetic sequence if there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence.

[0147] Optionally, the second determining module 304 further includes:

[0148] The second determining submodule is used to determine, if there is no reference phonetic sequence in the correspondence that completely matches the target phonetic sequence, a candidate phonetic sequence with the highest matching degree with the target phonetic sequence from the reference phonetic sequence containing a preset character, wherein the preset character is not an International Phonetic Alphabet and is located in a position other than the preset position in the reference phonetic sequence in which it is located, and the phonetic symbols in the candidate phonetic sequence at other positions other than the position where the preset character is located are the same as the phonetic symbols at the corresponding positions in the target phonetic sequence;

[0149] The second determining submodule is used to determine the lip shape parameters that match the target lip shape sequence based on the lip shape parameters corresponding to the candidate lip shape sequence.

[0150] Optionally, the second determining submodule includes:

[0151] The third determining submodule is used to determine the mean or median of the lip shape parameters corresponding to each candidate phonetic sequence as the lip shape parameters that match the target phonetic sequence if there are multiple candidate phonetic sequence sequences.

[0152] The fourth determining submodule is used to determine the lip shape parameter corresponding to the candidate phonetic symbol sequence as the lip shape parameter that matches the target phonetic symbol sequence if there is only one candidate phonetic symbol sequence.

[0153] Optionally, the correspondence is constructed by a construction device, wherein the construction device includes:

[0154] The acquisition module is used to acquire multiple facial videos;

[0155] The second extraction module is used to extract lip shape parameters from each video frame of each facial video; the third determination module is used to determine the sample IPA sequence corresponding to the audio in the facial video and the sample pronunciation duration of each sample phonetic symbol in the sample IPA sequence; the fourth determination module is used to determine the video segment corresponding to each sample phonetic symbol from the facial video according to the sample IPA sequence and the pronunciation duration of each sample phonetic symbol; the third extraction module is used to extract an N-gram phonetic symbol string containing the sample phonetic symbol from the sample IPA sequence for each sample phonetic symbol to obtain a reference phonetic symbol sequence; the fifth determination module is used to determine the parameter sequence composed of the lip shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol as the first candidate parameter sequence corresponding to the reference phonetic symbol sequence;

[0156] The sixth determining module is used to determine the lip shape parameters corresponding to each of the reference phonetic symbols based on at least one first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

[0157] Optionally, the sixth determining module includes:

[0158] The fifth determining submodule is used to determine reference lip shape parameters from each of the first candidate parameter sequences corresponding to the reference phonetic symbol sequence.

[0159] The sixth determining submodule is used to determine the mean or median of each of the reference lip shape parameters as the lip shape parameter corresponding to the reference phonetic sequence.

[0160] Optionally, the fifth determining submodule includes:

[0161] The computation submodule is used to calculate the absolute value of the difference between adjacent lip shape parameters in the first candidate parameter sequence to obtain the gradient sequence;

[0162] The seventh determining submodule is used to determine the lip shape parameter at the Nth position in the first candidate parameter sequence as the reference lip shape parameter, wherein the maximum value in the gradient sequence is located at the Nth position of the gradient sequence.

[0163] Optionally, the sixth determining module further includes:

[0164] The elimination submodule is used to remove outlier data from all the reference lip shape parameters corresponding to the reference phonetic sequence before the sixth determining submodule determines the mean or median of each reference lip shape parameter as the lip shape parameter corresponding to the reference phonetic sequence.

[0165] The sixth determining submodule is used to determine the mean or median of the reference lip shape parameters obtained after removing outlier data as the lip shape parameters corresponding to the reference phonetic sequence.

[0166] Optionally, the construction apparatus further includes:

[0167] The configuration module is used to set k=1;

[0168] The fourth extraction module is used to extract (Nk)-gram phonetic string containing the sample phonetic symbol from the sample IPA sequence for each sample phonetic symbol to obtain a sample phonetic symbol sequence; the seventh determination module is used to determine the parameter sequence composed of the lip shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol as the second candidate parameter sequence corresponding to the sample phonetic symbol sequence.

[0169] The eighth determining module is used to determine the lip shape parameter corresponding to each sample phonetic symbol sequence based on at least one second candidate parameter sequence corresponding to the sample phonetic symbol sequence; the second expanding module is used to expand the sample phonetic symbol sequence into an N-gram phonetic symbol string by adding preset characters to the sample phonetic symbol sequence to obtain a reference phonetic symbol sequence; the ninth determining module is used to determine the lip shape parameter corresponding to the sample phonetic symbol sequence as the lip shape parameter corresponding to the expanded reference phonetic symbol sequence;

[0170] The triggering module is used to increment k by 1 and trigger the fourth extraction module to extract the (Nk)-gram phonetic symbol string containing the phonetic symbol of the sample from the sample IPA sequence until Nk = 1.

[0171] Optionally, the rendering module 305 includes: a filtering submodule, used to filter the lip shape parameters corresponding to the target phonetic symbol based on the lip shape parameters corresponding to the neighboring phonetic symbols of the target phonetic symbol, wherein the neighboring phonetic symbols are phonetic symbols in the extended sequence whose positional distance from the target phonetic symbol is less than a preset distance threshold;

[0172] The rendering submodule is used to input the filtered lip shape parameters corresponding to the target phonetic symbols into the preset rendering engine to render the lip shape of the target virtual image.

[0173] This disclosure also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the virtual avatar lip-driving method described above.

[0174] The following is for reference. Figure 7The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0175] like Figure 7 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0176] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 7 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0177] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0178] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0179] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0180] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0181] The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following: It determines the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence; it extends the target IPA sequence according to each target pronunciation duration to obtain an extended sequence; for each target phoneme in the extended sequence, it extracts an N-gram phoneme string containing the target phoneme from the extended sequence to obtain a target phoneme sequence, wherein N≥2, and the target phoneme is located at a preset position in the target phoneme sequence; it determines the lip shape parameter matching the target phoneme sequence according to a pre-established correspondence between a reference phoneme sequence and lip shape parameters, wherein the lip shape parameter corresponding to the reference phoneme sequence is used to characterize the lip shape parameter corresponding to the phoneme at the preset position in the reference phoneme sequence; and it performs lip shape rendering on the target virtual image based on the lip shape parameter corresponding to the target phoneme.

[0182] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0184] The modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules do not necessarily limit the module itself; for example, the first determining module can also be described as "a module for determining the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence".

[0185] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0186] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0187] According to one or more embodiments of this disclosure, Example 1 provides a virtual avatar lip-shape driving method, comprising: determining a target IPA sequence corresponding to a target text and a target pronunciation duration for each phoneme in the target IPA sequence; extending the target IPA sequence according to each target pronunciation duration to obtain an extended sequence; for each target phoneme in the extended sequence, extracting an N-gram phoneme string containing the target phoneme from the extended sequence to obtain a target phoneme sequence, wherein N≥2, and the target phoneme is located at a preset position in the target phoneme sequence; determining a lip-shape parameter matching the target phoneme sequence according to a pre-established correspondence between a reference phoneme sequence and lip-shape parameters, wherein the lip-shape parameter corresponding to the reference phoneme sequence is used to characterize the lip-shape parameter corresponding to the phoneme at the preset position in the reference phoneme sequence; and rendering the lip shape of the target virtual avatar based on the lip-shape parameter corresponding to the target phoneme.

[0188] According to one or more embodiments of this disclosure, Example 2 provides the method of Example 1, wherein determining the lip shape parameter that matches the target phonetic symbol sequence based on a pre-established correspondence between a reference phonetic symbol sequence and lip shape parameters includes: if there is a reference phonetic symbol sequence in the correspondence that completely matches the target phonetic symbol sequence, then the lip shape parameter corresponding to the completely matching reference phonetic symbol sequence is determined as the lip shape parameter that matches the target phonetic symbol sequence.

[0189] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, wherein determining the lip shape parameter matching the target phonetic sequence based on a pre-established correspondence between a reference phonetic sequence and lip shape parameters further includes: if there is no reference phonetic sequence in the correspondence that completely matches the target phonetic sequence, then determining a candidate phonetic sequence with the highest matching degree with the target phonetic sequence from the reference phonetic sequence containing a preset character, wherein the preset character is not an International Phonetic Alphabet and is located in a position other than the preset position in its reference phonetic sequence, and the phonetic symbols in the candidate phonetic sequence other than the position of the preset character are the same as the phonetic symbols in the corresponding positions in the target phonetic sequence; and determining the lip shape parameter matching the target phonetic sequence based on the lip shape parameter corresponding to the candidate phonetic sequence.

[0190] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, wherein determining the lip shape parameter matching the target lip shape parameter based on the lip shape parameter corresponding to the candidate lip shape sequence includes: if there are multiple candidate lip shape sequences, determining the mean or median of the lip shape parameter corresponding to each candidate lip shape sequence as the lip shape parameter matching the target lip shape sequence; if there is only one candidate lip shape sequence, determining the lip shape parameter corresponding to the candidate lip shape sequence as the lip shape parameter matching the target lip shape sequence.

[0191] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 1, wherein the correspondence is constructed in the following manner: acquiring multiple facial videos; for each facial video, extracting lip shape parameters from each video frame of the facial video; determining a sample IPA sequence corresponding to the audio in the facial video and a sample pronunciation duration for each sample phonetic symbol in the sample IPA sequence; determining a video segment corresponding to each sample phonetic symbol from the facial video based on the sample IPA sequence and each sample pronunciation duration; for each sample phonetic symbol, extracting an N-gram phonetic symbol string containing the sample phonetic symbol from the sample IPA sequence to obtain a reference phonetic symbol sequence; determining a parameter sequence composed of lip shape parameters from each video frame of the video segment corresponding to the sample phonetic symbol as a first candidate parameter sequence corresponding to the reference phonetic symbol sequence; for each reference phonetic symbol sequence, determining the lip shape parameters corresponding to the reference phonetic symbol sequence based on at least one first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

[0192] According to one or more embodiments of this disclosure, Example 6 provides the method of Example 5, wherein determining the lip shape parameter corresponding to the reference phonetic sequence based on at least one first candidate parameter sequence corresponding to the reference phonetic sequence includes: determining a reference lip shape parameter from the first candidate parameter sequence for each of the first candidate parameter sequences corresponding to the reference phonetic sequence; and determining the mean or median of each reference lip shape parameter as the lip shape parameter corresponding to the reference phonetic sequence.

[0193] According to one or more embodiments of this disclosure, Example 7 provides the method of Example 6, wherein determining a reference lip shape parameter from the first candidate parameter sequence includes: calculating the absolute value of the difference between adjacent lip shape parameters in the first candidate parameter sequence to obtain a gradient sequence; determining the lip shape parameter at the Nth position in the first candidate parameter sequence as the reference lip shape parameter, wherein the maximum value in the gradient sequence is located at the Nth position of the gradient sequence.

[0194] According to one or more embodiments of this disclosure, Example 8 provides the method of Example 6, wherein before the step of determining the mean or median of each of the reference lip-shape parameters as the lip-shape parameter corresponding to the reference phonetic sequence, the step of determining the lip-shape parameter corresponding to the reference phonetic sequence based on at least one first candidate parameter sequence corresponding to the reference phonetic sequence further includes: removing outlier data from all the reference lip-shape parameters corresponding to the reference phonetic sequence; the step of determining the mean or median of each of the reference lip-shape parameters as the lip-shape parameter corresponding to the reference phonetic sequence includes: determining the mean or median of the reference lip-shape parameters obtained after removing outlier data as the lip-shape parameter corresponding to the reference phonetic sequence.

[0195] According to one or more embodiments of this disclosure, Example 9 provides a method of any one of Examples 5-8, wherein the construction of the correspondence further includes the following steps: Let k = 1; for each sample phonetic symbol, extract the (Nk)-gram phonetic symbol string containing the sample phonetic symbol from the sample IPA sequence to obtain a sample phonetic symbol sequence; determine the parameter sequence composed of the lip-shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol as the second candidate parameter sequence corresponding to the sample phonetic symbol sequence; for each sample phonetic symbol sequence, determine the lip-shape parameter corresponding to the sample phonetic symbol sequence according to at least one second candidate parameter sequence corresponding to the sample phonetic symbol sequence; expand the sample phonetic symbol sequence into an N-gram phonetic symbol string by adding a preset character to the sample phonetic symbol sequence to obtain a reference phonetic symbol sequence; determine the lip-shape parameter corresponding to the sample phonetic symbol sequence as the lip-shape parameter corresponding to the expanded reference phonetic symbol sequence; let k increment by 1, and return to the step of extracting the (Nk)-gram phonetic symbol string containing the sample phonetic symbol from the sample IPA sequence until Nk = 1.

[0196] According to one or more embodiments of this disclosure, Example 10 provides a method of any one of Examples 1-8, wherein lip-shape rendering of a target virtual image based on the lip-shape parameters corresponding to the target phonetic symbol includes: filtering the lip-shape parameters corresponding to the target phonetic symbol according to the lip-shape parameters corresponding to the neighboring phonetic symbols of the target phonetic symbol, wherein the neighboring phonetic symbols are phonetic symbols in the extended sequence whose positional distance from the target phonetic symbol is less than a preset distance threshold; and inputting the filtered lip-shape parameters corresponding to the target phonetic symbol into a preset rendering engine to perform lip-shape rendering of the target virtual image.

[0197] According to one or more embodiments of this disclosure, Example 11 provides a virtual avatar lip-shape driving device, comprising: a first determining module, configured to determine a target IPA sequence corresponding to a target text and a target pronunciation duration of each phoneme in the target IPA sequence; a first expanding module, configured to extend the target IPA sequence according to each target pronunciation duration to obtain an expanded sequence; a first extraction module, configured to extract an N-gram phonetic string containing the target phoneme from the expanded sequence for each target phoneme in the expanded sequence to obtain a target phonetic sequence, wherein N≥2, and the target phoneme is located at a preset position in the target phonetic sequence; a second determining module, configured to determine a lip-shape parameter matching the target phonetic sequence according to a pre-established correspondence between a reference phonetic sequence and lip-shape parameters, wherein the lip-shape parameter corresponding to the reference phonetic sequence is used to characterize the lip-shape parameter corresponding to the phoneme at the preset position in the reference phonetic sequence; and a rendering module, configured to perform lip-shape rendering on the target virtual avatar based on the lip-shape parameter corresponding to the target phoneme.

[0198] According to one or more embodiments of this disclosure, Example 12 provides an apparatus of Example 11, wherein the second determining module includes: a first determining submodule, configured to determine the lip shape parameter corresponding to the fully matching reference phonetic sequence as the lip shape parameter matching the target phonetic sequence if there is a reference phonetic sequence that completely matches the target phonetic sequence in the correspondence.

[0199] According to one or more embodiments of this disclosure, Example 13 provides the apparatus of Example 12, wherein the second determining module further includes: a second determining submodule, configured to determine, if there is no reference phonetic sequence in the correspondence that completely matches the target phonetic sequence, a candidate phonetic sequence with the highest matching degree with the target phonetic sequence from the reference phonetic sequence containing a preset character, wherein the preset character is not an International Phonetic Alphabet and is located in a position other than the preset position in the reference phonetic sequence in which it is located, and the phonetic symbols in the candidate phonetic sequence other than the position where the preset character is located are the same as the phonetic symbols in the corresponding positions in the target phonetic sequence; the second determining submodule is configured to determine lip shape parameters that match the target phonetic sequence based on the lip shape parameters corresponding to the candidate phonetic sequence.

[0200] According to one or more embodiments of this disclosure, Example 14 provides the apparatus of Example 13, wherein the second determining submodule includes: a third determining submodule, configured to, if there are multiple candidate phonetic sequence sequences, determine the mean or median of the lip shape parameter corresponding to each candidate phonetic sequence as the lip shape parameter matching the target phonetic sequence; and a fourth determining submodule, configured to, if there is only one candidate phonetic sequence, determine the lip shape parameter corresponding to the candidate phonetic sequence as the lip shape parameter matching the target phonetic sequence.

[0201] According to one or more embodiments of this disclosure, Example 15 provides the apparatus of Example 11, wherein the correspondence is constructed by a construction device, wherein the construction device includes: an acquisition module for acquiring multiple facial videos; a second extraction module for extracting lip shape parameters from each video frame of each facial video; a third determination module for determining a sample IPA sequence corresponding to audio in the facial video and a sample pronunciation duration of each sample phonetic symbol in the sample IPA sequence; a fourth determination module for determining a video segment corresponding to each sample phonetic symbol from the facial video based on the sample IPA sequence and the sample pronunciation duration of each sample phonetic symbol; a third extraction module for extracting an N-gram phonetic symbol string containing the sample phonetic symbol from the sample IPA sequence for each sample phonetic symbol to obtain a reference phonetic symbol sequence; a fifth determination module for determining a parameter sequence composed of lip shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol as a first candidate parameter sequence corresponding to the reference phonetic symbol sequence; and a sixth determination module for determining lip shape parameters corresponding to each reference phonetic symbol sequence based on at least one first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

[0202] According to one or more embodiments of this disclosure, Example 16 provides the apparatus of Example 15, wherein the sixth determining module includes: a fifth determining submodule, configured to determine a reference lip shape parameter from the first candidate parameter sequence for each of the first candidate parameter sequences corresponding to the reference phonetic sequence; and a sixth determining submodule, configured to determine the mean or median of each of the reference lip shape parameters as the lip shape parameter corresponding to the reference phonetic sequence.

[0203] According to one or more embodiments of this disclosure, Example 17 provides an apparatus of Example 16, wherein the fifth determining submodule includes: a calculation submodule for calculating the absolute value of the difference between adjacent lip parameters in the first candidate parameter sequence to obtain a gradient sequence; and a seventh determining submodule for determining the lip parameter at the Nth position in the first candidate parameter sequence as a reference lip parameter, wherein the maximum value in the gradient sequence is located at the Nth position of the gradient sequence.

[0204] According to one or more embodiments of this disclosure, Example 18 provides the apparatus of Example 16, wherein the sixth determining module further includes: a removal submodule, configured to remove outlier data from all the reference lip-shape parameters corresponding to the reference phonetic sequence before the sixth determining submodule determines the mean or median of each of the reference lip-shape parameters as the lip-shape parameter corresponding to the reference phonetic sequence; the sixth determining submodule is configured to determine the mean or median of the reference lip-shape parameters obtained after removing the outlier data as the lip-shape parameter corresponding to the reference phonetic sequence.

[0205] According to one or more embodiments of this disclosure, Example 19 provides an apparatus of any one of Examples 15-18, the construction apparatus further comprising: a setting module for setting k=1; a fourth extraction module for extracting (Nk)-gram phonetic string containing the sample phonetic symbol from the sample IPA sequence for each sample phonetic symbol, to obtain a sample phonetic symbol sequence; a seventh determination module for determining a parameter sequence composed of lip-shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol as a second candidate parameter sequence corresponding to the sample phonetic symbol sequence; and an eighth determination module for determining, for each sample phonetic symbol sequence, a second candidate parameter sequence corresponding to the sample phonetic symbol sequence based on the sample phonetic symbol sequence. The system determines the lip shape parameters corresponding to the sample phonetic sequence by at least one second candidate parameter sequence; the second expansion module is used to expand the sample phonetic sequence into an N-gram phonetic string by adding preset characters to the sample phonetic sequence to obtain a reference phonetic sequence; the ninth determination module is used to determine the lip shape parameters corresponding to the sample phonetic sequence as the lip shape parameters corresponding to the expanded reference phonetic sequence; the trigger module is used to increment k by 1 and trigger the fourth extraction module to extract the (Nk)-gram phonetic string containing the sample phonetic symbol from the sample IPA sequence until Nk = 1.

[0206] According to one or more embodiments of this disclosure, Example 20 provides an apparatus of any one of Examples 11-18, wherein the rendering module includes: a filtering submodule, configured to filter the lip shape parameters corresponding to the target lip shape parameter based on the lip shape parameters corresponding to the neighboring lip shape parameters of the target lip shape parameter, wherein the neighboring lip shape parameter is a lip shape parameter in the extended sequence whose positional distance from the target lip shape parameter is less than a preset distance threshold; and a rendering submodule, configured to input the filtered lip shape parameters corresponding to the target lip shape parameter into a preset rendering engine to perform lip shape rendering on the target virtual image.

[0207] According to one or more embodiments of the present disclosure, Example 21 provides a computer-readable medium having a computer program stored thereon that, when executed by a processing device, implements the steps of the method described in any one of Examples 1-10.

[0208] According to one or more embodiments of the present disclosure, Example 22 provides an electronic device including: a storage device having at least one computer program stored thereon; and at least one processing device for executing the at least one computer program in the storage device to implement the steps of the method of any one of Examples 1-10.

[0209] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0210] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0211] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims. Regarding the apparatus in the above embodiments, the specific manner in which the various modules perform their operations has been described in detail in the embodiments relating to the method, and will not be elaborated upon here.

Claims

1. A method for driving the lip movements of a virtual character, characterized in that, include: Determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phoneme in the target IPA sequence; Based on the duration of each target pronunciation, the target IPA sequence is duration-extended to obtain an extended sequence; For each target phonetic symbol in the extended sequence, an N-gram phonetic symbol string containing the target phonetic symbol is extracted from the extended sequence to obtain a target phonetic symbol sequence, wherein N≥2, and the target phonetic symbol is located at a preset position in the target phonetic symbol sequence; If there is no reference phonetic sequence that completely matches the target phonetic sequence in the pre-established correspondence between reference phonetic sequence and lip shape parameters, then from the reference phonetic sequence containing preset characters, a candidate phonetic sequence with the highest matching degree with the target phonetic sequence is determined. The preset character is not an International Phonetic Alphabet symbol and is located in a position other than the preset position in its reference phonetic sequence. The phonetic symbols in the candidate phonetic sequence at other positions besides the position of the preset character are the same as the phonetic symbols at the corresponding positions in the target phonetic sequence. Based on the lip shape parameters corresponding to the candidate phonetic symbol sequence, determine the lip shape parameters that match the target phonetic symbol sequence, wherein the lip shape parameters corresponding to the reference phonetic symbol sequence are used to characterize the lip shape parameters corresponding to the phonetic symbols at the preset positions in the reference phonetic symbol sequence; Based on the lip shape parameters corresponding to the target phonetic symbol, the lip shape of the target virtual image is rendered.

2. The method according to claim 1, characterized in that, The method further includes: If there exists a reference phonetic sequence in the correspondence that completely matches the target phonetic sequence, then the lip shape parameter corresponding to the completely matching reference phonetic sequence is determined as the lip shape parameter that matches the target phonetic sequence.

3. The method according to claim 1, characterized in that, The step of determining the lip shape parameters that match the target phonetic sequence based on the lip shape parameters corresponding to the candidate phonetic sequence includes: If there are multiple candidate phonetic symbols, the mean or median of the lip shape parameters corresponding to each candidate phonetic symbol sequence is determined as the lip shape parameters that match the target phonetic symbol sequence. If there is only one candidate phonetic sequence, then the lip shape parameter corresponding to the candidate phonetic sequence is determined to be the lip shape parameter that matches the target phonetic sequence.

4. The method according to claim 1, characterized in that, The correspondence is constructed in the following way: Acquire multiple facial videos; For each facial video, lip shape parameters are extracted from each video frame of the facial video; a sample IPA sequence corresponding to the audio in the facial video and the sample pronunciation duration of each sample phonetic symbol in the sample IPA sequence are determined; based on the sample IPA sequence and the pronunciation duration of each sample phonetic symbol, a video segment corresponding to each sample phonetic symbol is determined from the facial video; for each sample phonetic symbol, an N-gram phonetic symbol string containing the sample phonetic symbol is extracted from the sample IPA sequence to obtain a reference phonetic symbol sequence; the parameter sequence composed of the lip shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol is determined as the first candidate parameter sequence corresponding to the reference phonetic symbol sequence; For each of the reference phonetic symbols, the lip shape parameters corresponding to the reference phonetic symbol sequence are determined based on at least one first candidate parameter sequence corresponding to the reference phonetic symbol sequence.

5. The method according to claim 4, characterized in that, The step of determining the lip shape parameters corresponding to the reference phonetic sequence based on at least one first candidate parameter sequence includes: For each of the first candidate parameter sequences corresponding to the reference phonetic symbol sequence, a reference lip shape parameter is determined from the first candidate parameter sequence; The mean or median of each of the reference lip shape parameters is determined as the lip shape parameter corresponding to the reference phonetic sequence.

6. The method according to claim 5, characterized in that, Determining the reference lip shape parameters from the first candidate parameter sequence includes: Calculate the absolute value of the difference between adjacent lip shape parameters in the first candidate parameter sequence to obtain the gradient sequence; The lip shape parameter at the Nth position in the first candidate parameter sequence is determined as the reference lip shape parameter, wherein the maximum value in the gradient sequence is located at the Nth position of the gradient sequence.

7. The method according to claim 5, characterized in that, Before the step of determining the mean or median of each of the reference lip shape parameters as the lip shape parameter corresponding to the reference phonetic sequence, the step of determining the lip shape parameter corresponding to the reference phonetic sequence based on at least one first candidate parameter sequence corresponding to the reference phonetic sequence further includes: Remove outlier data from all the reference lip shape parameters corresponding to the reference phonetic sequence; The step of determining the mean or median of each of the reference lip shape parameters as the lip shape parameter corresponding to the reference phonetic sequence includes: The mean or median of the reference lip shape parameters obtained after removing outlier data are determined as the lip shape parameters corresponding to the reference phonetic sequence.

8. The method according to any one of claims 4-7, characterized in that, The method for constructing the correspondence also includes the following steps: Let k=1; For each of the sample phonetic symbols, the (Nk)-gram phonetic symbol string containing the sample phonetic symbol is extracted from the sample IPA sequence to obtain the sample phonetic symbol sequence; the parameter sequence composed of the lip shape parameters of each video frame in the video segment corresponding to the sample phonetic symbol is determined as the second candidate parameter sequence corresponding to the sample phonetic symbol sequence. For each sample phonetic symbol sequence, the lip shape parameter corresponding to the sample phonetic symbol sequence is determined based on at least one second candidate parameter sequence corresponding to the sample phonetic symbol sequence; the sample phonetic symbol sequence is expanded into an N-gram phonetic symbol string by adding preset characters to the sample phonetic symbol sequence to obtain a reference phonetic symbol sequence; the lip shape parameter corresponding to the sample phonetic symbol sequence is determined as the lip shape parameter corresponding to the expanded reference phonetic symbol sequence. Increment k by 1 and return to the step of extracting the (Nk)-gram phonetic string containing the phonetic symbol of the sample from the sample IPA sequence until Nk=1.

9. The method according to any one of claims 1-7, characterized in that, The step of rendering the lip shape of the target virtual image based on the lip shape parameters corresponding to the target phonetic symbol includes: Based on the lip shape parameters corresponding to the neighboring lip shapes of the target lip shape, the lip shape parameters corresponding to the target lip shape are filtered. The neighboring lip shape is the lip shape in the extended sequence that is less than a preset distance threshold from the target lip shape. The lip shape parameters corresponding to the target phonetic symbol and obtained after filtering are input into the preset rendering engine to render the lip shape of the target virtual image.

10. A virtual avatar lip-syncing device, characterized in that, include: The first determining module is used to determine the target IPA sequence corresponding to the target text and the target pronunciation duration of each phonetic symbol in the target IPA sequence; The first extension module is used to extend the duration of the target IPA sequence according to the duration of each target pronunciation to obtain an extended sequence; The first extraction module extracts an N-gram phonetic string containing the target phonetic symbol from the extended sequence for each target phonetic symbol in the extended sequence to obtain a target phonetic symbol sequence, wherein N≥2 and the target phonetic symbol is located at a preset position in the target phonetic symbol sequence; The second determining module is used to determine, when there is no reference phonetic sequence that completely matches the target phonetic sequence in the pre-established correspondence between reference phonetic sequences and lip shape parameters, a candidate phonetic sequence with the highest matching degree with the target phonetic sequence from the reference phonetic sequence containing preset characters, wherein the preset characters are not International Phonetic Alphabet symbols and are located in positions other than the preset position in their respective reference phonetic sequences, and the phonetic symbols in the candidate phonetic sequence other than the position of the preset character are the same as the phonetic symbols in the corresponding positions in the target phonetic sequence; and to determine the lip shape parameters that match the target phonetic sequence based on the lip shape parameters corresponding to the candidate phonetic sequences, wherein the lip shape parameters corresponding to the reference phonetic sequences are used to characterize the lip shape parameters corresponding to the phonetic symbols in the reference phonetic sequences at the preset positions; The rendering module is used to render the lip shape of the target virtual image based on the lip shape parameters corresponding to the target phonetic symbol.

11. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by the processing device, the program implements the steps of the method according to any one of claims 1-9.

12. An electronic device, characterized in that, include: A storage device having at least one computer program stored thereon; At least one processing means is configured to execute the at least one computer program in the storage device to implement the steps of the method according to any one of claims 1-9.

Images