Text-based virtual object lip shape synchronization method and system

Abstract

The application discloses a kind of virtual object lip shape synchronization method and system based on text, method includes: obtaining target text;The target text is converted to obtain target voice, and the target text is converted to obtain orthography transcription;The target voice is labeled and aligned with the orthography transcription, to obtain the phoneme sequence corresponding to the target voice;According to the phoneme sequence, audiovisual position mapping is carried out, to obtain the visual position sequence corresponding to the phoneme sequence mapping, and the visual position parameter of the visual position sequence is determined;According to the visual position parameter, the rendering of target virtual object is controlled, and voice is played synchronously.The application can combine natural language understanding field and computer animation technology field, can play lip shape synchronization animation in extremely short time after inputting text, and can be widely applied in computer technology field.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a text-based virtual object lip synchronization method and system. Background Art

[0002] Virtual human technology is an emerging field spanning artificial intelligence, computer graphics, and computer vision. It utilizes the collaborative application of multiple technologies, including speech synthesis, speech recognition, semantic understanding, and dialogue generation, to power digitally constructed virtual avatars. Virtual humans are already common in film and television special effects, intelligent customer service, advertising, and marketing, and have broad application prospects.

[0003] One of the key sources of high realism for virtual humans is their ability to maintain human-like lip syncing when speaking (playing audio). Although facial capture technology combined with manual adjustments by animators can produce very natural and realistic virtual human animations, this requires expensive hardware, actors, and professional animators, and is also time-consuming. Therefore, a programmatic method to drive virtual human lip syncing is needed. Although there has been a lot of research on this, many methods require simultaneous audio and text input, and the problem of generating sufficiently human-like performance at an acceptable speed has not yet been solved. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides an efficient and accurate text-based virtual object lip synchronization method and system, which can combine the fields of natural language understanding and computer animation technology, and can play lip synchronization animation in a very short time after the text is input.

[0005] An aspect of an embodiment of the present invention provides a text-based virtual object lip synchronization method, comprising:

[0006] Get the target text;

[0007] Converting the target text into a target speech, and converting the target text into an orthographic transcription;

[0008] Aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech;

[0009] Performing phoneme-viseme mapping according to the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determining viseme parameters of the viseme sequence;

[0010] The rendering of the target virtual object is controlled according to the viseme parameters, and the voice is played synchronously.

[0011] Optionally, the step of aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech includes:

[0012] voice segmenting the target speech to determine voiced segment portions in the target speech;

[0013] filling the orthographic transcription into the voiced segment portions;

[0014] normalizing and text phonetizing the text of the voiced segment portions;

[0015] generating grammar constraints to enforce alignment of phonemes and audio to obtain phoneme sequences and durations of the phoneme sequences corresponding to the speech.

[0016] Optionally, the voice segmenting the target speech to determine voiced segment portions in the target speech comprises:

[0017] dividing windows in the audio signal according to fixed durations, and estimating root mean square of each window;

[0018] determining a target threshold according to statistical distribution of the values of the root mean square;

[0019] evaluating each window according to the target threshold, and labeling windows with root mean square values lower than the target threshold as silence, and labeling windows with root mean square values higher than the target threshold as voiced;

[0020] combining adjacent silence windows as silence segments, and combining adjacent voiced windows as voiced segments;

[0021] when a silence segment has a duration less than a first threshold, combining the silence segment with adjacent voiced segments to obtain a new voiced segment;

[0022] when a voiced segment has a duration less than the first threshold, combining the voiced segment with adjacent silence segments to obtain a new silence segment.

[0023] Optionally, the filling the orthographic transcription into the voiced segment portions comprises:

[0024] using a "#" symbol or a line break as a minimum silence pause symbol of the orthographic transcription;

[0025] filling the orthographic transcription into the voiced segment portions according to the orthographic transcription segmented by the silence pause symbol;

[0026] when the number of voiced portions in the orthographic transcription and the number of voiced portions segmented from the speech do not match, adjusting a volume threshold, a minimum duration of silence or voiced portions to obtain a matched number of voiced portions.

[0027] Optionally, the normalizing and text phonetizing the text of the voiced segment portions comprises:

[0028] The numbers, special symbols and other languages in orthographic transcription are processed separately, and are replaced by a standard input through a replacement dictionary;

[0029] The remaining part is processed by a user-provided language dictionary and a forward maximum matching algorithm;

[0030] The obtained individual word pieces are phonemized through a dictionary.

[0031] Optionally, the audio-visual position mapping according to the phoneme sequence, obtaining a visual position sequence corresponding to the phoneme sequence, and determining a visual position parameter of the visual position sequence, comprise:

[0032] Recording the mouth shape parameters of the virtual object corresponding to each phoneme, obtaining a visual position table corresponding to a phoneme table;

[0033] According to the visual position table, the parameters corresponding to the visual position are smoothly mixed on the time axis through the visual position sequence corresponding to the phoneme sequence.

[0034] Optionally, the orthographic transcription refers to a language corresponding to a phonetic, a text containing only normal use text and system defined symbols;

[0035] The minimum requirement of the orthographic transcription is a symbol representing a silence segment with a correct number and a correct position;

[0036] The orthographic transcription is obtained by using a rule-based algorithm to process a text with a comma, a period, a question mark, a semicolon, an exclamation mark or a bracket as a punctuation standard.

[0037] Another aspect of the embodiment of the present application also provides a virtual object lip shape synchronization system based on text, comprising:

[0038] A first module is configured to obtain a target text;

[0039] A second module is configured to convert the target text to obtain a target phonetic and convert the target text to obtain an orthographic transcription;

[0040] A third module is configured to align the target phonetic and the orthographic transcription to obtain a phoneme sequence corresponding to the target phonetic;

[0041] A fourth module is configured to perform audio-visual position mapping according to the phoneme sequence, obtain a visual position sequence corresponding to the phoneme sequence, and determine a visual position parameter of the visual position sequence;

[0042] A fifth module is configured to control the rendering of a target virtual object according to the visual position parameter, and synchronously play the phonetic.

[0043] Another aspect of the embodiment of the present application further provides an electronic device, comprising a processor and a memory;

[0044] The memory is used for storing a program.

[0045] The processor executes the program to realize the method as described above.

[0046] Another aspect of the embodiment of the present application further provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the method as described above.

[0047] The embodiment of the present application further discloses a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method as described above.

[0048] The embodiment of the present application acquires a target text; converts the target text to obtain a target voice and a orthography transcription; labels and aligns the target voice and the orthography transcription to obtain a phoneme sequence corresponding to the target voice; performs audio-visual position mapping according to the phoneme sequence to obtain a visual position sequence corresponding to the phoneme sequence, and determines a visual position parameter of the visual position sequence; controls rendering of a target virtual object according to the visual position parameter, and synchronously plays the voice. The present application can combine the natural language understanding field and the computer animation technology field, and can play the lip shape synchronous animation in a very short time after inputting the text. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0050] Figure 1 The overall step flowchart provided by the embodiment of the present application is provided.

[0051] Figure 2 The flowchart of automatic voice labeling and alignment of the embodiment of the present application is provided. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0053] In view of the problems existing in the prior art, an embodiment of the present invention provides a text-based virtual object lip synchronization method, comprising:

[0054] Get the target text;

[0055] Converting the target text into a target speech, and converting the target text into an orthographic transcription;

[0056] Aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech;

[0057] Performing phoneme-viseme mapping according to the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determining viseme parameters of the viseme sequence;

[0058] The rendering of the target virtual object is controlled according to the viseme parameters, and the voice is played synchronously.

[0059] Optionally, the step of aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech includes:

[0060] Segmenting the target speech into voiced segments to determine the voiced segments in the target speech;

[0061] Filling the orthographic transcription into the voiced segment;

[0062] Normalize and phonemize the text of the audio segment;

[0063] Generate grammatical constraints, force the alignment of phonemes and audio, and obtain the phoneme sequence corresponding to the speech and the duration of the phoneme sequence.

[0064] Optionally, the performing voiced segmentation on the target speech to determine the voiced segment portion of the target speech includes:

[0065] In the audio signal, the window is divided into fixed durations and the RMS value of each window is estimated;

[0066] Determining a target threshold value based on the statistical distribution of the root mean square value;

[0067] Evaluate each window according to the target threshold, labeling windows with RMS values ​​below the target threshold as silent, and labeling windows with RMS values ​​above the target threshold as vocal;

[0068] Combine adjacent silent windows into silent segments, and combine adjacent sounding windows into sounding segments;

[0069] When the duration of a silent segment is less than a first threshold, the silent segment is combined with an adjacent vocal segment to obtain a new vocal segment;

[0070] When the duration of a sound segment is less than a first threshold, the sound segment is combined with an adjacent silent segment to obtain a new silent segment.

[0071] Optionally, filling the orthographic transcription into the voiced segment portion comprises:

[0072] Use the "#" symbol or line break as a minimum silent pause in orthographic transcription;

[0073] Filling the orthographic transcription into the generated voiced part according to the orthographic transcription segmented by the silence pause symbol;

[0074] When the number of voiced parts in the orthographic transcription and the number of voiced parts segmented from the speech do not match, the volume threshold, silence, or minimum duration of the voiced parts are adjusted to obtain a matching number of voiced parts.

[0075] Optionally, the normalization and phoneme conversion of the text of the voiced segment includes:

[0076] Separate word segmentation for numbers, special symbols, and other languages ​​in orthographic transcriptions and use them as standard input by replacing dictionary specifications;

[0077] The remaining part is segmented using the language dictionary given by the user and the forward maximum matching algorithm;

[0078] Each word segment obtained is phonemized through a dictionary.

[0079] Optionally, performing phoneme-viseme mapping according to the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determining viseme parameters of the viseme sequence includes:

[0080] Recording the lip shape parameters of the virtual object corresponding to each phoneme to obtain a viseme table corresponding to the phoneme table;

[0081] According to the viseme table, the corresponding viseme sequence is mapped by the phoneme sequence, and the parameters corresponding to the visemes are smoothly mixed on the time axis.

[0082] Optionally, the orthographic transcription refers to a text corresponding to the language and corresponding to the phonetics, containing only formally used text and system-defined symbols;

[0083] The minimum requirement for the orthographic transcription is that it contains the correct number and placement of symbols representing silence segments;

[0084] The orthographic transcription is obtained by processing the text using a rule-based algorithm using commas, periods, question marks, semicolons, exclamation marks or brackets as sentence break criteria.

[0085] Another aspect of the present invention provides a text-based virtual object lip synchronization system, comprising:

[0086] The first module is used to obtain the target text;

[0087] A second module is configured to convert the target text into a target speech and convert the target text into an orthographic transcription;

[0088] A third module is configured to align the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech;

[0089] A fourth module is configured to perform phoneme-viseme mapping according to the phoneme sequence, obtain a viseme sequence corresponding to the phoneme sequence mapping, and determine viseme parameters of the viseme sequence;

[0090] The fifth module is used to control the rendering of the target virtual object according to the viseme parameters and synchronously play the voice.

[0091] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0092] The memory is used to store programs;

[0093] The processor executes the program to implement the method described above.

[0094] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0095] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0096] The specific implementation process of the present invention is described in detail below with reference to the accompanying drawings:

[0097] like Figure 1As shown, a text-based virtual human lip synchronization method of the present application includes the following steps:

[0098] Step 1: Text-to-speech, using text-to-speech technology to obtain the corresponding speech of the text.

[0099] Step 2: Text processing, processing the text to obtain orthographic transcription.

[0100] Step 3: Automatic speech annotation alignment, forcibly aligning the speech and orthographic transcription to obtain the phoneme sequence and its duration corresponding to the speech.

[0101] Step 4: Audio-visual mapping, mapping the phoneme sequence to its corresponding visual sequence, and smoothing the parameters corresponding to the visual sequence on the time axis.

[0102] Step 5: Virtual human animation generation, controlling the virtual human rendering by sending visual parameters in real time, and playing the speech at the same time to achieve lip synchronization effect.

[0103] Specifically, in the above-mentioned third step, the following steps (1)-(5) are included:

[0104] (1) voiced segment segmentation

[0105] First, in the audio signal, windows are divided according to fixed duration (default 20ms, which can be configured by the user), and the root-mean-square (RMS) value of each window is estimated. The root-mean-square value, also known as the effective value, is the result of taking the square of the signal waveform and then taking the square root of the average.

[0106] Then, the threshold Θ is determined by the statistical distribution of the root-mean-square value, as shown in formula 1:

[0107] Θ = min + μ - δ (1)

[0108] Where min is the minimum value of the root-mean-square values, μ is the average value, and δ is fixed by default as 1.5σ according to experience, and σ represents the coefficient of variation, which is a normalized measure of the degree of dispersion of the probability distribution, defined as the ratio of the standard deviation to the average value.

[0109] Then, using the threshold Θ to evaluate each window, the window whose root-mean-square value is lower or higher than the threshold is marked as silence or voiced. Then, combining consecutive silence windows or voiced windows, the entire audio is divided into silence segments or voiced segments.

[0110] If the duration of a silent segment is too short (less than 200ms by default), it will be deleted and then combined with the adjacent sound segment to form a new sound segment. After that, the sound segment that is too short (less than 300ms by default) is removed and combined with the adjacent silent segment to form a new silent segment.

[0111] (2) Fill in the spoken part with the orthographic transcription

[0112] Orthographic transcriptions must, at a minimum, include silent pauses, represented by "#" or line breaks. For orthographic transcriptions segmented by these symbols, the system attempts to fit them into the voiced parts generated in the previous step. If the number of voiced parts in the orthographic transcription does not match the number of voiced parts segmented from the speech, this step will automatically attempt to adjust parameters such as volume thresholds, silence, or minimum duration of voiced parts to achieve a matching number of voiced parts.

[0113] (3) Text normalization

[0114] First, the numbers, special symbols, and other languages ​​in the orthographic transcription are segmented separately and standardized by the replacement dictionary as standard input, such as ℃ will be replaced by Celsius, and 10001 will be replaced by ten thousand and one.

[0115] Then the remaining words are segmented using the language dictionary given by the user and the forward maximum matching algorithm. The steps of the forward maximum matching algorithm are:

[0116] 1. Take the unsegmented part from left to right.

[0117] 2. Look up the dictionary for a match.

[0118] 3. If the match is successful, split the part into a word.

[0119] 4. If the match is unsuccessful, remove the last word of the part and match again.

[0120] 5. Repeat the above steps until the entire sentence is segmented.

[0121] (4) Text phonemeization

[0122] The word segmentation is converted to phonemes using a dictionary, and an inference algorithm is used for unknown words, such as proper nouns, abbreviations, or emerging vocabulary. The main idea of ​​this algorithm is to infer the phonemes of an unknown word through pure analogy given a sufficient number of examples. This process involves exploring the word from left to right and right to left, looking for a match with the longest string in the dictionary. For details, refer to the previous step. The inferred phonemes of the unknown word can be manually adjusted by the user or added directly to the dictionary.

[0123] (5) Automatically generate grammatical constraints and use acoustic models to force the alignment of phonemes and audio to obtain the phoneme sequence and its duration corresponding to the speech.

[0124] Specifically, the specific implementation process of the fourth step includes the following steps (1)-(2):

[0125] (1) Prepare the viseme

[0126] Visemes are the visual representation of phonemes. The lip shape parameters of the virtual person corresponding to each phoneme are recorded to obtain a viseme table corresponding to the phoneme table.

[0127] (2) Mapping the phoneme sequence to its corresponding viseme sequence and smoothly mixing the parameters corresponding to the visemes on the time axis.

[0128] It should be noted that the orthographic transcription in the second step refers to text that corresponds to the language and phonetics and contains only formally used text and system-defined symbols. The minimum requirement for an orthographic transcription is the correct number and placement of symbols representing silence. This system's orthographic transcription is generated by processing the text using a rule-based algorithm, using commas, periods, question marks, semicolons, exclamation points, parentheses, and other text as punctuation criteria.

[0129] Furthermore, the text normalization and phonemeization in the third step require the preparation of a dictionary for the corresponding language, containing common words and their corresponding phonemes (including pronunciation variants). The forced alignment in the third step also requires the preparation of a trained acoustic model for the corresponding language. This system uses a Hidden Markov Model (HMM) trained using HTK.

[0130] The real-time link in step 5 refers to linking to the Unreal Engine via the Live Link plugin. The virtual human driven in step 5 refers to MetaHuman in Epic's Unreal Engine.

[0131] In summary, the present invention first acquires a target text; converts the target text into a target speech, and then converts the target text into an orthographic transcription; aligns the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech; performs a phoneme mapping based on the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determines viseme parameters of the viseme sequence; controls the rendering of a target virtual object based on the viseme parameters, and synchronously plays the speech. The present invention combines the fields of natural language understanding and computer animation technology, enabling lip-sync animation to be played very quickly after text input.

[0132] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0133] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0135] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0136] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0137] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0138] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0139] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0140] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A text-based virtual object lip synchronization method, characterized in that: include: Get the target text; Converting the target text into a target speech, and converting the target text into an orthographic transcription; Aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech includes: Segmenting the target speech into voiced segments to determine the voiced segments in the target speech includes: In the audio signal, the window is divided into fixed durations and the RMS value of each window is estimated; Determining a target threshold value based on the statistical distribution of the root mean square value; The formula used to determine the target threshold includes: Θ=min+μ-δ; Wherein, Θ is the target threshold, min is the minimum value of the root mean square value, μ is the mean value, δ is 1.5σ, and σ represents the coefficient of variation, which is a normalized measure of the degree of dispersion of the probability distribution and is defined as the ratio of the standard deviation to the mean value; Filling the orthographic transcription into the voiced segment; Normalize and phonetize the text of the voiced segment, including: Separate word segmentation for numbers, special symbols, and other languages ​​in orthographic transcriptions and use them as standard input by replacing dictionary specifications; The remaining part is segmented using the language dictionary given by the user and the forward maximum matching algorithm; Phonemize each segmented word obtained through the dictionary; The word segmentation is converted into phonemes through the dictionary, and the inference algorithm is used for unknown words; Performing phoneme-viseme mapping according to the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determining viseme parameters of the viseme sequence; The rendering of the target virtual object is controlled according to the viseme parameters, and the voice is played synchronously.

2. The method for lip synchronization of a text-based virtual object according to claim 1, wherein: The step of annotating and aligning the target speech with the orthographic transcription to obtain a phoneme sequence corresponding to the target speech includes: Generate grammatical constraints, force the alignment of phonemes and audio, and obtain the phoneme sequence corresponding to the speech and the duration of the phoneme sequence.

3. The method for lip synchronization of a text-based virtual object according to claim 1, wherein: The step of segmenting the target speech into voiced segments and determining the voiced segments in the target speech includes: Evaluate each window according to the target threshold, labeling windows with RMS values ​​below the target threshold as silent, and labeling windows with RMS values ​​above the target threshold as vocal; Combine adjacent silent windows into silent segments, and combine adjacent sounding windows into sounding segments; When the duration of a silent segment is less than a first threshold, the silent segment is combined with an adjacent vocal segment to obtain a new vocal segment; When the duration of a sound segment is less than a first threshold, the sound segment is combined with an adjacent silent segment to obtain a new silent segment.

4. The method for lip synchronization of a text-based virtual object according to claim 1, wherein: Filling the orthographic transcription into the voiced segment portion includes: Use the "#" symbol or line break as a minimum silent pause in orthographic transcription; Filling the orthographic transcription into the generated voiced part according to the orthographic transcription segmented by the silence pause symbol; When the number of voiced parts in the orthographic transcription and the number of voiced parts segmented from the speech do not match, the volume threshold, silence, or minimum duration of the voiced parts are adjusted to obtain a matching number of voiced parts.

5. The method for lip synchronization of a text-based virtual object according to claim 1, wherein: The performing of phoneme-viseme mapping according to the phoneme sequence to obtain a viseme sequence corresponding to the phoneme sequence mapping, and determining viseme parameters of the viseme sequence, includes: Recording the lip shape parameters of the virtual object corresponding to each phoneme to obtain a viseme table corresponding to the phoneme table; According to the viseme table, the corresponding viseme sequence is mapped by the phoneme sequence, and the parameters corresponding to the visemes are smoothly mixed on the time axis.

6. A text-based virtual object lip synchronization method according to any one of claims 1 to 5, characterized in that: The orthographic transcription is a text that corresponds to the language and the phonetics and contains only formally used text and system-defined symbols; The minimum requirement for the orthographic transcription is that it contains the correct number and placement of symbols representing silence segments; The orthographic transcription is obtained by processing the text using a rule-based algorithm using commas, periods, question marks, semicolons, exclamation marks or brackets as sentence break criteria.

7. A text-based virtual object lip synchronization system, used to implement the text-based virtual object lip synchronization method according to any one of claims 1 to 6, characterized in that: include: The first module is used to obtain the target text; A second module is configured to convert the target text into a target speech and convert the target text into an orthographic transcription; The third module is configured to perform annotation alignment on the target speech and the orthographic transcription to obtain a phoneme sequence corresponding to the target speech, including: Segmenting the target speech into voiced segments to determine the voiced segments in the target speech includes: In the audio signal, the window is divided into fixed durations and the RMS value of each window is estimated; Determining a target threshold value based on the statistical distribution of the root mean square value; The formula used to determine the target threshold includes: Θ=min+μ-δ; Wherein, Θ is the target threshold, min is the minimum value of the root mean square value, μ is the mean value, δ is 1.5σ, and σ represents the coefficient of variation, which is a normalized measure of the degree of dispersion of the probability distribution and is defined as the ratio of the standard deviation to the mean value; A fourth module is configured to perform phoneme-viseme mapping according to the phoneme sequence, obtain a viseme sequence corresponding to the phoneme sequence mapping, and determine viseme parameters of the viseme sequence; The fifth module is used to control the rendering of the target virtual object according to the viseme parameters and synchronously play the voice.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images