Digital human display method, device, equipment and storage medium
By determining the lip shaped sequence corresponding to the phoneme sequence in the digital human display method and adding intermediate lip shaped, the problem of the mismatch between the digital lip shaped picture and the real person is solved, and a more natural lip shaped transition and a better user experience is achieved.
Patent Information
- Application Number
- CN202311528663.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-11-16
AI Technical Summary
When digital people interact with users, the lip-shaped screen does not match the lip-shaped screen of the live speech, resulting in unnatural screen switching and affecting the user experience.
By obtaining the audio and phoneme data corresponding to the text interaction information, determine the lip sequence corresponding to the phoneme sequence, and when outputting the audio, the digital person displays multiple intermediate lip patterns to reduce the gap between adjacent lip pictures, and realize Natural transition.
It improves the naturalness of the lip-shaped picture displayed by digital people, makes it closer to real-person expressions, and improves user interaction experience.
Smart Images

Figure CN117710538B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a digital human display method, device, equipment and storage medium. Background Art
[0002] Digital Human is a digital human image close to human image created by digital technology. In application scenarios such as e-commerce live broadcast, game explanation, and intelligent customer service, digital human can be used to replace real people to interact with users. In actual application, digital human makes users feel like they are interacting with real people by showing realistic appearance, expressions, lip shape, body language, etc. and broadcasting corresponding voice.
[0003] However, in the process of using digital humans to interact with users, it is relatively difficult to make the lip shape images displayed by the digital humans realistic and close to the lip shape changes of real people speaking. It is often easy for the lip shape images to not match the lip shape of real people speaking, such as: the switching between different lip shape images is unnatural, and the lip shape images do not match the output voice content, etc. This will affect the display effect of the digital humans and bring a bad interactive experience to users. Summary of the invention
[0004] The embodiments of the present invention provide a digital human display method, device, equipment and storage medium, which are used to improve the display effect of the digital human and make the displayed lip-sync images closer to real-person expressions.
[0005] In a first aspect, an embodiment of the present invention provides a digital human display method, the method comprising:
[0006] Acquire audio and phoneme data corresponding to the text interaction information to be output, wherein the phoneme data includes a phoneme sequence;
[0007] Determining the lip shape sequence corresponding to the phoneme sequence according to the preset correspondence relationship between the phonemes and the lip shapes;
[0008] For any two adjacent first and second lip shapes in the lip shape sequence, determining a plurality of first intermediate lip shapes for transitioning from the first lip shape to the second lip shape according to lip shape state parameters respectively corresponding to the first lip shape and the second lip shape;
[0009] When the audio is output, the digital human is controlled to display a lip-sync picture that matches the lip-sync sequence and the plurality of first intermediate lip-syncs.
[0010] In a second aspect, an embodiment of the present invention provides a digital human display device, the device comprising:
[0011] An acquisition module, used to acquire audio and phoneme data corresponding to the text interaction information to be output, wherein the phoneme data includes a phoneme sequence;
[0012] A processing module, configured to determine a mouth shape sequence corresponding to a phoneme sequence according to a preset correspondence relationship between phonemes and mouth shapes; and for any two adjacent first mouth shapes and second mouth shapes in the mouth shape sequence, determine a plurality of first intermediate mouth shapes for transitioning from the first mouth shape to the second mouth shape according to mouth shape state parameters corresponding to the first mouth shape and the second mouth shape respectively;
[0013] The display module is used to control the digital human to display the lip shape pictures matching the lip shape sequence and the multiple first intermediate lip shapes when outputting the audio.
[0014] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor can at least implement the digital human display method as described in the first aspect.
[0015] In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor can at least implement the digital human display method as described in the first aspect.
[0016] In the solution provided by the embodiment of the present invention, the digital human is used to simulate a real person to perform human-computer interaction with a user. When responding to the interactive information sent by the user, the interactive information can be output in the form of voice, and the digital human can be synchronously controlled to display an expression or action that matches the content of the interactive information. This solution is specifically used to realize the control of the lip shape picture displayed by the digital human. In the specific implementation process, first, the audio and phoneme data corresponding to the text interactive information to be output are obtained, wherein the phoneme data includes a phoneme sequence; then, according to the preset correspondence between the phoneme and the lip shape, the lip shape sequence corresponding to the phoneme sequence is determined; then, for any two adjacent first lip shapes and second lip shapes in the lip shape sequence, according to the lip shape state parameters corresponding to the first lip shape and the second lip shape respectively, multiple first intermediate lip shapes that transition from the first lip shape to the second lip shape are determined; finally, when outputting the audio, the digital human is controlled to display a lip shape picture that matches the lip shape sequence and the multiple first intermediate lip shapes. In this solution, by determining multiple first intermediate lip shapes between two adjacent lip shapes in the lip shape sequence, it is achieved to add multiple intermediate lip shape pictures between the lip shape pictures corresponding to two adjacent lip shapes in the lip shape sequence. Therefore, when controlling the digital human to display the lip shape pictures, the gap between the two adjacent lip shape pictures is smaller, avoiding the situation of abrupt jump from one lip shape picture to another, and the transition between the lip shape pictures is more natural and closer to real-life expression. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0018] Figure 1 A schematic diagram of a digital population screen switching provided by an embodiment of the present invention;
[0019] Figure 2 A flowchart of a digital human display method provided by an embodiment of the present invention;
[0020] Figure 3 A schematic diagram of another digital population screen switching provided by an embodiment of the present invention;
[0021] Figure 4 A flowchart of another digital human display method provided by an embodiment of the present invention;
[0022] Figure 5 A flowchart of an interaction method provided by an embodiment of the present invention;
[0023] Figure 6A schematic diagram of a training process of a target language model provided by an embodiment of the present invention;
[0024] Figure 7 A structural schematic diagram of a digital human display device provided by an embodiment of the present invention;
[0025] Figure 8 For Figure 7 A schematic diagram of the structure of an electronic device corresponding to the digital human display device provided in the illustrated embodiment. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0027] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0028] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.
[0029] Digital Human is a digital human image close to human image created by digital technology. In application scenarios such as e-commerce live broadcast, game explanation, and intelligent customer service, digital human can be used to replace real people to interact with users. In actual application, digital human makes users feel like they are interacting with real people by showing realistic appearance, expression, mouth shape, body shape, etc. and broadcasting corresponding voice.
[0030] However, in the process of using digital humans to interact with users, it is difficult to make the lip movements displayed by the digital humans realistic and close to the lip movements of real people. It is often easy for the lip movements to not match the lip movements of real people, such as abrupt and unnatural switching between different lip movements.
[0031] For example, Figure 1 A schematic diagram of a digital population screen switching provided by an embodiment of the present invention, Figure 1The left and right pictures in the video are two adjacent lip-syncing images when displayed. Figure 1 In the lip-syncing image on the left, the digital human's lips are closed. Figure 1 In the lip-syncing image on the right of FIG, the digital human's lips are open to the extent that the teeth are exposed. It is understandable that if the digital human is directly Figure 1 The lip-syncing screen in the middle left picture switches to Figure 1 The lip-syncing screen in the middle right picture will make the user clearly feel the switching between the lip-syncing screens, and the switching is not smooth, which affects the user experience.
[0032] In order to solve at least one of the above technical problems, an embodiment of the present invention provides a digital human display method, by adding multiple intermediate lip shape pictures between the lip shape pictures corresponding to two adjacent lip shapes in a determined lip shape sequence, so that when controlling the digital human to display the lip shape pictures, the gap between the two adjacent lip shape pictures is smaller, avoiding the abrupt jump from one lip shape picture to another, and the transition between the lip shape pictures is more natural and closer to real-life expression.
[0033] The digital human display method provided in the embodiment of the present invention can be executed by an electronic device, which can be a terminal device such as a PC, a laptop, a smart phone, etc., or a server. The server can be a physical server including an independent host, or a virtual server, or a cloud server or a server cluster.
[0034] Figure 2 A flowchart of a digital human display method provided by an embodiment of the present invention is shown in FIG. Figure 2 As shown, the following steps may be included:
[0035] 201. Acquire audio and phoneme data corresponding to text interaction information to be output, where the phoneme data includes a phoneme sequence.
[0036] 202. Determine a lip shape sequence corresponding to the phoneme sequence according to a preset correspondence relationship between phonemes and lip shapes.
[0037] 203. For any two adjacent first lip shapes and second lip shapes in the lip shape sequence, determine a plurality of first intermediate lip shapes for transitioning from the first lip shape to the second lip shape according to lip shape state parameters respectively corresponding to the first lip shape and the second lip shape.
[0038] 204. When outputting the audio, control the digital human to display a lip shape picture that matches the lip shape sequence and the plurality of first intermediate lip shapes.
[0039] In the process of human-computer interaction, there are various ways to display interactive information to users, such as transmitting interactive information used to respond to user operations to users through text, voice, video, etc. In this solution, interactive information is transmitted to users through digital humans. Specifically, the interactive information is output in the form of audio and the digital humans are synchronously controlled to display corresponding images, wherein the digital human's lip-sync images match the audio.
[0040] In this embodiment, after the text interaction information to be output is obtained, the text interaction information to be output is not directly output to the user, but the audio corresponding to the text interaction information to be output is obtained through a technology such as text-to-speech conversion, and the phoneme data in the audio is extracted. The phoneme data includes a phoneme sequence and the duration of each phoneme in the phoneme sequence, and the phonemes in the phoneme sequence and the audio are arranged in chronological order from early to late.
[0041] Among them, phonemes are the smallest speech units divided according to the natural properties of speech, and phoneme data is used to control the display of the digital human's lip shape screen. In general, a mapping relationship between "phonemes-phoneme categories-lip shapes" can be established in advance, and the lip shape sequence corresponding to the phoneme sequence can be determined based on the correspondence between phonemes and lip shapes in the mapping relationship. The lip shape sequence contains multiple lip shapes that correspond to phonemes one by one, and each lip shape has its lip shape state parameters, such as: the position coordinates of the key points of the lips, etc. The state parameters corresponding to the lip shape can be used to generate the lip shape screen corresponding to the lip shape. Therefore, after determining the lip shape sequence corresponding to the phoneme sequence, when playing the audio, the digital human can be controlled to display the lip shape screen corresponding to each lip shape in the lip shape sequence in turn.
[0042] In actual applications, different phonemes may correspond to similar lip shapes, so phonemes with similar lip shapes can be classified into the same category, and phonemes of the same category correspond to the same lip shape. Optionally, different "phoneme-phoneme category-lip shape" mapping relationships can be established according to actual needs. For example: when the vividness of the lip shape picture displayed by the digital human is required to be high, the phonemes can be divided into more phoneme categories, and then more lip shapes can be configured, so that when outputting audio, the digital human can be controlled to display more different lip shape pictures, making the picture more vivid; and when the vividness of the lip shape picture displayed by the digital human is required to be low, the phonemes can be divided into fewer phoneme categories, and then fewer lip shapes can be configured, and when outputting audio, the digital human can show that the lip shape picture switches with the audio playback.
[0043] For ease of understanding, for example, assume that the preset "phoneme-phoneme category-mouth shape" mapping relationship is: phoneme 1-category 1-mouth shape 1, phoneme 2-category 1-mouth shape 1, phoneme 3-category 2-mouth shape 2, and phoneme 4-category 3-mouth shape 3. If the phoneme sequence in the phoneme data corresponding to the text interaction information to be output is: {phoneme 1, phoneme 2, phoneme 3, phoneme 4}, then according to the correspondence between phonemes and mouth shapes in the preset "phoneme-phoneme category-mouth shape" mapping relationship, the mouth shape sequence corresponding to the phoneme sequence is determined to be {mouth shape 1, mouth shape 1, mouth shape 2, mouth shape 3}.
[0044] In this embodiment, in order to make the transition between lip shape images more natural, after determining the lip shape sequence corresponding to the phoneme sequence, further, for any two adjacent first lip shapes and second lip shapes in the lip shape sequence, multiple first intermediate lip shapes for transitioning from the first lip shape to the second lip shape are determined according to the lip shape state parameters respectively corresponding to the first lip shape and the second lip shape.
[0045] During the specific implementation process, optionally, a mouth shape state parameter change curve from the first mouth shape to the second mouth shape can be determined according to the mouth shape state parameters corresponding to the first mouth shape and the second mouth shape respectively, and then, multiple groups of mouth shape state parameters are sampled from the mouth shape state parameter change curve, wherein each group of sampled mouth shape state parameters corresponds to a mouth shape, that is, the first intermediate mouth shape.
[0046] For ease of understanding, for example, the first lip shape is lip shape 2 in the lip shape sequence {lip shape 1, lip shape 1, lip shape 2, lip shape 3}, and the second lip shape is lip shape 3. Assume that the lip shape picture corresponding to lip shape 2 is as follows: Figure 1 As shown in the left picture, the lip shape picture corresponding to lip shape 3 is as follows Figure 1 As shown in the right figure in . Assume that the coordinates of the center point of the upper lip in the lip state parameters of lip shape 2 are (x1, y1), and the coordinates of the center point of the upper lip in the lip state parameters of lip shape 3 are (x2, y2). According to (x1, y1) and (x2, y2), a straight line L passing through (x1, y1) and (x2, y2) is determined. Afterwards, two coordinate points (x3, y3) and (x4, y4) are obtained by sampling on the straight line L, where the values of x3 and x4 are between x1 and x2, and the values of y3 and y4 are between y1 and y2. (x3, y3) are the coordinates of the center point of the upper lip in the lip state parameters of the first intermediate lip shape 1, and (x4, y4) are the coordinates of the center point of the upper lip in the lip state parameters of the first intermediate lip shape 2. Similarly, other values in the lip state parameters of the first intermediate lip shape 1 and the first intermediate lip shape 2 can be sampled.
[0047] Figure 3 A schematic diagram of another digital population type screen switching provided by an embodiment of the present invention, such as Figure 3As shown, between the lip shape picture corresponding to lip shape 2 and the lip shape picture corresponding to lip shape 3, the lip shape picture corresponding to the first intermediate lip shape 1 and the lip shape picture corresponding to the first intermediate lip shape 2 are added. Therefore, when controlling the digital human to display the lip shape picture, the lip shape picture gaps between lip shape 2 and the first intermediate lip shape 1, the first intermediate lip shape 1 and the first intermediate lip shape 2, and the first intermediate lip shape 2 and lip shape 3 are all smaller than the lip shape picture gap between lip shape 2 and lip shape 3, avoiding the situation of abruptly jumping from one lip shape picture to another, and the transition between the lip shape pictures is more natural.
[0048] It should be noted that the above-mentioned sampling process of the first intermediate mouth shapes is only for illustration. In practical applications, sampling can be performed according to actual needs to determine a corresponding number of first intermediate mouth shapes.
[0049] After obtaining the middle lip shape between any two adjacent lip shapes in the lip shape sequence, the text interaction information content to be output is output in the form of playing audio. While outputting the audio, the digital human is controlled to display the lip shape pictures matching the lip shape sequence and the multiple first middle lip shapes.
[0050] In this solution, by determining multiple first intermediate lip shapes between two adjacent lip shapes in the lip shape sequence, it is achieved to add multiple intermediate lip shape pictures between the lip shape pictures corresponding to two adjacent lip shapes in the lip shape sequence. Therefore, when controlling the digital human to display the lip shape pictures, the gap between the two adjacent lip shape pictures is smaller, avoiding the situation of abrupt jump from one lip shape picture to another, and the transition between the lip shape pictures is more natural and closer to real-life expression.
[0051] In actual applications, the digital human will have an initial state, for example, the state displayed by the digital human when no human-computer interaction is performed. In the initial state, the digital human's lip shape picture is a preset lip shape picture, for example, the preset lip shape picture is the lip shape picture corresponding to the closed mouth shape. It is understandable that when the human-computer interaction starts, the digital human needs to transition from the preset lip shape picture to the lip shape picture corresponding to the first lip shape in the lip shape sequence; when the human-computer interaction ends, the digital human needs to transition from the lip shape picture corresponding to the last lip shape in the lip shape sequence to the preset lip shape picture.
[0052] In order to achieve a smooth transition of the lip shape picture when starting and ending human-computer interaction, optionally, for a target lip shape with only one adjacent lip shape in the lip shape sequence, that is, the first lip shape and / or the last lip shape in the lip shape sequence, multiple second intermediate lip shapes for transitioning from the preset lip shape to the target lip shape and / or multiple second intermediate lip shapes for transitioning from the target lip shape to the preset lip shape are determined according to the lip shape state parameters corresponding to the preset lip shape and the lip shape state parameters corresponding to the target lip shape; wherein the preset lip shape matches the preset lip shape picture displayed by the digital human when no interaction is performed.
[0053] To elaborate, assuming that the preset lip shape is a closed lip shape, if the target lip shape is the first lip shape in the lip shape sequence, then according to the lip shape state parameters corresponding to the closed lip shape and the lip shape state parameters corresponding to the first lip shape in the lip shape sequence, determine the multiple second intermediate lip shapes that transition from the closed lip shape to the first lip shape in the lip shape sequence; if the target lip shape is the last lip shape in the lip shape sequence, then according to the lip shape state parameters corresponding to the closed lip shape and the lip shape state parameters corresponding to the last lip shape in the lip shape sequence, determine the multiple second intermediate lip shapes that transition from the last lip shape in the lip shape sequence to the closed lip shape.
[0054] After determining the plurality of second intermediate lip shapes, when outputting the audio, the digital human is controlled to display a lip shape picture matching the lip shape sequence, the plurality of first intermediate lip shapes, and the second intermediate lip shapes.
[0055] The above-mentioned embodiment introduces the smoothing of the lip-syncing images of the digital human during the display process. In actual applications, whether the lip-syncing images displayed by the digital human match the currently output audio content, that is, whether the sound and image are consistent, will also affect the display effect of the digital human.
[0056] Figure 4 A flowchart of another digital human display method provided by an embodiment of the present invention, such as Figure 4 As shown, the method comprises the following steps:
[0057] 401. Acquire audio and phoneme data corresponding to text interaction information to be output, where the phoneme data includes a phoneme sequence and a duration of each phoneme in the phoneme sequence.
[0058] 402. Determine a lip shape sequence corresponding to the phoneme sequence according to a preset correspondence relationship between phonemes and lip shapes.
[0059] 403. Determine the lip shape image display time corresponding to each lip shape in the lip shape sequence according to the duration of each phoneme in the phoneme sequence.
[0060] 404. For any two adjacent first lip shapes and second lip shapes in the lip shape sequence, determine a plurality of first intermediate lip shapes for transitioning from the first lip shape to the second lip shape according to lip shape state parameters respectively corresponding to the first lip shape and the second lip shape.
[0061] 405. Determine target lip shape display times corresponding to the second lip shape and the plurality of first intermediate lip shapes, respectively, according to the lip shape display time corresponding to the second lip shape.
[0062] 406. When outputting the audio, according to the target lip shape picture display time, the digital human is controlled to display the lip shape pictures matching the lip shape sequence and the first plurality of intermediate lip shapes.
[0063] In this embodiment, the specific implementation process of step 401, step 402, and step 404 can refer to the above-mentioned embodiment and will not be described in detail here.
[0064] In this solution, the phoneme data also includes the duration of each phoneme in the phoneme sequence. Optionally, the phoneme data can be represented in the form of an array, which includes both the phoneme sequence and the duration of each phoneme. Optionally, the duration of each phoneme can be represented by the proportion of the duration of the phoneme in the overall duration corresponding to the phoneme sequence.
[0065] After obtaining the duration of each factor in the phoneme sequence, since the lip shape sequence corresponds to the phoneme sequence, the lip shape screen display time corresponding to each lip shape in the lip shape sequence can be determined according to the duration of each phoneme in the phoneme sequence. Specifically, the lip shape screen display time corresponding to each lip shape in the lip shape sequence is equal to the duration of its corresponding phoneme.
[0066] Since in this scheme, for any two adjacent first and second lip shapes in the lip shape sequence, multiple first intermediate lip shapes that transition from the first lip shape to the second lip shape are also determined. Therefore, further, it is necessary to determine the lip shape picture display time corresponding to the multiple first intermediate lip shapes. Optionally, the target lip shape picture display time corresponding to the second lip shape and the multiple first intermediate lip shapes can be determined according to the lip shape picture display time corresponding to the second lip shape. For example: there are two first intermediate lip shapes between the first lip shape and the second lip shape, and the lip shape picture display time corresponding to the second lip shape is t1 to t2, wherein the interval between t1 and t2 is 60ms. Optionally, the lip shape picture display time corresponding to the second lip shape can be evenly divided into the two first intermediate lip shapes and the second lip shape. In this case, the target lip shape display time corresponding to the two intermediate lip shapes is t1 to t1+20ms, t1+21ms to t1+40ms, and the target lip shape display time corresponding to the second lip shape is t1+41ms to t2.
[0067] Afterwards, when outputting audio, according to the target lip shape picture display time, the digital human is controlled to display the lip shape pictures matching the lip shape within the target lip shape picture display time corresponding to the lip shape sequence and the first intermediate lip shapes. For example, the lip shape picture of lip shape 1 is displayed within the target lip shape picture display time corresponding to lip shape 1.
[0068] In addition, for the first lip shape in the lip shape sequence, as described above, the second intermediate lip shape that transitions from the preset lip shape to the first lip shape can be determined based on the lip shape state parameters of the preset lip shape displayed by the digital human when no interaction is performed. Therefore, similarly, the target lip shape picture display time corresponding to the first lip shape and the multiple second intermediate lip shapes can be determined based on the lip shape picture display time corresponding to the first lip shape. Therefore, when outputting audio, the digital human is controlled to display the lip shape pictures that match the lip shape sequence, the multiple first intermediate lip shapes, and the multiple second intermediate lip shapes based on the target lip shape picture display time.
[0069] In this solution, the lip shape picture display time corresponding to each lip shape in the lip shape sequence is determined according to the duration of each phoneme in the phoneme sequence, and the lip shape picture display time corresponding to each lip shape in the lip shape sequence is further determined according to the lip shape picture display time corresponding to the lip shape in the lip shape sequence and the target lip shape picture display times corresponding to multiple intermediate lip shapes are respectively determined. This ensures that when the digital human displays the lip shape picture along with the output of the audio, the lip shape picture can better correspond to the audio.
[0070] However, in reality, due to various algorithm errors, the total duration of the phoneme sequence extracted from the audio is often shorter than the audio duration. This results in the lip-syncing image displayed by the digital human not matching the audio when the audio is output. For example, the lip-syncing image lags behind the current output audio content, or the lip-syncing image is ahead of the current output audio content.
[0071] In an optional embodiment, the problem of the output audio and the lip-syncing picture being out of sync can be solved by:
[0072] First, according to the audio corresponding to the text interaction information to be output, the first target time in the silent state is determined, and the duration of the first target time is greater than or equal to the set time threshold. Then, the target preset lip shape corresponding to the silent state is screened out from the second lip shape and multiple first intermediate lip shapes; and according to the target lip shape screen display time corresponding to the target preset lip shape, the second target time is determined to be greater than the set time threshold, and the second target time is continuous and composed of the target lip shape screen display time corresponding to the target preset lip shape. Finally, according to the first target time and the second target time, the duration of each phoneme in the phoneme sequence is adjusted to re-determine the target lip shape screen display time of the lip shape, the first intermediate lip shape and / or the second intermediate lip shape in the lip shape sequence.
[0073] In fact, this embodiment determines whether the audio and video are synchronized by determining whether the time corresponding to the target state in the audio is consistent with the target lip-sync picture display time corresponding to the target state in the lip-sync picture. Since the time in the audio cannot be modified, when they are inconsistent, the target lip-sync picture display time corresponding to the lip-sync picture is adjusted to make them consistent in the end, thereby achieving audio and video synchronization. Among them, the target state in this embodiment is a silent state.
[0074] Since pauses of different time intervals will occur during the speaking process, in order to avoid misjudgment of the silent state, in this embodiment, the state in which the volume in the audio is zero and the duration is greater than or equal to the set time threshold is called the silent state, and the time corresponding to the silent state in the audio is called the first target time. It can be understood that there can be more than one first target time.
[0075] In this embodiment, the target preset lip shape corresponding to the silent state is a closed lip shape. It is understandable that a closed lip shape may actually appear during the transformation process, but the closed lip shape at this time does not represent a silent state. In this embodiment, in order to determine the second target time corresponding to the silent state in the lip shape picture, first, the closed lip shape is selected from the second lip shape and multiple first intermediate lip shapes, and then the second target time whose duration is greater than or equal to the set time threshold is determined from the target lip shape picture display time corresponding to the closed lip shape.
[0076] It should be noted that when there are multiple adjacent closed mouth shapes, the target mouth shape display times corresponding to the multiple closed mouth shapes are accumulated, and the accumulated result is compared with the set time threshold. If the accumulated result is greater than or equal to the set time threshold, the second target time is determined to be the time corresponding to the accumulated result.
[0077] Optionally, the target state may be other states except the silent state, and the target preset mouth shape corresponding to the target state may be other mouth shapes except the closed mouth shape.
[0078] Optionally, according to the first target time and the second target time, the duration of each phoneme in the phoneme sequence is adjusted to re-determine the target lip shape screen display time, including: determining an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence according to the ratio between the first target time and the second target time; taking the product of the duration of each phoneme in the phoneme sequence and the adjustment parameter as the duration of each phoneme in the adjusted phoneme sequence, and then re-determining the target lip shape screen display time of the lip shape, the first middle lip shape and / or the second middle lip shape in the lip shape sequence according to the duration of each phoneme in the adjusted phoneme sequence. Wherein, when calculating the ratio between the first target time and the second target time, the target time point or target time slice in the first target time and the second target time can be taken for ratio calculation, for example: the target time point is the starting time point in the first target time and the second target time. It can be understood that when the ratio between the first target time and the second target time is 1, it indicates that the first target time corresponds to the second target time, and there is no need to adjust the duration of each phoneme in the phoneme sequence.
[0079] Optionally, both the first target time and the second target time are multiple. In this case, determining the adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence according to the ratio between the first target time and the second target time includes: first, determining multiple ratios of multiple first target times to multiple second target times in chronological order; then, determining the adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence according to the multiple ratios.
[0080] For example, assuming that in time sequence, the first target time includes: T11, T12 and T13, and the second target time includes: T21, T22 and T23, then the ratios of the first target time to the second target time are: δ1=T11 / T21, δ2=T12 / T22, δ3=T13 / T23. Optionally, the adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence can be determined by weighted averaging of δ1, δ2 and δ3.
[0081] In practical applications, the ratio of the first target time to the second target time is not always reasonable, and some data have obvious errors. In order to eliminate the influence of the data with errors on the adjustment parameters, multiple target ratios within a preset numerical range can be determined from multiple ratios of multiple first target times to multiple second target times, and then the adjustment parameters for adjusting the duration of each phoneme in the phoneme sequence are determined based on the multiple target ratios. For example: the preset numerical range is 1 to 1.8, assuming that δ1=2, δ2=1.1, and δ3=1.2, the target ratios are determined to be δ2 and δ3, and then the adjustment parameters for adjusting the duration of each phoneme in the phoneme sequence are determined by weighted averaging of δ2 and δ3.
[0082] It is understandable that one adjustment may not make the time corresponding to the target state in the audio consistent with the target lip-sync picture display time corresponding to the target state in the lip-sync picture. Therefore, the above process can be executed repeatedly until the first target time is consistent with the second target time, that is, the ratio is 1, or the error between the first target time and the second target time is within a preset range.
[0083] The above embodiment introduces the specific process of outputting the text interaction information to be output in the form of a digital human. In practical applications, the image of the digital human can be diverse, for example, it can be the image of a modern person or the image of an ancient person. When the digital human is the image of an ancient person, if the text interaction information to be output is in the form of classical Chinese, it can further enhance the user's immersive experience during human-computer interaction. The following will explain how to obtain the text interaction information to be output in the form of classical Chinese through specific embodiments.
[0084] Figure 5A flowchart of an interactive method provided by an embodiment of the present invention is used to obtain text interactive information to be output in the form of classical Chinese, such as Figure 5 As shown, the method comprises the following steps:
[0085] 501. Acquire target interaction information sent when the user interacts with the digital human.
[0086] 502. Input the target interaction information into the target language model to generate text interaction information to be output for responding to the target interaction information, wherein the text interaction information to be output is classical Chinese.
[0087] The target language model is trained in the following way:
[0088] First, the first corpus information obtained that is related to the ancient people represented by the digital human is input into the first language model to filter out the corpus information with correct content from the first corpus information as the second corpus information. Then, the second corpus information is input into the second language model to generate a question-answer pair that matches the content of the second corpus information, the questions in the question-answer pair are in classical Chinese or vernacular Chinese, and the answers in the question-answer pair are in classical Chinese. Finally, the question-answer pair is input into the language model to be trained to obtain the target language model; wherein the language model to be trained includes a pre-trained third language model and a target weight matrix constructed for the weight matrix in the third language model; during training, the model parameters of the third language model remain unchanged, and the weight parameters of the target weight matrix are adjusted.
[0089] Understandably, using Figure 5 The interactive method in the embodiment generates the process of text interactive information to be output, that is, the process of using the target language model. For ease of understanding, the training process of the target language model is first described, and then the process of using the target language model is described.
[0090] In this embodiment, the language model can be any language model in the related art, and this solution does not limit the model category of the language model. Since the multiple language models involved in this solution are used to implement different functions, for the convenience of description, they are distinguished by "first", "second", etc. according to their different functions, such as: first language model, second language model, etc.
[0091] Figure 6 A schematic diagram of a training process of a target language model provided in an embodiment of the present invention is shown below in combination with Figure 6 The training process of the target language model is described.
[0092] Since the target language model is used to generate text interaction information to be output in the form of classical Chinese, before model training, it is necessary to first obtain corpus information related to the ancient people represented by the digital human.
[0093] In actual applications, the content of the corpus information obtained through various search engines is not always correct. For example, the obtained corpus information describes "Zhu Xi is from the Yuan Dynasty", but in fact Zhu Xi is from the Southern Song Dynasty. In order to ensure the accuracy of model training, it is necessary to judge the correctness of the content of the obtained corpus information. In this solution, the first language model is used to judge the correctness of the content of the corpus information.
[0094] For ease of distinction, in this scheme, the corpus information before the content correctness judgment is performed is referred to as the first corpus information, the corpus information whose content is determined to be correct after the content correctness judgment is performed on the first corpus information is referred to as the second corpus information, and the corpus information whose content is determined to be incorrect after the content correctness judgment is performed on the first corpus information is referred to as the third corpus information. The first corpus information includes the second corpus information and the third corpus information.
[0095] In the specific implementation process, the first corpus information is input into the first language model, and the first language model determines whether the first corpus information is correct, so as to filter out the corpus information with correct content from the first language information as the second corpus information, so that the second corpus information can be used for training the target language model.
[0096] Optionally, the second corpus information filtered out by the first language model may be manually screened again to further ensure the correctness of the corpus information content and to make up for the defect of the first language model causing misjudgment due to the inability to judge the correctness of the content of individual corpus information.
[0097] The target language model is applied to human-computer interaction scenarios, and accordingly, its training samples should at least include question-answer pairs, so that the target language model can realize question-answer interaction during use.
[0098] In this solution, the second language model is used to generate question-answer pairs based on the input second corpus information as training samples for the target language model. In view of the fact that the target language model in this solution is to generate text interaction information to be output in the form of classical Chinese based on the target interaction information input by the user, and realize classical Chinese question-answering, the answer to any of the question-answer pairs generated by the second language model is in classical Chinese, and the questions in the question-answer pairs can be in classical Chinese or in vernacular Chinese. Among them, the training samples of the second language model include multiple groups of "corpus information-question-answer pair" samples. The second language model learns from multiple groups of "corpus information-question-answer pair" samples so that the model has the ability to generate question-answer pairs based on corpus information, wherein the corpus information can be a sentence, a paragraph, etc.
[0099] It is understandable that the content of the question-answer pair generated by the second language model should match the content contained in the second corpus information in terms of meaning. For example, assuming that the second corpus information input into the second language model is "Zhu Xi, a poet in the Southern Song Dynasty, born in 1130 and died in 1200", the corresponding question-answer pairs generated may include: "'What is your name?' 'I am Zhu Xi'", "'Which dynasty are you from?' 'I am a person from the Southern Song Dynasty'", etc.
[0100] Optionally, after generating question-answer pairs through the second language model, these question-answer pairs can also be input into the fourth language model to obtain target question-answer pairs that match the content of these question-answer pairs. In this solution, the fourth language model is used to expand the number of generated question-answer pairs, and the number of target question-answer pairs generated by the fourth language model is greater than the number of question-answer pairs input into the fourth language model. When expanding the question-answer pairs, the target question-answer pairs generated by the fourth language model differ from the question-answer pairs input into the fourth language model in expression, for example: the target question-answer pairs are obtained by replacing certain words in the question-answer pairs input into the fourth language model with synonyms or near-synonyms, or the target question-answer pairs are obtained by adjusting the word order of the questions or answers in the question-answer pairs input into the fourth language model.
[0101] After obtaining the question-answer pair and / or the target question-answer pair, the question-answer pair and / or the target question-answer pair are input into the language model to be trained to obtain the target language model. The following is an example of inputting the question-answer pair into the language model to be trained to obtain the target language model.
[0102] In order to speed up the training of the target language model, the language model to be trained includes a pre-trained third language model and a target weight matrix constructed for the weight matrix in the third language model. During training, the model parameters of the third language model are kept unchanged, and only the weight parameters of the target weight matrix are adjusted. Since the target weight matrix is constructed for the weights in the third language model, the adjusted target weight matrix can be used in conjunction with the third language model to interfere with the results produced by the third language model. The results produced after the third language model is interfered with are the output results of the target language model. Assuming that the weight matrix of the third language model is A and the target weight matrix after training is B, the weight matrix W corresponding to the target language model when performing model inference is W=A+B.
[0103] In this scheme, from the perspective of model scale, the model scale of the third language model is significantly larger than the target weight matrix. Compared with directly adjusting the weight parameters of the third language model, the weight parameters of the target weight matrix are adjusted in this scheme, and the number of parameters that need to be adjusted is smaller, so the model training speed is also faster.
[0104] The above is the training process of the target language model. The following is an explanation of the use process of the target language model.
[0105] In this embodiment, after acquiring the target interaction information sent by the user when interacting with the digital human, the target interaction information is input into the target language model to generate text interaction information to be output for responding to the target interaction information based on the target language model. The target interaction information can be classical Chinese or vernacular Chinese, and the expression form of the text interaction information to be output is limited to classical Chinese with the characteristics of the target ancient language.
[0106] When the target language model generates text interaction information to be output based on the target interaction information, in an optional embodiment, target data having a similarity with the target interaction information greater than a set threshold can be determined from a training sample set corresponding to the target language model, the training sample set including a first training sample set for training the target language model and a second training sample set for training the third language model; thereafter, text interaction information to be output for responding to the target interaction information is generated based on the target data.
[0107] Judging from the number of samples included in the training sample sets, the number of samples included in the first training sample set is smaller than that in the second training sample set. In order to speed up the generation of text interaction information to be output, optionally, the target data can be first determined from the first training sample set. When the target data does not exist in the first training sample set, the target data can be determined from the second training sample set.
[0108] Specifically, if there is first sample data in the first training sample set whose similarity with the target interaction information is greater than a first set threshold, the target data is determined according to the first sample data. If there is no first sample data in the first training sample set whose similarity with the target interaction information is greater than the first set threshold, the target data is determined according to second sample data in the second training sample set whose similarity with the target interaction information is greater than a second set threshold. Optionally, the second set threshold may be equal to the first set threshold, or may be less than the first set threshold.
[0109] It can be understood that the first training sample set at least includes question-answer pairs input into the language model to be trained. In the specific implementation process, optionally, if there is a target question in the question-answer pairs included in the first training sample set whose similarity with the target interactive information is greater than a first set threshold, then according to the target question, the answer in the question-answer pair corresponding to the target question is determined as the target data. For example, assume that the first training sample includes: question-answer pair 1, question-answer pair 2, ..., question-answer pair N (N is an integer), wherein question-answer pair 1 includes question 1 and answer 1, question 2 includes question 2 and answer 2, and so on. When determining the text interactive information to be output, first, determine whether there is a target question in question 1, question 2, ... question N of the first training sample whose similarity with the target interactive information is greater than the first set threshold. Assuming that the similarity between the target interactive information of question 1 is the largest and greater than the set threshold, the target question is determined to be question 1, and the answer 1 corresponding to question 1 is determined as the target data. Optionally, the answer 1 may be directly used as the text interaction information to be output, or part of the content in the answer 1 may be replaced or the word order may be adjusted, thereby ensuring the diversity of the output text interaction information to be output.
[0110] In summary, in this solution, the target language model is used to obtain the text interaction information to be output in the form of classical Chinese, thereby enhancing the user's immersive experience when interacting with digital ancient people. In the process of generating the target language model, the first language model is used to judge the correctness of the first corpus information obtained, and the second corpus information with correct content is selected for the training of the target language model, thereby ensuring the accuracy of the training samples and improving the accuracy of the target language model; in addition, the model parameters of the third language model are kept unchanged, and only the weight parameters of the target weight matrix are adjusted, which can significantly improve the training efficiency of the model.
[0111] The following will describe in detail one or more embodiments of the digital human display device of the present invention. Those skilled in the art will appreciate that these devices can be configured using commercially available hardware components through the steps taught in this solution.
[0112] Figure 7 A schematic diagram of the structure of a digital human display device provided by an embodiment of the present invention is shown in FIG. Figure 7 As shown, the device includes: an acquisition module 11, a processing module 12, and a display module 13.
[0113] The acquisition module 11 is used to acquire audio and phoneme data corresponding to the text interaction information to be output, wherein the phoneme data includes a phoneme sequence.
[0114] The processing module 12 is used to determine the mouth shape sequence corresponding to the phoneme sequence according to the preset correspondence between the phonemes and the mouth shapes; for any two adjacent first mouth shapes and second mouth shapes in the mouth shape sequence, determine multiple first intermediate mouth shapes that transition from the first mouth shape to the second mouth shape according to the mouth shape state parameters corresponding to the first mouth shape and the second mouth shape respectively.
[0115] The display module 13 is used to control the digital human to display the lip shape pictures matching the lip shape sequence and the multiple first intermediate lip shapes when outputting the audio.
[0116] Optionally, the processing module 12 is further used to determine, for a target mouth shape with only one adjacent mouth shape in the mouth shape sequence, a plurality of second intermediate mouth shapes transitioning from the preset mouth shape to the target mouth shape and / or a plurality of second intermediate mouth shapes transitioning from the target mouth shape to the preset mouth shape according to the mouth shape state parameters corresponding to the preset mouth shape and the mouth shape state parameters corresponding to the target mouth shape; wherein the preset mouth shape matches the preset mouth shape picture displayed by the digital human when no interaction is performed.
[0117] Correspondingly, the display module 13 is also used to control the digital human to display the lip shape pictures matching the lip shape sequence, the multiple first intermediate lip shapes and the second intermediate lip shapes when outputting the audio.
[0118] Optionally, the phoneme data also includes the duration of each phoneme in the phoneme sequence. The processing module 12 is further configured to determine the lip shape display time corresponding to each lip shape in the lip shape sequence according to the duration of each phoneme in the phoneme sequence; and determine the target lip shape display time corresponding to the second lip shape and the plurality of first intermediate lip shapes according to the lip shape display time corresponding to the second lip shape.
[0119] Correspondingly, the display module 13 is also used to control the digital human to display the lip shape pictures matching the lip shape sequence and the multiple first intermediate lip shapes according to the target lip shape picture display time when outputting the audio.
[0120] Optionally, the processing module 12 is further used to determine a first target time in a silent state based on the audio, the duration of the first target time being greater than or equal to a set time threshold; screen out a target preset lip shape corresponding to the silent state from the second lip shape and the multiple first intermediate lip shapes; determine a second target time greater than the set time threshold based on the target lip shape screen display time corresponding to the target preset lip shape, the second target time being continuous and composed of the target lip shape screen display time corresponding to the target preset lip shape; adjust the duration of each phoneme in the phoneme sequence based on the first target time and the second target time to re-determine the target lip shape screen display time.
[0121] Optionally, the processing module 12 is specifically used to determine an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence based on the ratio between the first target time and the second target time; and take the product of the duration of each phoneme in the phoneme sequence and the adjustment parameter as the adjusted duration of each phoneme in the phoneme sequence.
[0122] Optionally, the first target time and the second target time are both multiple. The processing module 12 is further specifically configured to determine multiple ratios of the multiple first target times to the multiple second target times in chronological order; and determine, based on the multiple ratios, an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence.
[0123] Optionally, the processing module 12 is further specifically configured to determine a plurality of target ratios within a preset numerical range from the plurality of ratios; and determine, based on the plurality of target ratios, an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence.
[0124] Optionally, the acquisition module 11 is further used to acquire target interaction information sent when the user interacts with the digital human.
[0125] Correspondingly, the processing module 12 is also used to input the target interaction information into the target language model to generate the text interaction information to be outputted in response to the target interaction information, and the text interaction information to be outputted is classical Chinese. Wherein, the target language model is trained in the following manner: the first corpus information obtained and associated with the ancient person represented by the digital person is input into the first language model to filter out the corpus information with correct content from the first corpus information as the second corpus information; the second corpus information is input into the second language model to generate a question-answer pair matching the content of the second corpus information, the question in the question-answer pair is classical Chinese or vernacular Chinese, and the answer in the question-answer pair is classical Chinese; the question-answer pair is input into the language model to be trained to obtain the target language model; wherein, the language model to be trained includes a pre-trained third language model and a target weight matrix constructed for the weight matrix in the third language model; during training, the model parameters of the third language model remain unchanged, and the weight parameters of the target weight matrix are adjusted.
[0126] Optionally, the processing module 12 is specifically used to determine target data whose similarity with the target interaction information is greater than a set threshold from a training sample set corresponding to the target language model, the training sample set including a first training sample set for training to obtain the target language model and a second training sample set for training to obtain the third language model; and generate the text interaction information to be output in response to the target interaction information according to the target data.
[0127] Figure 7 The device shown can execute the steps introduced in the aforementioned embodiments. For detailed execution process and technical effects, please refer to the description in the aforementioned embodiments, which will not be repeated here.
[0128] In one possible design, the above Figure 7 The structure of the digital human display device shown can be realized as an electronic device, such as Figure 8 As shown, the electronic device may include: a memory 21, a processor 22, and a communication interface 23. The memory 21 stores executable codes, and when the executable codes are executed by the processor 22, the processor 22 can at least implement the digital human display method provided in the above-mentioned embodiment.
[0129] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of an electronic device, the processor can at least implement the digital human display method provided in the aforementioned embodiment.
[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. Those of ordinary skill in the art may understand and implement the present invention without creative effort.
[0131] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on such an understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A digital human display method, characterized in that: include: Acquire audio and phoneme data corresponding to the text interaction information to be output, wherein the phoneme data includes a phoneme sequence and a duration of each phoneme in the phoneme sequence; Determining the lip shape sequence corresponding to the phoneme sequence according to the preset correspondence relationship between the phonemes and the lip shapes; For any two adjacent first and second lip shapes in the lip shape sequence, determining a plurality of first intermediate lip shapes for transitioning from the first lip shape to the second lip shape according to lip shape state parameters respectively corresponding to the first lip shape and the second lip shape; Determining the lip shape picture display time corresponding to each lip shape in the lip shape sequence according to the duration of each phoneme in the phoneme sequence; Determine target lip shape display times corresponding to the second lip shape and the plurality of first intermediate lip shapes, respectively, according to the lip shape display time corresponding to the second lip shape; Determine, according to the audio, a first target time in a silent state, wherein the duration of the first target time is greater than or equal to a set time threshold; Filtering a target preset lip shape corresponding to the silent state from the second lip shape and the plurality of first intermediate lip shapes; According to the target lip shape screen display time corresponding to the target preset lip shape, determining a second target time whose duration is greater than the set time threshold, wherein the second target time is continuous and is composed of the target lip shape screen display time corresponding to the target preset lip shape; determining, according to a ratio between the first target time and the second target time, an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence; The product of the duration of each phoneme in the phoneme sequence and the adjustment parameter is used as the adjusted duration of each phoneme in the phoneme sequence to re-determine the lip-sync screen display time; When the audio is output, the digital human is controlled to display the lip shape pictures matching the lip shape sequence and the plurality of first intermediate lip shapes according to the re-determined target lip shape picture display time.
2. The method according to claim 1, characterized in that The method further comprises: For a target mouth shape with only one adjacent mouth shape in the mouth shape sequence, a plurality of second intermediate mouth shapes transitioning from the preset mouth shape to the target mouth shape and / or a plurality of second intermediate mouth shapes transitioning from the target mouth shape to the preset mouth shape are determined according to the mouth shape state parameters corresponding to the preset mouth shape and the mouth shape state parameters corresponding to the target mouth shape; wherein the preset mouth shape matches the preset mouth shape picture displayed by the digital human when no interaction is performed; When outputting the audio, controlling the digital human to display a lip shape picture matching the lip shape sequence and the plurality of first intermediate lip shapes comprises: When the audio is output, the digital human is controlled to display a lip shape picture matching the lip shape sequence, the plurality of first intermediate lip shapes and the second intermediate lip shape.
3. The method according to claim 1, characterized in that: The first target time and the second target time are both multiple; and determining the adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence according to the ratio between the first target time and the second target time includes: Determining, in chronological order, a plurality of ratios of a plurality of the first target times to a plurality of the second target times; An adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence is determined according to the plurality of ratios.
4. The method according to claim 3, characterized in that The step of determining, according to the multiple ratios, an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence comprises: Determining a plurality of target ratios within a preset value range from the plurality of ratios; An adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence is determined according to the multiple target ratios.
5. The method according to claim 1, characterized in that: The method further comprises: Acquiring target interaction information sent when the user interacts with the digital human; Inputting the target interaction information into a target language model to generate the text interaction information to be outputted in response to the target interaction information, wherein the text interaction information to be outputted is classical Chinese; The target language model is trained in the following way: Inputting the acquired first corpus information associated with the ancient person represented by the digital person into the first language model, so as to filter out corpus information with correct content from the first corpus information as the second corpus information; Inputting the second corpus information into a second language model to generate a question-answer pair matching the content of the second corpus information, wherein the question in the question-answer pair is in classical Chinese or vernacular Chinese, and the answer in the question-answer pair is in classical Chinese; The question-answer pair is input into a language model to be trained to obtain a target language model; wherein the language model to be trained includes a pre-trained third language model and a target weight matrix constructed for the weight matrix in the third language model; during training, the model parameters of the third language model remain unchanged, and the weight parameters of the target weight matrix are adjusted.
6. The method according to claim 5, characterized in that The step of inputting the target interaction information into a target language model to generate the text interaction information to be output in response to the target interaction information includes: Determine, from a training sample set corresponding to the target language model, target data having a similarity with the target interaction information greater than a set threshold, the training sample set comprising a first training sample set for training to obtain the target language model and a second training sample set for training to obtain the third language model; The text interaction information to be outputted for responding to the target interaction information is generated according to the target data.
7. A digital human display device, characterized in that: include: An acquisition module, used to acquire audio and phoneme data corresponding to the text interaction information to be output, wherein the phoneme data includes a phoneme sequence and a duration of each phoneme in the phoneme sequence; A processing module is used to determine a lip shape sequence corresponding to a phoneme sequence according to a preset correspondence between phonemes and lip shapes; for any two adjacent first and second lip shapes in the lip shape sequence, determine a plurality of first intermediate lip shapes that transition from the first lip shape to the second lip shape according to lip shape state parameters corresponding to the first and second lip shapes respectively; determine a lip shape screen display time corresponding to each lip shape in the lip shape sequence according to the duration of each phoneme in the phoneme sequence; determine a target lip shape screen display time corresponding to the second lip shape and the plurality of first intermediate lip shapes respectively according to the lip shape screen display time corresponding to the second lip shape; and determine a first target time in a silent state according to the audio, wherein the duration of the first target time is greater than or equal to a set time threshold; A target preset lip shape corresponding to the silent state is selected from the second lip shape and the plurality of first intermediate lip shapes; a second target time having a duration greater than the set time threshold is determined according to the target lip shape screen display time corresponding to the target preset lip shape, wherein the second target time is continuous and is composed of the target lip shape screen display time corresponding to the target preset lip shape; an adjustment parameter for adjusting the duration of each phoneme in the phoneme sequence is determined according to the ratio between the first target time and the second target time; and the product of the duration of each phoneme in the phoneme sequence and the adjustment parameter is used as the adjusted duration of each phoneme in the phoneme sequence to re-determine the lip shape screen display time; The display module is used to control the digital human to display the lip shape pictures matching the lip shape sequence and the plurality of first intermediate lip shapes according to the re-determined target lip shape picture display time when outputting the audio.
8. An electronic device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the digital human display method as described in any one of claims 1 to 6.
9. A non-transitory machine-readable storage medium, characterized in that: The non-transitory machine-readable storage medium stores executable codes, and when the executable codes are executed by a processor of an electronic device, the processor is caused to execute the digital human display method as described in any one of claims 1 to 6.
Citation Information
Patent Citations
Mouth shape animation display method and device based on audio information and storage medium
CN111915707A
Voice mouth shape synchronous generation method and device, electronic equipment and storage medium
CN115966194A
Three-dimensional image smooth pronunciation mouth shape simulation method
CN116228934A
Mouth shape animation generation method and device, electronic equipment and storage medium
CN116363268A
Knowledge graph generation type question answering method and system based on large language model
CN117033608A
