Digital human driving method, device and computer program product

Abstract

The present disclosure provides a digital human driving method, device, electronic device, storage medium and computer program product, which relate to the field of artificial intelligence, specifically to deep learning, digital humans, augmented reality, virtual reality, mixed reality, extended reality, metaverse and other technologies. The specific implementation scheme is as follows: based on the rhythm point information of the acquired target melody audio, a search is performed from a K-dimensional tree that represents the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio; initial audio is generated according to the target lyrics text; based on the rhythm point information, the initial audio is adjusted to obtain the target audio, and the audio features of the target audio are determined; lip movement information is determined based on the audio features; the target audio is used as the digital human's voice, and the lip movement information is used as the digital human's lip movement driving information to drive the digital human to operate. The present disclosure improves the display effect of driving a digital human to sing based on melody audio.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, specifically to technical fields such as deep learning, digital humans, augmented reality, virtual reality, mixed reality, extended reality, and the metaverse, and in particular to digital human driving methods, devices, and training methods, devices, electronic devices, storage media, and computer program products for lip movement information determination models. Background Art

[0002] The generation of three-dimensional digital humans has long been a research hotspot in academia and has broad industrial applications. Common applications include virtual anchors, virtual customer service representatives, virtual assistants, virtual teachers, and virtual idols. The lip movement information of 3D digital humans must be determined during their activation.

[0003] In related technologies, the lip movement coefficients of the digital human are generally determined through singing audio to drive the operation of the digital human. The driving method is relatively simple and lacks interest. Summary of the Invention

[0004] The present disclosure provides a digital human driving method and device, as well as a training method and device for a lip movement information determination model, an electronic device, a storage medium, and a computer program product.

[0005] According to a first aspect, a digital human driving method is provided, comprising: based on the acquired rhythm point information of the target melody audio, searching from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text, to determine the target lyrics text corresponding to the target melody audio; generating initial audio according to the target lyrics text; adjusting the initial audio according to the rhythm point information to obtain the target audio, and determining the audio features of the target audio; determining lip movement information according to the audio features; and driving the digital human to operate using the target audio as the digital human's voice and the lip movement information as the digital human's lip movement driving information.

[0006] According to the second aspect, a training method for a lip movement information determination model is provided, comprising: based on the rhythm point information of the acquired melody audio, searching from a K-dimensional tree that characterizes the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio; generating initial audio according to the target lyrics text; adjusting the initial audio according to the rhythm point information to obtain sample audio, and determining the audio features of the sample audio; generating training samples based on the audio features and lip movement information corresponding to the same sample audio to obtain a training sample set, wherein the lip movement information is obtained based on an initial video matching the melody audio; using a machine learning algorithm, taking the audio features as input and the lip movement information labels corresponding to the input audio features as expected output, to train the initial lip movement information determination model and obtain a trained lip movement information determination model.

[0007] According to a third aspect, a digital human driving device is provided, comprising: a first determination unit, configured to search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text based on the acquired rhythm point information of the target melody audio, and determine the target lyrics text corresponding to the target melody audio; a first generation unit, configured to generate initial audio according to the target lyrics text; a second determination unit, configured to adjust the initial audio according to the rhythm point information, obtain the target audio, and determine the audio features of the target audio; a third determination unit, configured to determine the lip movement information according to the audio features; and a driving unit, configured to drive the digital human to operate using the target audio as the digital human's voice and the lip movement information as the lip movement driving information of the digital human.

[0008] According to a fourth aspect, a training device for a lip movement information determination model is provided, comprising: a fourth determination unit, configured to search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text based on the rhythm point information of the acquired melody audio, to determine the target lyrics text corresponding to the melody audio; a second generation unit, configured to generate initial audio according to the target lyrics text; a fifth determination unit, configured to adjust the initial audio according to the rhythm point information, obtain sample audio, and determine the audio features of the sample audio; a third generation unit, configured to generate training samples based on the audio features and lip movement information corresponding to the same sample audio, to obtain a training sample set, wherein the lip movement information is obtained based on an initial video matching the melody audio; a training unit, configured to adopt a machine learning algorithm, take the audio features as input, and take the lip movement information labels corresponding to the input audio features as expected output, to train the initial lip movement information determination model, and obtain a trained lip movement information determination model.

[0009] According to the fifth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to execute the method described in any implementation of the first aspect or the second aspect.

[0010] According to a sixth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any one of the implementations of the first and second aspects.

[0011] According to a seventh aspect, a computer program product is provided, comprising: a computer program, which implements the method described in any implementation manner of the first aspect or the second aspect when executed by a processor.

[0012] According to the technology disclosed in the present invention, a digital human driving method is provided, which performs a K-dimensional tree search based on the rhythm point information of the acquired melody audio to determine the lyrics text corresponding to the melody audio, thereby improving the accuracy of the determined lyrics text; and adjusts the initial audio generated based on the lyrics text according to the rhythm point information, so that the target audio obtained by adjusting the initial audio is more consistent with the original singing audio corresponding to the melody audio; then, the audio features of the adjusted target audio are obtained, and the lip movement information is determined according to the audio features, and the digital human is driven to operate according to the lip movement information and the target audio, thereby improving the display effect of driving the digital human to sing based on the melody audio.

[0013] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0015] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0016] Figure 2 is a flow chart of an embodiment of a digital human driving method according to the present disclosure;

[0017] Figure 3 is a schematic diagram of rhythm points in target melody audio according to the present disclosure;

[0018] Figure 4 is a schematic diagram of an application scenario of the digital human driving method according to this embodiment;

[0019] Figure 5 is a schematic diagram of adapting initial audio with reference to target melody audio according to the present disclosure;

[0020] Figure 6 is a schematic diagram of adjusting initial audio according to audio style information of target melody audio according to the present disclosure;

[0021] Figure 7 is a flow chart of another embodiment of the digital human driving method according to the present disclosure;

[0022] Figure 8 is a data flow diagram of the digital human driving method according to the present disclosure;

[0023] Figure 9 is a flowchart of an embodiment of a method for training a lip movement information determination model according to the present disclosure;

[0024] Figure 10 is a structural diagram of an embodiment of a digital human driving device according to the present disclosure;

[0025] Figure 11 is a structural diagram of an embodiment of a training device for determining a model based on lip movement information according to the present disclosure;

[0026] Figure 12 It is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. DETAILED DESCRIPTION

[0027] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0029] Figure 1 An exemplary architecture 100 is shown to which the digital human driving method and apparatus, and the lip movement information determination model training method and apparatus of the present disclosure can be applied.

[0030] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 constitute a topological network, and network 104 is used to provide a medium for communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0031] Terminal devices 101, 102, and 103 can be hardware devices or software that support network connection for data interaction and data processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing, and other functions, including but not limited to image acquisition devices, smart phones, tablet computers, e-book readers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitations are given here.

[0032] Server 105 can be a server that provides various services. For example, based on the target melody audio provided by terminal devices 101, 102, and 103, it can determine the target audio and lip movement information to drive the digital human to sing. For another example, based on the melody audio provided by terminal devices 101, 102, and 103, it can generate a training sample set and train a lip movement information determination model. As an example, server 105 can be a cloud server.

[0033] It should be noted that the server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., software or software modules for providing distributed services), or as a single software or software module. No specific limitations are given here.

[0034] It should also be noted that the digital human driving method and lip movement information determination model training method provided in the embodiments of the present disclosure can be executed by a server, a terminal device, or a server and a terminal device in coordination with each other. Accordingly, the various components (e.g., various units) included in the digital human driving device and the lip movement information determination model training device can be entirely located in the server, entirely located in the terminal device, or separately located in the server and the terminal device.

[0035] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the system is merely illustrative. Any number of terminal devices, networks, and servers may be used depending on implementation needs. When the electronic device on which the digital human driving method and the training method for determining a lip movement information model are running does not need to transmit data with other electronic devices, the system architecture may only include the electronic device (e.g., a server or terminal device) on which the digital human driving method and the training method for determining a lip movement information model are running.

[0036] Please refer to Figure 2 , Figure 2 This is a flowchart of a digital human driving method provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0037] Step 201 : Based on the acquired rhythm point information of the target melody audio, a search is performed from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio.

[0038] In this embodiment, the execution subject of the digital human driving method (for example, Figure 1The terminal device or server in the network can obtain the target melody audio remotely or locally based on a wired network connection method or a wireless network connection method, and based on the rhythm point information of the obtained target melody audio, search from the K-dimensional tree that characterizes the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio.

[0039] The target melody audio is an audio that only includes melody data but does not include vocal data. In the field of music, the target melody audio can be melody data in a song audio released by any singer.

[0040] By using the melody audio automatic dotting technology, all the rhythm point positions in the target melody audio can be determined, and the rhythm point information of the target melody audio can be determined. Figure 3 As shown, a schematic diagram 300 is shown showing rhythm point information in a target melody audio.

[0041] After obtaining the target melody's rhythm information, the execution entity can search a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio. A K-dimensional tree is a binary tree data structure that uses a K-dimensional hyperplane to partition a set of points in a K-dimensional space.

[0042] In this implementation, the execution entity may construct a vector pair based on the corresponding feature vectors of the lyrics text and the feature vectors of the melody audio, and then establish a KD-tree (K-dimensional tree) based on the multiple vector pairs.

[0043] Step 202: Generate initial audio based on the target lyrics text.

[0044] In this embodiment, the execution entity may generate initial audio according to the target lyrics text.

[0045] As an example, the execution subject may obtain the machine audio corresponding to the lyrics text as the initial audio through TTS (Text To Speech) technology.

[0046] Step 203: Adjust the initial audio according to the rhythm point information to obtain the target audio, and determine the audio features of the target audio.

[0047] In this embodiment, the execution entity may adjust the initial audio according to the rhythm point information to obtain the target audio, and determine the audio features of the target audio.

[0048] As an example, the execution entity may adjust the initial audio based on the rhythm represented by the rhythm point information of the target melody audio so that the rhythm of the initial audio matches the rhythm of the target melody audio. Furthermore, the execution entity may extract features from the target audio using a feature extraction network to obtain audio features of the target audio. As an example, the feature extraction network may be a network model such as a convolutional neural network or a recurrent neural network.

[0049] Step 204: Determine lip movement information based on the audio features.

[0050] In this embodiment, the execution entity may determine the lip movement information based on the audio features.

[0051] As an example, the execution subject may determine the lip movement information according to the audio features using a pre-trained lip movement information determination model, wherein the lip movement information determination model represents the correspondence between the audio features and the lip movement information.

[0052] In some optional implementations, the execution entity may further determine target coding information of the selected speaker from a preset coding set, wherein the coding information in the preset coding set represents different speakers corresponding to different melody audios.

[0053] The speaker corresponding to the melody data is the speaker who utters the singing voice represented by the singing voice data corresponding to the melody data, and can specifically be a different singer or vocalist. The preset coding set includes coding information corresponding to multiple different speakers. Each coding information uniquely identifies the corresponding speaker. In this embodiment, the speaker can be encoded using any form of information, such as images or characters.

[0054] As an example, different speakers are encoded using a One Hot encoding method. For each speaker, the encoding information is represented using a One Hot encoding with a length of 32.

[0055] In this implementation, the execution entity or an electronic device communicatively connected to the execution entity may receive a selection instruction from a user and, through the selection instruction, determine target encoding information for a selected speaker from a preset encoding set. As an example, the execution entity may display a list of speakers from which the user can select a target speaker, and the user may issue the selection instruction by means of a gesture, voice, or other command.

[0056] The execution entity may obtain lip movement information using a pre-trained lip movement information determination model based on the audio features and target encoding information. The lip movement information determination model is used to characterize the correspondence between the audio features, target encoding information, and lip movement information. For example, the execution entity may use the audio features and target encoding information as input to the lip movement information determination model to obtain lip movement information as output.

[0057] The lip movement information determination model can adopt any neural network model that can generate lip movement information, including but not limited to existing neural network models such as convolutional neural networks, convolutional neural networks, or network models improved based on existing neural network models.

[0058] As an example, the network structure of the lip movement information determination model includes a Formant Analysis Network, an Articulation Network, an Aware Attention Network, a Fully Connected Layer, and an Output Layer.

[0059] Specifically, the lip movement information determination model can be obtained through training using the training method shown in the subsequent embodiment 900.

[0060] Step 204: Using the target audio as the digital human's voice and the lip movement information as the digital human's lip movement driving information, the digital human is driven to move.

[0061] In this embodiment, the execution subject can use the target audio as the digital human's voice and the lip movement information as the digital human's lip movement driving information to drive the digital human to run.

[0062] As an example, the execution entity can match each frame of target audio with the lip movement information obtained based on each frame of target audio. For each frame of target audio, the target audio is used as the digital human's voice, and the lip movement information corresponding to the target audio is used as the digital human's lip movement driving information to drive the digital human.

[0063] Continue to see Figure 4 , Figure 4 FIG4 is a schematic diagram 400 of an application scenario of the digital human driving method according to this embodiment. Figure 4In the application scenario, the terminal device 401 obtains the target melody audio selected by the user 402 in real time and sends the target melody audio 403 to the server 404. The server 404 first searches the K-dimensional tree 406 representing the correspondence between the melody audio and the lyrics text based on the rhythm point information 405 of the obtained target melody audio 403, and determines the target lyrics text 407 corresponding to the target melody audio; generates initial audio 408 based on the target lyrics text 407; adjusts the initial audio 408 based on the rhythm point information 405 to obtain the target audio 409, and determines the audio features of the target audio 409; determines the lip movement information 410 based on the audio features; and drives the digital human 411 to operate using the target audio 409 as the voice of the digital human 411 and the lip movement information 410 as the lip movement driving information of the digital human 411.

[0064] In this embodiment, a digital human driving method is provided, which performs a K-dimensional tree search based on the rhythm point information of the acquired melody audio to determine the lyrics text corresponding to the melody audio, thereby improving the accuracy of the determined lyrics text; and adjusts the initial audio generated based on the lyrics text according to the rhythm point information, so that the target audio obtained by adjusting the initial audio is more consistent with the original singing audio corresponding to the melody audio; then, the audio features of the adjusted target audio are obtained, and the lip movement information is determined according to the audio features, and the digital human is driven to operate according to the lip movement information and the target audio, thereby improving the display effect of the digital human singing driven by the melody audio.

[0065] In some optional implementations of this embodiment, the execution entity may perform step 201 as follows:

[0066] First, based on the acquired rhythm point information of the target melody audio, an audio segment of the lyrics text to be matched is determined from the target melody audio to obtain an audio segment set.

[0067] As an example, the execution entity can determine the rhythmic features of the target melody audio based on the rhythmic information and perform a timing operation by calculating the locations of the most dramatic tempo changes in the melody audio. Based on the timing operation, the melody audio can be divided into multiple audio segments, each of which is to be matched with text. All audio segments in the melody audio are combined into an audio segment set.

[0068] Second, audio frame features of a target audio frame of an audio segment in the audio segment set are determined.

[0069] The audio frame features include audio features of a target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment.

[0070] As an example, for each audio segment in the obtained audio segment set, the audio features such as the numerical value and speed of the current audio frame (target audio frame) are determined every preset number of audio frames, as well as the numerical value and speed of the audio frames at the start position and end position of the target audio segment to which the target audio frame belongs, and the numerical value and speed of the audio frames at the start position and end position of the next audio segment of the target audio segment. The obtained audio features of the target audio frame, the audio features of the target audio segment to which the target audio frame belongs, and the audio features of the audio segments adjacent to the target audio segment can be used as a set of vectors to represent the audio frame features of the current audio frame for matching operations in a K-dimensional tree.

[0071] Third, based on the audio frame features, a search is performed from the K-dimensional tree to determine the target lyrics text corresponding to the target melody audio.

[0072] In this implementation, the audio segment of the target melody audio to be matched with the lyrics text is first determined, and then the audio frame features of the target audio frame of the audio segment are determined to search and determine the target lyrics text in the K-dimensional tree, thereby improving the accuracy of the determined lyrics text.

[0073] In some optional implementations of this embodiment, for each type of melody audio, the execution entity or an electronic device communicatively connected to the execution entity is provided with a K-dimensional tree corresponding to the type of melody audio. In this implementation, the execution entity may perform the third step as follows:

[0074] First, the target category to which the target melody audio belongs is determined; then, based on the audio frame features, a search is performed from the K-dimensional tree corresponding to the target category to determine the target lyrics text corresponding to the target melody audio.

[0075] As an example, the execution entity may determine the target category of the target melody audio using a pre-trained melody classification model. The melody classification model is used to determine the target category of the target melody audio and may be trained based on a neural network such as a convolutional neural network or a residual neural network.

[0076] As another example, the execution entity can segment the target melody audio into audio segments of a preset duration (e.g., 2 seconds), input the audio segments into a wav2vec model, and the model can output a feature vector representing the characteristics of the currently input audio segment. This feature vector is then used to perform a distance measurement with multiple pre-set category vectors, and the category represented by the category vector with the closest distance is determined as the target category to which the target melody audio belongs.

[0077] In this implementation, each type of melody audio corresponds to a K-dimensional tree, so that the above-mentioned execution entity can search according to the K-dimensional tree corresponding to the type of the target melody audio, thereby improving the pertinence and accuracy of the lyrics text search process.

[0078] In some optional implementations of this embodiment, the execution entity may adjust the initial audio according to the rhythm point information in the following manner:

[0079] First, the pronunciation time range of each text unit in the target lyrics text in the initial audio is determined; based on the rhythm point information and the pronunciation time range, the initial audio is adapted with reference to the target melody audio.

[0080] Specifically, the execution entity may use vosk (an open source automatic speech recognition tool) to detect the pronunciation time range of each text unit in the lyrics text in the original audio. The pronunciation time range represents the pronunciation duration of the text unit in the audio.

[0081] As an example, the above-mentioned execution entity can first determine the pronunciation time range of each single word in the lyrics text in the initial audio; then, segment the lyrics text to obtain the segmented text units included in the lyrics text; finally, for each text unit, the pronunciation time ranges of the single words included in the text unit are combined to obtain the pronunciation time range corresponding to the text unit.

[0082] The rhythmic information in the melody audio is often closely related to the text units in the lyrics. The above-mentioned execution entity can determine the matching relationship between the rhythmic information and the text units; then, by stretching or compressing the pronunciation time range of the original audio to match the rhythmic duration of the target melody audio.

[0083] Continue to refer Figure 5 , showing a schematic diagram of adapting the initial audio with reference to the target melody audio 500. For each text unit in the initial audio 501, the pronunciation time range corresponding to the text unit is aligned with the duration of the corresponding rhythm point information in the target melody audio 502.

[0084] In this implementation, a method is provided for adjusting the initial audio based on the rhythm point information of the target melody audio and the pronunciation time range of the initial audio, so that the adjusted initial audio is more consistent with the target melody audio, thereby improving the accuracy of the lip movement information obtained based on the target melody audio.

[0085] In some optional implementations of this embodiment, the execution entity may further adjust the initial audio in the following manner:

[0086] First, the audio style information of the target melody audio is determined according to the rhythm point information.

[0087] The audio style information may be specifically expressed as audio attribute information such as the pitch and frequency of the target melody audio.

[0088] Then, the adapted initial audio is adjusted according to the audio style information to obtain the target audio.

[0089] In this implementation, the execution subject may refer to the audio style information of the target melody audio and adjust the pitch, frequency and other information of the initial audio so that the audio style information of the target audio matches the audio style information of the target melody audio.

[0090] Please continue to refer to Figure 6 , a schematic diagram 600 is shown of obtaining a target audio 602 by adjusting an adapted initial audio 601 according to audio style information of a target melody audio.

[0091] In this implementation method, on the basis of adjusting the pronunciation time range of each text unit in the initial audio, the initial audio is further adjusted with reference to the audio style information of the target melody audio, so that the obtained target audio is more compatible with the target melody audio, and is further attached to the vocal audio in the song audio corresponding to the target melody audio, thereby further improving the accuracy of the lip movement information obtained based on the target melody audio.

[0092] In some optional implementations of this embodiment, the above-mentioned execution entity adjusts the audio style of the initial audio in the following manner: for each rhythm point group in the rhythm point information, the audio segment adapted to the rhythm point group in the adapted initial audio is adjusted according to the audio style information corresponding to the rhythm point group to obtain the target audio.

[0093] The rhythm point group represents a cluster of adjacent rhythm points in the target melody audio. For the corresponding rhythm point group and the audio segment in the initial audio, the audio segment is adjusted according to the audio style information corresponding to the rhythm point group.

[0094] In this implementation, during the process of audio style adjustment, style adjustment is performed part by part, so that the obtained target audio is more compatible with the target melody audio, and further fits into the song audio corresponding to the target melody audio.

[0095] In some optional implementations of this embodiment, the execution entity may determine the audio features of the target audio in the following manner:

[0096] First, the target coding information of the selected speaker is determined from the preset coding set.

[0097] The coding information in the preset coding set represents different speakers corresponding to different melody audios.

[0098] Second, the target audio is normalized according to the speech attribute information of the target speaker and the audio attribute information of the target audio corresponding to the target coding information.

[0099] The target speaker's voice attribute information may be information characterizing the speaker's pronunciation characteristics. For example, the voice attribute information may include voice attribute information such as pitch and mean pitch. The target audio's audio attribute information may be information characterizing the target audio's audio characteristics. For example, the audio attribute information may include audio attribute information such as pitch and mean pitch.

[0100] Second, the normalized autocorrelation features of the target audio are extracted as audio features.

[0101] Autocorrelation feature, also called sequence correlation feature, is the cross-correlation between a speech signal and itself at different time points.

[0102] For example, for each 385-millisecond audio window in the target audio, the audio is divided into 64 audio segments. For each audio segment, the autocorrelation coefficient of 32 components is extracted to form a 64×32-dimensional autocorrelation feature as the audio feature of the target audio. The autocorrelation feature is selected because it is strongly correlated with speech phonemes, which are strongly correlated with lip shapes, thus improving the generalization of the information determination model.

[0103] Based on the 64×32 dimensional autocorrelation features and the 32-length target coding information, 65×32 dimensional information can be obtained as the input of the lip movement information determination model.

[0104] In this implementation, the target audio is normalized based on the speech attribute information of the target speaker and the audio attribute information of the target audio, and then the autocorrelation features of the normalized target audio are extracted, thereby further improving the accuracy of the obtained lip movement information.

[0105] In some optional implementations of this embodiment, the execution entity may perform the second step in the following manner:

[0106] First, the target maximum value is determined based on the maximum pitch of the target speaker and the maximum pitch of the target audio; then, the target mean value is determined based on the mean pitch of the target speaker and the mean pitch of the target audio; finally, the target audio is normalized based on the target maximum value and the target mean value.

[0107] As an example, based on the maximum pitch of the target speaker and the maximum pitch of the target audio, the mean or weighted average of the two is determined as the target maximum; similarly, based on the mean pitch of the target speaker and the mean pitch of the target audio, the mean or weighted average of the two is determined as the target mean.

[0108] Furthermore, the above execution body can normalize the target audio by the following formula:

[0109]

[0110] Among them, xn and x represent the normalized audio and the audio before normalization, respectively, and xmean and xmax represent the target mean and target maximum, respectively.

[0111] This implementation provides a specific implementation method for normalizing the target audio, thereby improving the feasibility and accuracy of the normalization process.

[0112] Continue to refer Figure 7 , shows a schematic process 700 of another embodiment of the digital human driving method according to the present disclosure, including the following steps:

[0113] Step 701: Based on the acquired rhythm point information of the target melody audio, determine the audio segments of the lyrics text to be matched from the target melody audio to obtain an audio segment set.

[0114] Step 702: Determine audio frame features of a target audio frame of an audio segment in the audio segment set.

[0115] The audio frame features include audio features of a target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment.

[0116] Step 703: Based on the audio frame features, a search is performed in the K-dimensional tree to determine the target lyrics text corresponding to the target melody audio.

[0117] Step 704: Generate initial audio based on the target lyrics text.

[0118] Step 705: Determine the pronunciation time range of each text unit in the target lyrics text in the initial audio.

[0119] Step 706: Adapt the initial audio with reference to the target melody audio according to the rhythm point information and the pronunciation time range.

[0120] Step 707: Determine the audio style information of the target melody audio according to the rhythm point information.

[0121] Step 708: Adjust the adapted initial audio according to the audio style information to obtain the target audio.

[0122] Step 709: Determine the target encoding information of the selected speaker from the preset encoding set.

[0123] The coding information in the preset coding set represents different speakers corresponding to different melody audios.

[0124] Step 710 : normalize the target audio according to the speech attribute information of the target speaker and the audio attribute information of the target audio corresponding to the target coding information.

[0125] Step 711: extract the normalized autocorrelation features of the target audio as audio features.

[0126] Step 712: Obtain lip movement information through a pre-trained lip movement information determination model based on the audio features and target coding information.

[0127] Among them, the lip movement information determination model is used to characterize the correspondence between audio features, target coding information and lip movement information.

[0128] Step 713: Use the target audio as the digital human's voice and the lip movement information as the digital human's lip movement driving information to drive the digital human to run.

[0129] Please continue to refer to Figure 8 , showing Figure 7 The information flow diagram 800 of the corresponding embodiment.

[0130] It can be seen from this embodiment that Figure 2 Compared with the corresponding embodiments, the process 700 of the digital human driving method in this embodiment specifically illustrates the process of obtaining the target audio, the process of obtaining the audio features of the target audio, and the process of driving the digital human to run through lip movement information and target songs, thereby improving the accuracy of the lip movement information and the lip movement display effect of the digital human, and realizing the effect of the digital human singing based on the melody audio.

[0131] Continue to refer Figure 9 , shows a schematic process 900 of an embodiment of a method for training a lip movement information determination model according to the present disclosure, comprising the following steps:

[0132] Step 901 : Based on the acquired rhythm point information of the melody audio, a search is performed from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio.

[0133] In this embodiment, the execution subject of the training method of the lip movement information determination model (for example, Figure 1The terminal device or server in the system can obtain melody audio remotely or locally based on a wired network connection method or a wireless network connection method, and based on the rhythm point information of the obtained melody audio, search from a K-dimensional tree that represents the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio.

[0134] Melody audio is the audio that only includes melody data and does not include singing voice data.In the field of music, melody audio can be the melody data in the song audio that any singer releases.

[0135] By using the melody audio automatic dotting technology, the positions of all rhythm points in the melody audio can be determined to determine the rhythm point information of the melody audio.

[0136] After obtaining the melody's rhythm information, the execution entity can search a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio. The K-dimensional tree is a binary tree data structure that uses a K-dimensional hyperplane to partition a set of points in a K-dimensional space.

[0137] In this implementation, the execution entity may construct a vector pair based on the corresponding feature vectors of the lyrics text and the feature vectors of the melody audio, and then establish a KD-tree (K-dimensional tree) based on the multiple vector pairs.

[0138] Step 902: Generate initial audio according to the target lyrics text.

[0139] In this embodiment, the execution entity may generate initial audio according to the target lyrics text.

[0140] As an example, the execution subject may obtain the machine audio corresponding to the lyrics text as the initial audio through TTS technology.

[0141] Step 903: Adjust the initial audio according to the rhythm point information to obtain a sample audio, and determine the audio features of the sample audio.

[0142] In this embodiment, the execution entity may adjust the initial audio according to the rhythm point information to obtain the sample audio, and determine the audio features of the sample audio.

[0143] As an example, the execution entity may adjust the initial audio based on the rhythm represented by the rhythm point information of the target melody audio so that the rhythm of the initial audio matches the rhythm of the melody audio. Furthermore, the execution entity may extract features from the sample audio using a feature extraction network to obtain audio features of the sample audio. As an example, the feature extraction network may be a network model such as a convolutional neural network or a recurrent neural network.

[0144] It should be noted that, for one or more melody audios, the above execution entity can obtain multiple sample audios.

[0145] Step 904: Generate training samples based on the audio features and lip movement information corresponding to the same sample audio to obtain a training sample set.

[0146] In this embodiment, the execution entity may generate training samples based on the audio features and lip movement information corresponding to the same sample audio to obtain a training sample set, wherein the lip movement information is obtained based on the initial video that matches the melody audio.

[0147] As an example, the audio features and lip movement information corresponding to the same sample audio are used as sample feature data and labels of the training samples respectively, so as to combine multiple training samples to obtain a training sample set.

[0148] It should be noted that the matching melody audio and video can be data corresponding to different speakers. The melody audio and video corresponding to the same speaker are matched. The melody audio has matching human voice audio. The matching of the melody audio and video means that for each melody audio frame in the melody audio, the video includes a video frame representing the facial movements when the human voice audio frame corresponding to the melody audio frame is uttered.

[0149] As an example, the initial melody audio is the melody audio of a song, and the initial video is a video of a singer singing the song.

[0150] The above-mentioned execution subject can first perform frame-by-frame face position detection on the speaker object in the video, and then use a three-dimensional parameterized face model to reconstruct the facial expression of the face part to obtain lip movement information.

[0151] Step 905: Using a machine learning algorithm, taking the audio features as input and the lip movement information labels corresponding to the input audio features as the desired output, the initial lip movement information determination model is trained to obtain a trained lip movement information determination model.

[0152] In this embodiment, the above-mentioned execution entity can adopt a machine learning algorithm, take the audio features as input, and take the lip movement information labels corresponding to the input audio features as the expected output, train the initial lip movement information determination model, and obtain the trained lip movement information determination model.

[0153] During the training process, the above-mentioned execution entity can select untrained training samples from the training sample set, use the audio features in the training samples as the input of the initial lip movement information determination model, and obtain actual lip movement information; then determine the loss between the actual lip movement information and the lip movement information label, and then determine the gradient information based on the loss information, and use the gradient descent method to update the parameters of the initial lip movement information determination model.

[0154] By cyclically executing the above training operation, in response to reaching a preset end condition, a trained lip movement information determination model is obtained. The preset end condition may be, for example, that the training time exceeds a preset time threshold, the number of training times exceeds a preset number threshold, or the training loss converges.

[0155] In some optional implementations, the training samples also include encoded information about the speaker corresponding to the sample audio. In this implementation, the execution entity can employ a machine learning algorithm, taking the audio features and encoded information as input and the lip movement information labels corresponding to the input audio features and encoded information as the desired output, to train a lip movement information determination model.

[0156] The trained lip movement information determination model is used to implement the digital human driving method in the above embodiment 200, and the coding information of the speaker corresponding to different training samples constitutes a preset coding set.

[0157] In this implementation, by encoding and identifying different speakers, the lip movement information determination model is compatible with different speakers during the training process, thereby improving the generalization effect of the lip movement information determination model on the input speech and the accuracy of the obtained lip movement information.

[0158] In some optional implementations of this embodiment, the execution entity may perform step 901 as follows:

[0159] First, based on the acquired rhythm point information of the melody audio, an audio segment of the lyrics text to be matched is determined from the melody audio to obtain an audio segment set.

[0160] As an example, the execution entity can determine the rhythmic features of the melody audio based on the rhythmic information and perform a rhythmic operation by calculating the locations in the melody audio where the tempo changes most dramatically. Based on the rhythmic operation, the melody audio can be divided into multiple audio segments, each of which is to be matched with text. All audio segments in the melody audio are combined into an audio segment set.

[0161] Second, audio frame features of a target audio frame of an audio segment in the audio segment set are determined.

[0162] The audio frame features include audio features of a target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment.

[0163] As an example, for each audio segment in the obtained audio segment set, the audio features such as the numerical value and speed of the current audio frame (target audio frame) are determined every preset number of audio frames, as well as the numerical value and speed of the audio frames at the start position and end position of the target audio segment to which the target audio frame belongs, and the numerical value and speed of the audio frames at the start position and end position of the next audio segment of the target audio segment. The obtained audio features of the target audio frame, the audio features of the target audio segment to which the target audio frame belongs, and the audio features of the audio segments adjacent to the target audio segment can be used as a set of vectors to represent the audio frame features of the current audio frame for matching operations in a K-dimensional tree.

[0164] Third, based on the audio frame features, a search is performed from the K-dimensional tree to determine the target lyrics text corresponding to the melody audio.

[0165] In this implementation, the audio segment of the melody audio to be matched with the lyrics text is first determined, and then the audio frame features of the target audio frame of the audio segment are determined to search and determine the target lyrics text in the K-dimensional tree, thereby improving the accuracy of the determined lyrics text.

[0166] In some optional implementations of this embodiment, for each type of melody audio, the execution entity or an electronic device communicatively connected to the execution entity is provided with a K-dimensional tree corresponding to the type of melody audio. In this implementation, the execution entity may perform the third step as follows:

[0167] First, the target category to which the melody audio belongs is determined; then, based on the audio frame features, a search is performed from the K-dimensional tree corresponding to the target category to determine the target lyrics text corresponding to the melody audio.

[0168] As an example, the execution subject may determine the target category of the melody audio using a pre-trained melody classification model. The melody classification model is used to determine the target category of the melody audio and may be trained based on a neural network such as a convolutional neural network or a residual neural network.

[0169] As another example, the execution entity can segment the melody audio into audio segments of a preset duration (e.g., 2 seconds), input the audio segments into a wav2vec model, and the model can output a feature vector representing the characteristics of the currently input audio segment. This feature vector is then used to perform a distance measurement with multiple pre-set category vectors, and the category represented by the category vector with the closest distance is determined as the target category to which the target melody audio belongs.

[0170] In this implementation, each type of melody audio corresponds to a K-dimensional tree, so that the above-mentioned execution entity can search according to the K-dimensional tree corresponding to the type of melody audio, thereby improving the pertinence and accuracy of the lyrics text search process.

[0171] In some optional implementations of this embodiment, the execution entity may adjust the initial audio according to the rhythm point information in the following manner:

[0172] First, the pronunciation time range of each text unit in the target lyrics text in the initial audio is determined; based on the rhythm point information and the pronunciation time range, the initial audio is adapted with reference to the melody audio.

[0173] Specifically, the execution entity may use vosk (an open source automatic speech recognition tool) to detect the pronunciation time range of each text unit in the lyrics text in the original audio. The pronunciation time range represents the pronunciation duration of the text unit in the audio.

[0174] As an example, the above-mentioned execution entity can first determine the pronunciation time range of each single word in the lyrics text in the initial audio; then, segment the lyrics text to obtain the segmented text units included in the lyrics text; finally, for each text unit, the pronunciation time ranges of the single words included in the text unit are combined to obtain the pronunciation time range corresponding to the text unit.

[0175] The rhythmic information in the melody audio is often closely related to the text units in the lyrics. The above-mentioned execution entity can determine the matching relationship between the rhythmic information and the text units; then, by stretching or compressing the pronunciation time range in the original audio to match the rhythmic duration in the melody audio.

[0176] In this implementation, a method is provided for adjusting the initial audio based on the rhythm point information of the melody audio and the pronunciation time range of the initial audio, so that the adjusted initial audio is more consistent with the melody audio, thereby improving the accuracy of the lip movement information obtained based on the melody audio.

[0177] In some optional implementations of this embodiment, the execution entity may further adjust the initial audio in the following manner:

[0178] First, the audio style information of the melody audio is determined according to the rhythm point information.

[0179] The audio style information can be specifically expressed as audio attribute information such as the pitch and frequency of the melody audio.

[0180] Then, the adapted initial audio is adjusted according to the audio style information to obtain the target audio.

[0181] In this implementation, the execution subject may refer to the audio style information of the melody audio to adjust the pitch, frequency and other information of the initial audio so that the audio style information of the target audio matches the audio style information of the melody audio.

[0182] In this implementation method, on the basis of adjusting the pronunciation time range of each text unit in the initial audio, the initial audio is further adjusted with reference to the audio style information of the melody audio, so that the obtained target audio is more compatible with the melody audio, and is further attached to the vocal audio in the song audio corresponding to the melody audio, thereby further improving the accuracy of the lip movement information obtained based on the melody audio.

[0183] In some optional implementations of this embodiment, the above-mentioned execution entity adjusts the audio style of the initial audio in the following manner: for each rhythm point group in the rhythm point information, the audio segment adapted to the rhythm point group in the adapted initial audio is adjusted according to the audio style information corresponding to the rhythm point group to obtain the target audio.

[0184] The rhythm point group represents a cluster of adjacent rhythm points in the melody audio. For the corresponding rhythm point group and a portion of the original audio, the portion of the audio is adjusted according to the audio style information corresponding to the rhythm point group.

[0185] In this implementation, during the process of audio style adjustment, style adjustment is performed part by part, so that the obtained target audio is more compatible with the melody audio and further fits into the song audio corresponding to the melody audio.

[0186] In some optional implementations of this embodiment, the execution entity may determine the audio features of the target audio in the following manner:

[0187] First, the target coding information of the selected speaker is determined from the preset coding set.

[0188] The coding information in the preset coding set represents different speakers corresponding to different melody audios.

[0189] Second, the target audio is normalized according to the speech attribute information of the target speaker and the audio attribute information of the target audio corresponding to the target coding information.

[0190] The target speaker's voice attribute information may be information characterizing the speaker's pronunciation characteristics. For example, the voice attribute information may include voice attribute information such as pitch and mean pitch. The target audio's audio attribute information may be information characterizing the target audio's audio characteristics. For example, the audio attribute information may include audio attribute information such as pitch and mean pitch.

[0191] Second, the normalized autocorrelation features of the target audio are extracted as audio features.

[0192] Autocorrelation feature, also called sequence correlation feature, is the cross-correlation between a speech signal and itself at different time points.

[0193] For example, for each 385-millisecond audio window in the target audio, the audio is divided into 64 audio segments. For each audio segment, the autocorrelation coefficient of 32 components is extracted to form a 64×32-dimensional autocorrelation feature as the audio feature of the target audio. The autocorrelation feature is selected because it is strongly correlated with speech phonemes, which are strongly correlated with lip shapes, thus improving the generalization of the information determination model.

[0194] Based on the 64×32 dimensional autocorrelation features and the 32-length target coding information, 65×32 dimensional information can be obtained as the input of the lip movement information determination model.

[0195] In this implementation, the target audio is normalized based on the speech attribute information of the target speaker and the audio attribute information of the target audio, and then the autocorrelation features of the normalized target audio are extracted, thereby further improving the accuracy of the obtained lip movement information.

[0196] In some optional implementations of this embodiment, the execution entity may perform the second step in the following manner:

[0197] First, the target maximum value is determined based on the maximum pitch of the target speaker and the maximum pitch of the target audio; then, the target mean value is determined based on the mean pitch of the target speaker and the mean pitch of the target audio; finally, the target audio is normalized based on the target maximum value and the target mean value.

[0198] As an example, based on the maximum pitch of the target speaker and the maximum pitch of the target audio, the mean or weighted average of the two is determined as the target maximum; similarly, based on the mean pitch of the target speaker and the mean pitch of the target audio, the mean or weighted average of the two is determined as the target mean.

[0199] Furthermore, the above execution body can normalize the target audio by the following formula:

[0200]

[0201] Among them, x n , x represent the normalized audio and the audio before normalization, respectively, x mean 、x max represent the target mean and target maximum, respectively.

[0202] This implementation provides a specific implementation method for normalizing the target audio, thereby improving the feasibility and accuracy of the normalization process.

[0203] Continue to refer Figure 10 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a digital human driving device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0204] like Figure 10 As shown, the digital human driving device includes: a first determination unit 1001, configured to search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text based on the rhythm point information of the acquired target melody audio, and determine the target lyrics text corresponding to the target melody audio; a first generation unit 1002, configured to generate initial audio according to the target lyrics text; a second determination unit 1003, configured to adjust the initial audio according to the rhythm point information, obtain the target audio, and determine the audio features of the target audio; a third determination unit 1004, configured to determine the lip movement information according to the audio features; a driving unit, configured to drive the digital human to run with the target audio as the digital human's voice and the lip movement information as the lip movement driving information of the digital human.

[0205] In some optional implementations of this embodiment, the first determination unit 1001 is further configured to: determine the audio segment of the lyrics text to be matched from the target melody audio based on the acquired rhythm point information of the target melody audio, and obtain an audio segment set; determine the audio frame features of the target audio frame of the audio segment in the audio segment set, wherein the audio frame features include the audio features of the target audio frame, the audio features of the target audio segment to which the target audio frame belongs, and the audio features of the audio segments adjacent to the target audio segment; based on the audio frame features, search from the K-dimensional tree to determine the target lyrics text corresponding to the target melody audio.

[0206] In some optional implementations of this embodiment, different types of melody audio are correspondingly provided with different K-dimensional trees, and the first determination unit 1001 is further configured to: determine the target type to which the target melody audio belongs; based on the audio frame features, search from the K-dimensional tree corresponding to the target type to determine the target lyrics text corresponding to the target melody audio.

[0207] In some optional implementations of this embodiment, the second determination unit 1003 is further configured to: determine the pronunciation time range of each text unit in the target lyrics text in the initial audio; and adapt the initial audio with reference to the target melody audio based on the rhythm point information and the pronunciation time range.

[0208] In some optional implementations of this embodiment, the second determining unit 1003 is further configured to: determine audio style information of the target melody audio according to the rhythm point information; and adjust the adapted initial audio according to the audio style information to obtain the target audio.

[0209] In some optional implementations of this embodiment, the second determination unit 1003 is further configured to: for each rhythm point group in the rhythm point information, adjust the audio segment in the adapted initial audio that is adapted to the rhythm point group according to the audio style information corresponding to the rhythm point group to obtain the target audio.

[0210] In some optional implementations of this embodiment, the second determination unit 1003 is further configured to: determine the target coding information of the selected speaker from a preset coding set, wherein the coding information in the preset coding set represents different speakers corresponding to different melody audio; normalize the target audio according to the speech attribute information of the target speaker corresponding to the target coding information and the audio attribute information of the target audio; and extract the autocorrelation features of the normalized target audio as audio features.

[0211] In some optional implementations of this embodiment, the second determination unit 1003 is further configured to: determine the target maximum value based on the maximum pitch of the target speaker and the maximum pitch of the target audio; determine the target mean based on the mean pitch of the target speaker and the mean pitch of the target audio; and normalize the target audio based on the target maximum value and the target mean.

[0212] In this embodiment, a digital human driving device is provided, which performs a K-dimensional tree search based on the rhythm point information of the acquired melody audio to determine the lyrics text corresponding to the melody audio, thereby improving the accuracy of the determined lyrics text; and adjusts the initial audio generated based on the lyrics text according to the rhythm point information, so that the target audio obtained by adjusting the initial audio is more consistent with the original singing audio corresponding to the melody audio; then, the audio features of the adjusted target audio are obtained, and the lip movement information is determined according to the audio features, and the digital human is driven to operate according to the lip movement information and the target audio, thereby improving the display effect of the digital human singing driven by the melody audio.

[0213] Continue to refer Figure 11 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a training device for determining a lip movement information model. Figure 9 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0214] like Figure 11As shown, the training device for the lip movement information determination model includes: a fourth determination unit 1101, which is configured to search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text based on the rhythm point information of the acquired melody audio, and determine the target lyrics text corresponding to the melody audio; a second generation unit 1102, which is configured to generate an initial audio according to the target lyrics text; a fifth determination unit 1103, which is configured to adjust the initial audio according to the rhythm point information, obtain a sample audio, and determine the audio features of the sample audio; a third generation unit 1104, which is configured to generate a training sample based on the audio features and lip movement information corresponding to the same sample audio, to obtain a training sample set, wherein the lip movement information is obtained based on the initial video matching the melody audio; a training unit 1105, which is configured to adopt a machine learning algorithm, take the audio features as input, and take the lip movement information labels corresponding to the input audio features as the expected output, to train the initial lip movement information determination model, and obtain a trained lip movement information determination model.

[0215] In some optional implementations of this embodiment, the fourth determination unit 1101 is further configured to: determine the audio segment of the lyrics text to be matched from the melody audio based on the rhythm point information of the acquired melody audio, and obtain an audio segment set; determine the audio frame features of the target audio frame of the audio segment in the audio segment set, wherein the audio frame features include the audio features of the target audio frame, the audio features of the target audio segment to which the target audio frame belongs, and the audio features of the audio segments adjacent to the target audio segment; based on the audio frame features, search from the K-dimensional tree to determine the target lyrics text corresponding to the melody audio.

[0216] In some optional implementations of this embodiment, different K-dimensional trees are provided for different types of melody audio, and the fourth determination unit 1101 is further configured to: determine the target category to which the melody audio belongs; and based on the audio frame features, search from the K-dimensional tree corresponding to the target category to determine the target lyrics text corresponding to the melody audio.

[0217] In some optional implementations of this embodiment, the fifth determination unit 1103 is further configured to: determine the pronunciation time range of each text unit in the target lyrics text in the initial audio; and adapt the initial audio with reference to the melody audio based on the rhythm point information and the pronunciation time range.

[0218] In some optional implementations of this embodiment, the fifth determining unit 1103 determines audio style information of the melody audio according to the rhythm point information; and adjusts the adapted initial audio according to the audio style information to obtain sample audio.

[0219] In some optional implementations of this embodiment, the fifth determination unit 1103 is further configured to: for each rhythm point group in the rhythm point information, adjust the audio segment in the adapted initial audio that is adapted to the rhythm point group according to the audio style information corresponding to the rhythm point group, and obtain the sample audio.

[0220] In some optional implementations of this embodiment, the fifth determination unit 1103 is further configured to: determine the target coding information of the selected speaker from a preset coding set, wherein the coding information in the preset coding set represents different speakers corresponding to different melody audio; normalize the sample audio according to the speech attribute information of the target speaker corresponding to the target coding information and the audio attribute information of the sample audio; and extract the autocorrelation features of the normalized sample audio as audio features.

[0221] In some optional implementations of this embodiment, the fifth determination unit 1103 is further configured to: determine the target maximum value based on the maximum pitch of the target speaker and the maximum pitch of the sample audio; determine the target mean based on the mean pitch of the target speaker and the mean pitch of the sample audio; and normalize the sample audio based on the target maximum value and the target mean.

[0222] In this embodiment, a training device for a lip movement information determination model is provided, which improves the generalization effect of the lip movement information determination model on the input speech and the accuracy of the obtained lip movement information.

[0223] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it can implement the digital human driving method and the lip movement information determination model training method described in any of the above embodiments.

[0224] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the digital human driving method and the lip movement information determination model training method described in any of the above embodiments when executed.

[0225] The embodiments of the present disclosure provide a computer program product, which, when executed by a processor, can implement the digital human driving method and the lip movement information determination model training method described in any of the above embodiments.

[0226] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0227] like Figure 12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the device 1200 can also be stored in the RAM 1203. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0228] Various components in device 1200 are connected to I / O interface 1205, including an input unit 1206, such as a keyboard and mouse; an output unit 1207, such as various types of displays and speakers; a storage unit 1208, such as a magnetic disk and optical disk; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0229] The computing unit 1201 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the digital human driving method. For example, in some embodiments, the digital human driving method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the digital human driving method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the digital human driving method in any other appropriate manner (for example, by means of firmware).

[0230] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0231] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0232] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0233] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0234] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0235] A computer system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS). It may also be a server in a distributed system or a server integrated with blockchain.

[0236] According to the technical solution of the embodiment of the present disclosure, a digital human driving method is provided, which performs a K-dimensional tree search based on the rhythm point information of the acquired melody audio to determine the lyrics text corresponding to the melody audio, thereby improving the accuracy of the determined lyrics text; and adjusts the initial audio generated based on the lyrics text according to the rhythm point information, so that the target audio obtained by adjusting the initial audio is more consistent with the original singing audio corresponding to the melody audio; then, the audio features of the adjusted target audio are obtained, and the lip movement information is determined according to the audio features, and the digital human is driven to operate according to the lip movement information and the target audio, thereby improving the display effect of driving the digital human to sing based on the melody audio.

[0237] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided by this disclosure can be achieved. This is not a limitation herein.

[0238] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A digital human driving method, comprising: Based on the acquired rhythm point information of the target melody audio, determining audio segments of the lyrics text to be matched from the target melody audio to obtain an audio segment set, wherein the rhythm point information is determined by an automatic melody audio notation technology; determining audio frame features of a target audio frame of an audio segment in the audio segment set, wherein the audio frame features include audio features of the target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment; Based on the audio frame features, searching from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio; generating an initial audio according to the target lyrics text; Adjusting the initial audio according to the rhythm point information to obtain target audio, and determining audio features of the target audio; Determining lip movement information according to audio features of the target audio; The target audio is used as the digital human's voice, and the lip movement information is used as the digital human's lip movement driving information to drive the digital human to run.

2. The method according to claim 1, wherein Different types of melody audio have different K-dimensional trees, and The step of searching the K-dimensional tree based on the audio frame features to determine a target lyrics text corresponding to the target melody audio includes: determining a target category to which the target melody audio belongs; Based on the audio frame features, a search is performed from the K-dimensional tree corresponding to the target category to determine the target lyrics text corresponding to the target melody audio.

3. The method according to claim 1, wherein The adjusting the initial audio according to the rhythm point information includes: Determining a pronunciation time range of each text unit in the target lyrics text in the initial audio; The initial audio is adapted with reference to the target melody audio according to the rhythm point information and the pronunciation time range.

4. The method according to claim 3, wherein: The adjusting the initial audio according to the rhythm point information to obtain the target audio further includes: determining audio style information of the target melody audio according to the rhythm point information; The adapted initial audio is adjusted according to the audio style information to obtain the target audio.

5. The method according to claim 4, wherein The adjusting the adapted initial audio according to the audio style information to obtain the target audio includes: For each rhythm point group in the rhythm point information, the audio segment adapted to the rhythm point group in the adapted initial audio is adjusted according to the audio style information corresponding to the rhythm point group to obtain the target audio.

6. The method according to claim 1, wherein The determining of the audio features of the target audio includes: Determining target coding information of the selected speaker from a preset coding set, wherein the coding information in the preset coding set represents different speakers corresponding to different melody audios; Normalizing the target audio according to the speech attribute information of the target speaker and the audio attribute information of the target audio corresponding to the target coding information; Extract the normalized autocorrelation features of the target audio as the audio features.

7. The method according to claim 6, wherein: Normalizing the target audio according to the speech attribute information of the target speaker and the audio attribute information of the target audio corresponding to the target coding information includes: Determining a target maximum value based on the maximum pitch of the target speaker and the maximum pitch of the target audio; Determining a target mean value based on the pitch mean value of the target speaker and the pitch mean value of the target audio; The target audio is normalized according to the target maximum value and the target mean value.

8. A method for training a lip movement information determination model, comprising: Based on the acquired rhythm point information of the melody audio, an audio segment of the lyrics text to be matched is determined from the melody audio to obtain an audio segment set, wherein the rhythm point information is determined by an automatic melody audio notation technology; determining audio frame features of a target audio frame of an audio segment in the audio segment set, wherein the audio frame features include audio features of the target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment; Based on the audio frame features, searching from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio; generating an initial audio according to the target lyrics text; Adjusting the initial audio according to the rhythm point information to obtain a sample audio, and determining audio features of the sample audio; Generating training samples based on audio features and lip movement information corresponding to the same sample audio to obtain a training sample set, wherein the lip movement information is obtained based on an initial video matching the melody audio; A machine learning algorithm is used, with audio features as input and lip movement information labels corresponding to the input audio features as expected output, to train an initial lip movement information determination model and obtain a trained lip movement information determination model.

9. The method according to claim 8, wherein Different types of melody audio have different K-dimensional trees, and The step of searching the K-dimensional tree based on the audio frame features to determine a target lyric text corresponding to the melody audio includes: determining a target category to which the melody audio belongs; Based on the audio frame features, a search is performed from the K-dimensional tree corresponding to the target category to determine the target lyrics text corresponding to the melody audio.

10. The method according to claim 8, wherein The adjusting the initial audio according to the rhythm point information includes: Determining a pronunciation time range of each text unit in the target lyrics text in the initial audio; The initial audio is adapted with reference to the melody audio according to the rhythm point information and the pronunciation time range.

11. The method according to claim 10, wherein: The step of adjusting the initial audio according to the rhythm point information to obtain a sample audio further includes: determining audio style information of the melody audio according to the rhythm point information; The sample audio is obtained by adjusting the adapted initial audio according to the audio style information.

12. The method according to claim 11, wherein The adjusting the adapted initial audio according to the audio style information to obtain the sample audio includes: For each rhythm point group in the rhythm point information, the audio segment adapted to the rhythm point group in the adapted initial audio is adjusted according to the audio style information corresponding to the rhythm point group to obtain the sample audio.

13. The method according to claim 8, wherein The determining of the audio features of the sample audio includes: Determining target coding information of the selected speaker from a preset coding set, wherein the coding information in the preset coding set represents different speakers corresponding to different melody audios; Normalizing the sample audio according to the speech attribute information of the target speaker corresponding to the target coding information and the audio attribute information of the sample audio; Extract the normalized autocorrelation features of the sample audio as the audio features.

14. The method according to claim 13, wherein Normalizing the sample audio according to the speech attribute information of the target speaker corresponding to the target coding information and the audio attribute information of the sample audio includes: Determining a target maximum value based on the maximum pitch of the target speaker and the maximum pitch of the sample audio; Determining a target mean value based on the pitch mean value of the target speaker and the pitch mean value of the sample audio; The sample audio is normalized according to the target maximum value and the target mean value.

15. A digital human driving device, comprising: The first determining unit is configured to determine, based on the acquired rhythm point information of the target melody audio, an audio segment of the target melody audio with the lyrics text to be matched, to obtain an audio segment set, wherein the rhythm point information is determined by a melody audio automatic dotting technology; determine audio frame features of a target audio frame of an audio segment in the audio segment set, wherein the audio frame features include audio features of the target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment; and based on the audio frame features, search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the target melody audio; A first generating unit is configured to generate an initial audio according to the target lyrics text; a second determining unit configured to adjust the initial audio according to the rhythm point information to obtain target audio, and determine audio features of the target audio; a third determining unit, configured to determine lip movement information according to the audio feature; The driving unit is configured to drive the digital human to move by using the target audio as the digital human's voice and the lip movement information as the digital human's lip movement driving information.

16. A training device for a lip movement information determination model, comprising: The fourth determining unit is configured to determine, based on the acquired rhythm point information of the melody audio, an audio segment to be matched with the lyrics text from the melody audio, to obtain an audio segment set, wherein the rhythm point information is determined by a melody audio automatic dotting technology; determine audio frame features of a target audio frame of an audio segment in the audio segment set, wherein the audio frame features include audio features of the target audio frame, audio features of a target audio segment to which the target audio frame belongs, and audio features of audio segments adjacent to the target audio segment; and based on the audio frame features, search from a K-dimensional tree representing the correspondence between the melody audio and the lyrics text to determine the target lyrics text corresponding to the melody audio; A second generating unit is configured to generate an initial audio according to the target lyrics text; a fifth determining unit configured to adjust the initial audio according to the rhythm point information to obtain a sample audio, and determine audio features of the sample audio; a third generating unit configured to generate training samples based on audio features and lip movement information corresponding to the same sample audio to obtain a training sample set, wherein the lip movement information is obtained based on an initial video matching the melody audio; The training unit is configured to adopt a machine learning algorithm, take the audio features as input, take the lip movement information labels corresponding to the input audio features as the expected output, train the initial lip movement information determination model, and obtain the trained lip movement information determination model.

17. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

19. A computer program product comprising: A computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images