Lip synchronization evaluation method and apparatus for digital human, and computing device cluster

By using an automated lip-sync evaluation method and leveraging automatic speech recognition and time alignment technologies, the problem of time-consuming and labor-intensive manual scoring is solved, enabling rapid and objective evaluation and optimization of lip-sync accuracy in digital human videos.

WO2024244469A1PCT designated stage expired Publication Date: 2024-12-05HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/070777
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-07-10
Filing Date
2024-01-05
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

In existing technologies, digital human lip-sync evaluation mainly relies on manual scoring, which is time-consuming, labor-intensive, and lacks objective evidence. It is difficult to provide fine-grained lip-sync accuracy at the character, word, and sentence levels, making it difficult for digital human video producers to quickly locate areas for improvement.

Method used

By automatically performing speech recognition on the audio of digital human videos, the audio is segmented into multiple sub-texts, and the lip-sync accuracy of each sub-video is calculated. The accuracy is then aligned using timestamps, and a graph or table is output to display the accuracy. Users can select the calculation method and adjust the production process.

Benefits of technology

It enables rapid and objective identification of areas in digital human videos where lip-sync needs improvement, thereby enhancing the efficiency and accuracy of lip-sync accuracy evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024070777_05122024_PF_FP_ABST
    Figure CN2024070777_05122024_PF_FP_ABST
Patent Text Reader

Abstract

A lip synchronization evaluation method for a digital human. The method comprises: acquiring a digital-human video input by a user; converting audio in the digital-human video into first text, and segmenting the first text, so as to obtain a plurality of pieces of sub-text; determining the degree of matching between mouth shapes of a digital human in each sub-video included in the digital-human video and the audio in the corresponding sub-video, so as to obtain a plurality of lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; on the basis of timestamps of the sub-text and timestamps of the sub-videos corresponding to the lip synchronization accuracy rates, temporally aligning the sub-text with the lip synchronization accuracy rates, so as to obtain lip synchronization accuracy rates corresponding to the sub-text; and outputting the lip synchronization accuracy rates corresponding to the sub-text. In this way, by means of the lip synchronization accuracy rates corresponding to sub-text, a user can quickly locate which part of a digital-human video requires improvement in terms of lip synchronization, and can optimize a production process of digital-human videos.
Need to check novelty before this filing date? Find Prior Art

Description

A digital human lip synchronization evaluation method, device and computing device cluster

[0001] This application claims priority to the Chinese patent applications filed with the State Intellectual Property Office of China on June 1, 2023, with application number 202310644516.9 and application name “A method for evaluating lip synchronization of digital humans”, and filed with the State Intellectual Property Office of China on July 10, 2023, with application number 202310840758.5 ​​and application name “A method, device and computing device cluster for evaluating lip synchronization of digital humans”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of information technology (IT), and in particular to a method, apparatus, and computing device cluster for evaluating lip synchronization of a digital human. Background Art

[0003] A digital human is a virtual character or AI entity generated through technologies such as artificial intelligence (AI). Currently, digital human generation methods mainly include voice-driven and visual-driven methods. Voice-driven methods use voice information to drive a given image or video to generate a digital human. Video-driven methods use video information to drive a given image or video to generate a digital human. Whether voice-driven or visual-driven, lip synchronization is a key consideration. This means ensuring that the voice information matches the digital human's lip movements, making the digital human as close to a real person as possible. Therefore, evaluating the lip synchronization of digital humans is a technical problem that needs to be solved urgently.

[0004] Summary of the Invention

[0005] The present application provides a method, apparatus, computing device cluster, computer storage medium and computer product for evaluating the lip synchronization of a digital human, which can realize the evaluation of the lip synchronization of a digital human.

[0006] In a first aspect, the present application provides a method for evaluating lip synchronization of a digital human, comprising: obtaining a digital human video input by a user; converting the audio in the digital human video into a first text, and segmenting the first text to obtain multiple sub-texts; determining the degree of matching between the digital human's mouth shape in each sub-video contained in the digital human video and the audio in the corresponding sub-video, to obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, aligning each sub-text with each lip synchronization accuracy rate in time to obtain the lip synchronization accuracy rate corresponding to each sub-text; and outputting the lip synchronization accuracy rate corresponding to each sub-text.

[0007] In this way, by calculating the lip synchronization accuracy of each sub-text corresponding to the audio contained in the digital human video, the user can quickly locate which parts of the digital human video need to be improved in lip synchronization based on the lip synchronization accuracy corresponding to each sub-text, so that the user can optimize the digital human video production process based on the parts that need to be improved to improve the lip synchronization accuracy of the digital human video.

[0008] In one possible implementation, the method may further include: determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each subtext; and outputting the lip synchronization accuracy of the digital human video. In this way, the user can understand the overall lip synchronization accuracy of the digital human video.

[0009] In one possible implementation, outputting the lip synchronization accuracy rate corresponding to each subtext includes: outputting a graph, wherein the vertical axis of the graph represents the lip synchronization accuracy rate, the horizontal axis of the graph represents time, the data points in the graph represent the lip synchronization accuracy rate corresponding to each subtext, and the subtext corresponding to each data point is displayed on the graph. In this way, the user can intuitively understand the lip synchronization accuracy rate corresponding to each subtext through the graph.

[0010] In one possible implementation, segmenting a first text to obtain multiple subtexts specifically includes: obtaining a text segmentation method input by a user, where the text segmentation method includes any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation, or paragraph segmentation; and segmenting the first text based on the text segmentation method to obtain multiple subtexts. In this way, the user can segment the text based on the segmentation granularity they desire, so that the resulting subtexts meet their needs.

[0011] In one possible implementation, the multiple subtexts include a first subtext. Obtaining the lip synchronization accuracy corresponding to the first subtext includes: if multiple sub-videos exist between the start and end timestamps of the first subtext, determining the lip synchronization accuracy corresponding to the first subtext based on the lip synchronization accuracy corresponding to the multiple sub-videos; and if one sub-video exists between the start and end timestamps of the first subtext, using the lip synchronization accuracy corresponding to the one sub-video as the lip synchronization accuracy corresponding to the first subtext.

[0012] In one possible implementation, determining the lip synchronization accuracy of the first subtext based on the lip synchronization accuracy of multiple sub-videos specifically includes: obtaining an accuracy calculation method input by the user; and processing the lip synchronization accuracy of the multiple sub-videos based on the accuracy calculation method selected by the user to obtain the lip synchronization accuracy of the first subtext. In this way, the user can select the appropriate accuracy calculation method based on their needs, so that the calculation result meets their needs.

[0013] In one possible implementation, the multiple subtexts include a second subtext. In this case, after outputting the lip sync accuracy corresponding to each subtext, the process also includes adjusting the digital human's mouth shape, skeletal points, or expression base at the time corresponding to the second subtext, or replacing one or more of the digital human's driving algorithms, where the lip sync accuracy corresponding to the second subtext falls below a preset accuracy threshold. This optimizes the digital human video production process and improves the lip sync accuracy of subsequently generated digital human videos.

[0014] In a second aspect, the present application provides a lip synchronization evaluation device for a digital human, comprising: an acquisition module and a processing module. The acquisition module is used to acquire a digital human video input by a user. The processing module is used to convert the audio in the digital human video into a first text, and segment the first text to obtain multiple sub-texts; and determine the degree of matching between the digital human's mouth shape in each sub-video contained in the digital human video and the audio in the corresponding sub-video, to obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video. The processing module is also used to align each sub-text with each lip synchronization accuracy rate in time based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, to obtain the lip synchronization accuracy rate corresponding to each sub-text, and to output the lip synchronization accuracy rate corresponding to each sub-text.

[0015] In a possible implementation, the processing module is further configured to: determine the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each sub-text; and output the lip synchronization accuracy of the digital human video.

[0016] In one possible implementation, when the processing module outputs the lip synchronization accuracy corresponding to each sub-text, it is specifically used to: output a curve graph, the vertical axis of the curve graph is the lip synchronization accuracy, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy corresponding to each sub-text, and the sub-text corresponding to each data point is displayed on the curve graph.

[0017] In one possible implementation, when the processing module segments the first text to obtain multiple sub-texts, it is specifically used to: obtain a text segmentation method input by the user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; based on the text segmentation method, segment the first text to obtain multiple sub-texts.

[0018] In one possible implementation, the multiple subtexts include a first subtext. In this case, when obtaining the lip synchronization accuracy rate corresponding to the first subtext, the processing module is specifically configured to: if multiple sub-videos exist between the start and end timestamps of the first subtext, determine the lip synchronization accuracy rate corresponding to the first subtext based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; and if one sub-video exists between the start and end timestamps of the first subtext, use the lip synchronization accuracy rate corresponding to the one sub-video as the lip synchronization accuracy rate corresponding to the first subtext.

[0019] In one possible implementation, when the processing module determines the lip synchronization accuracy corresponding to the first sub-text based on the lip synchronization accuracy corresponding to multiple sub-videos, the processing module is specifically used to: obtain the accuracy calculation method input by the user; and process the lip synchronization accuracy corresponding to the multiple sub-videos based on the accuracy calculation method to obtain the lip synchronization accuracy corresponding to the first sub-text.

[0020] In one possible implementation, the multiple subtexts include a second subtext. In this case, after outputting the lip synchronization accuracy rates corresponding to each subtext, the processing module is further configured to: adjust the digital human's mouth shape, skeletal points, or expression base at the time corresponding to the second subtext, or replace one or more items in the digital human's driving algorithm, wherein the lip synchronization accuracy rate corresponding to the second subtext is lower than a preset accuracy threshold.

[0021] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device, the computing device performs the method described in the first aspect or any possible implementation of the first aspect; alternatively, when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0023] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation of the first aspect; or, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0024] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0026] FIG1 is a schematic diagram of the architecture of a digital human lip synchronization evaluation system provided in an embodiment of the present application;

[0027] FIG2 is a schematic diagram of a lip synchronization evaluation result of a digital human provided in an embodiment of the present application;

[0028] FIG3 is a flow chart of a method for evaluating lip synchronization of a digital human provided in an embodiment of the present application;

[0029] FIG4 is a schematic structural diagram of a digital human lip synchronization evaluation device provided in an embodiment of the present application;

[0030] FIG5 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0031] FIG6 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0032] FIG7 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0034] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0035] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0036] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0037] Generally, in digital human videos, only when the digital human's lip movements match the audio information in the video can the digital human resemble a real person as closely as possible, thereby improving the digital human's performance. However, related technologies often rely on manual scoring to evaluate lip sync in digital human videos. This method is time-consuming and labor-intensive, and lacks objective evidence. Furthermore, this manual scoring method generally provides the lip sync accuracy of the entire digital human video, but it is difficult to provide lip sync accuracy at the character, word, sentence, or paragraph level. This makes it difficult for digital human creators to quickly identify which parts of the lip sync need improvement.

[0038] In view of this, an embodiment of the present application provides a method for evaluating digital human lip synchronization. By calculating the lip synchronization accuracy corresponding to each sub-video contained in a digital human video and aligning them temporally with the sub-texts obtained by segmenting the text corresponding to the audio contained in the digital human video, the lip synchronization accuracy corresponding to each sub-text in the text corresponding to the audio contained in the digital human video can be obtained. This allows users to view the lip synchronization accuracy corresponding to each sub-text in the text corresponding to the audio contained in the digital human video, thereby quickly locating which parts of the digital human video need lip synchronization improvement and optimizing the production process corresponding to the parts that need improvement.

[0039] For example, Figure 1 shows a schematic diagram of the architecture of a digital human lip synchronization evaluation system provided in an embodiment of the present application. As shown in Figure 1 , the digital human lip synchronization evaluation system 100 may include: an audio and video extraction module 110, a frame-level lip synchronization determination module 120, a speech recognition module 130, a text segmentation module 140, and a time alignment module 150.

[0040] The audio and video extraction module 110 is primarily used to extract the video frames and audio contained in the digital human video to obtain the audio and individual video frames contained in the digital human video. Exemplarily, the audio and video extraction module 110 can use specific functions or methods to extract audio and video frames from the digital human video. For example, the audio and video extraction module 110 can use the AudioFileClip class in the moviepy library to load the digital human video to extract the audio; the audio and video extraction module 110 can use the get_frame method of the VideoFileClip class in the moviepy library to extract the video frames from the digital human video.

[0041] The frame-level lip synchronization identification module 120 is primarily used to calculate the degree of match between the digital human's mouth shape within each frame window and the audio corresponding to the corresponding frame window, thereby obtaining the lip synchronization accuracy of the digital human corresponding to each frame window. For example, when the minimum value of the digital human within a frame window completely matches the audio corresponding to the frame window (i.e., the matching degree is 100%), the lip synchronization accuracy of the digital human within the frame window is 100%; when the minimum value of the digital human within a frame window completely mismatches the audio corresponding to the frame window (i.e., the matching degree is 0), the lip synchronization accuracy of the digital human within the frame window is 0; and when the minimum value of the digital human within a frame window completely matches the audio corresponding to the frame window (i.e., the matching degree is 60%), the lip synchronization accuracy of the digital human within the frame window is 60%. For example, a frame window may include at least one video frame. Furthermore, a frame window may also be referred to as a sub-video. For example, the frame-level lip synchronization identification module 120 may be a neural network model, which can be trained using a large number of real-person speaking videos.

[0042] The speech recognition module 130 is mainly used to recognize the audio in the digital human video through an automatic speech recognition (ASR) algorithm to convert the audio into text.

[0043] The text segmentation module 140 is mainly used to segment the text recognized by the speech recognition module 130 to obtain multiple sub-texts. Exemplarily, the text segmentation method performed by the text segmentation module 140 may include: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation. Exemplarily, the text segmentation module 140 may be, but is not limited to, a neural network model. In some embodiments, before the text segmentation module 140 segments the text, the digital human lip synchronization evaluation system 100 may also display a text segmentation method selection interface through a display module thereon (not shown in the figure). Through this text segmentation method selection interface, the user can select a corresponding text segmentation method based on the segmentation granularity of their own interest. After the user selects the text segmentation method, the text segmentation module 140 can obtain the text segmentation method selected by the user and perform text segmentation according to the text segmentation method selected by the user. In addition, when outputting the segmented sub-texts, the text segmentation module 140 may also output the start timestamp and the end timestamp of each sub-text, so as to align them with the sub-videos in time later.

[0044] The time alignment module 150 is primarily used to temporally align each subtext with each subvideo based on the timestamps of each subtext (e.g., the start and end timestamps, etc.) and the timestamps of each subvideo, to obtain the lip synchronization accuracy corresponding to each subtext. Since the lip synchronization accuracy corresponding to each subvideo has been calculated by the frame-level lip synchronization identification module 120, after temporally aligning each subtext with each subvideo, each subtext can be aligned with the calculated lip synchronization accuracy. When a subtext is aligned with multiple lip synchronization accuracy rates, the lip synchronization accuracy corresponding to the subtext can be calculated from these multiple lip synchronization accuracy rates, such as by averaging or weighted averaging. When a subtext is aligned with a lip synchronization accuracy rate, this lip synchronization accuracy rate can be used as the lip synchronization accuracy corresponding to the subtext. For example, the text after segmentation is "today", "weather", and "sunny". There are two sub-videos in the duration corresponding to the text "today", and the lip synchronization accuracy rates of the two sub-videos are both 100%. There are also two sub-videos in the duration corresponding to the text "weather", and the lip synchronization accuracy rates of the two sub-videos are 20% and 70% respectively. There is one sub-video in the duration corresponding to the text "sunny", and the lip synchronization accuracy rates of the sub-videos are both 40%. If the average calculation method is used, the lip synchronization accuracy rate of the text "today" is 100%, the lip synchronization accuracy rate of the text "weather" is 45%, and the lip synchronization accuracy rate of the text "sunny" is 40%. In some embodiments, when the time alignment module 150 performs time alignment, the digital human lip synchronization evaluation system 100 can also display a lip synchronization accuracy calculation method selection interface through the display module thereon (not shown in the figure). Through this lip synchronization accuracy calculation method selection interface, the user can select the corresponding lip synchronization accuracy calculation method based on their own needs, such as average calculation or weighted average calculation. After the user selects the lip synchronization accuracy calculation method, the time alignment module 150 may obtain the lip synchronization accuracy calculation method selected by the user and perform lip synchronization accuracy calculation using the lip synchronization accuracy calculation method selected by the user.

[0045] Furthermore, after calculating the lip synchronization accuracy for each subtext, the time alignment module 150 can output the lip synchronization accuracy for each subtext. In some embodiments, the time alignment module 150 can output a table that records the lip synchronization accuracy for each subtext. In other embodiments, as shown in FIG2 , the time alignment module 150 can output a graph. The vertical axis of the graph represents the lip synchronization accuracy, and the horizontal axis represents time. The data points in the graph represent the lip synchronization accuracy for each subtext, and the subtext corresponding to each data point can be displayed on the graph. This allows digital human video producers to intuitively identify which parts of the lip synchronization need improvement through the graph. In some embodiments, referring again to FIG2 , in addition to outputting the lip synchronization accuracy for each subtext, the time alignment module 150 can also output the audio synchronization accuracy for the entire digital human video, allowing digital human video producers to understand the overall audio synchronization accuracy of the digital human video. The overall audio synchronization accuracy of the digital human video can be obtained by, but is not limited to, averaging the lip synchronization accuracy corresponding to each sub-text.

[0046] In some embodiments, the digital human lip synchronization evaluation system 100 can be, but is not limited to, configured in a cloud computing platform. A cloud computing platform refers to an Internet-based computing model that provides various computing resources and services through the network, including computing power, storage space, databases, network functions, and the like. Cloud computing platforms can aggregate and centrally manage computing resources through virtualization technology, allowing users to access and use these resources on demand through the cloud computing platform without having to own and maintain their own physical equipment and infrastructure. Cloud computing platforms are typically composed of large data centers equipped with high-performance computing equipment (such as servers, etc.), storage devices, and network equipment, as well as reliable power and network connections.

[0047] The above is an introduction to the digital human lip synchronization evaluation system provided by the embodiment of the present application. Next, based on the digital human lip synchronization evaluation system described above, a digital human lip synchronization evaluation method provided by the embodiment of the present application is introduced.

[0048] For example, FIG3 shows a flow chart of a method for evaluating lip synchronization of a digital human provided by an embodiment of the present application. It is understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, it can be executed by a cloud computing platform or a terminal device. For ease of description, the following description uses the cloud computing platform as an example. As shown in FIG3 , the method for evaluating lip synchronization of a digital human may include the following steps:

[0049] S301: Obtain the digital human video input by the user.

[0050] In this embodiment, digital human video producers can purchase the digital human lip sync evaluation service provided by the cloud computing platform to use this service. When using this service, digital human video producers can upload the digital human video to be evaluated to the cloud computing platform through a client (e.g., a desktop application, mobile application, web application, or web-based application). After the digital human video producer completes the upload, the cloud computing platform can obtain the digital human video input by the digital human video producer.

[0051] S302: Convert the audio in the digital human video into a first text, and segment the first text to obtain multiple sub-texts.

[0052] In this embodiment, the cloud computing platform can extract audio from the digital human video and convert the audio into a first text using an ASR algorithm. The cloud computing platform can then segment the first text using a preset text segmentation method to obtain multiple subtexts. For example, if the first text is "It's sunny today," the segmented subtexts may be "today," "weather," and "sunny." In some embodiments, the cloud computing platform can also present a text segmentation method selection interface to the user through the client, allowing the user to select a corresponding text segmentation method. After the user completes the text segmentation method selection through the client, the cloud computing platform can obtain the user-selected text segmentation method and segment the first text using the user-selected text segmentation method. Exemplarily, the text segmentation method used to segment the first text may include any one of: phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation, or paragraph segmentation.

[0053] S303: Determine the degree of matching between the lip shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video.

[0054] In this embodiment, the cloud computing platform can extract audio and video frames from the digital human video, and calculate the degree of match between the digital human's mouth shape in each sub-video contained in the digital human video and the audio in the corresponding sub-video through the aforementioned frame-level lip synchronization identification module 120 to obtain multiple lip synchronization accuracy rates. A sub-video may include at least one video frame. Furthermore, a lip synchronization accuracy rate can be associated with a sub-video.

[0055] S304 , based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, align each sub-text with each lip synchronization accuracy rate in time to obtain the lip synchronization accuracy rate corresponding to each sub-text.

[0056] In this embodiment, after completing the segmentation of the first text and calculating the lip synchronization accuracy corresponding to each sub-video, the cloud computing platform can align each sub-text with each lip synchronization accuracy based on the timestamp of each sub-text and the timestamp of the sub-video corresponding to each lip synchronization accuracy, so as to calculate the lip synchronization accuracy corresponding to each sub-text. In particular, if there are multiple sub-videos between the start and end timestamps of a sub-text, the lip synchronization accuracy corresponding to the sub-text can be determined based on the lip synchronization accuracy corresponding to these multiple sub-videos, for example, by averaging or weighted averaging the lip synchronization accuracy corresponding to these multiple sub-videos. If there is only one sub-video between the start and end timestamps of a sub-text, the lip synchronization accuracy corresponding to this sub-video can be used as the lip synchronization accuracy corresponding to the sub-text. In some embodiments, the cloud computing platform can also present an accuracy calculation method selection interface to the user through the client, so that the user can select the corresponding accuracy calculation method. After the user completes the accuracy calculation method selection through the client, the cloud computing platform can obtain the accuracy calculation method selected by the user (for example, average calculation or weighted average calculation, etc.) and calculate the lip synchronization accuracy corresponding to each sub-text using the accuracy calculation method selected by the user.

[0057] S305: Output the lip synchronization accuracy corresponding to each sub-text.

[0058] In this embodiment, after calculating the lip synchronization accuracy rate for each subtext, the cloud computing platform can output the lip synchronization accuracy rate for each subtext. For example, the cloud computing platform can output a graph, such as shown in FIG2 , to present the lip synchronization accuracy rate for each subtext. The overall accuracy rate for the digital human video shown in FIG2 can be output based on actual needs and is not a mandatory limitation in this embodiment.

[0059] In this way, by calculating the lip synchronization accuracy of each sub-text corresponding to the audio contained in the digital human video, the digital human video producer can quickly locate which parts of the digital human video need to be improved in lip synchronization based on the lip synchronization accuracy corresponding to each sub-text, so that the digital human video producer can optimize the digital human video production process based on the parts that need to be improved to improve the lip synchronization accuracy of the digital human video.

[0060] In some embodiments, when optimizing the digital human video production process, if the lip sync accuracy rate corresponding to a subtext in the digital human video is lower than a preset accuracy threshold, the digital human video producer can manually adjust the digital human's mouth shape at the time corresponding to the subtext, or adjust the digital human's skeleton points and / or expression base at the time corresponding to the subtext. Of course, adjustments can also be made automatically. For example, the digital human video producer or a cloud computing platform can input the lip sync accuracy rates corresponding to each subtext in the digital human video into a device or application (APP) used to produce the digital human video. Afterwards, the device or APP can automatically adjust the digital human's mouth shape at the time corresponding to the subtext with low lip sync accuracy, or adjust the digital human's skeleton points and / or expression base at the time corresponding to the subtext, or replace the driving algorithm at the time corresponding to the subtext. This ensures that the lip sync accuracy rate corresponding to each subtext meets the requirements.

[0061] It should be understood that the digital human lip synchronization evaluation method shown in FIG3 is described with the cloud computing platform as the execution subject. The process of the digital human lip synchronization evaluation method when other devices or clients are the execution subject can refer to the execution process of the cloud computing platform and will not be described in detail here. Furthermore, the digital human lip synchronization evaluation method when other devices or clients are the execution subject remains within the scope of protection of this application.

[0062] It can be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0063] The above is an introduction to the digital human lip synchronization evaluation method provided by the embodiment of the present application. Next, based on the method in the above embodiment, a digital human lip synchronization evaluation device provided by the embodiment of the present application is introduced.

[0064] For example, FIG4 shows a schematic diagram of the structure of a digital human lip synchronization evaluation device provided in an embodiment of the present application. As shown in FIG4 , the digital human lip synchronization evaluation device 400 may include an acquisition module 401 and a processing module 402. Acquisition module 401 is configured to acquire a digital human video input by a user. Processing module 402 is configured to convert the audio in the digital human video into a first text, segment the first text to obtain multiple subtexts, and determine the degree of match between the digital human's lip shape in each sub-video contained in the digital human video and the audio in the corresponding sub-video to obtain multiple lip synchronization accuracy rates, wherein each lip synchronization accuracy rate is associated with each sub-video. Processing module 402 is further configured to temporally align each sub-text with each lip synchronization accuracy rate based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, obtain the lip synchronization accuracy rate corresponding to each sub-text, and output the lip synchronization accuracy rate corresponding to each sub-text.

[0065] In some embodiments, the processing module 402 is further configured to: determine the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each sub-text; and output the lip synchronization accuracy of the digital human video.

[0066] In some embodiments, when outputting the lip synchronization accuracy corresponding to each sub-text, the processing module 402 is specifically used to: output a curve graph, the vertical axis of the curve graph is the lip synchronization accuracy, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy corresponding to each sub-text, and the sub-text corresponding to each data point is displayed on the curve graph.

[0067] In some embodiments, when the processing module 402 segments the first text to obtain multiple sub-texts, it is specifically used to: obtain the text segmentation method input by the user, the text segmentation method including: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; based on the text segmentation method, segment the first text to obtain multiple sub-texts.

[0068] In some embodiments, the multiple subtexts include a first subtext. In this case, when obtaining the lip synchronization accuracy rate corresponding to the first subtext, the processing module 402 is specifically configured to: if multiple sub-videos exist between the start and end timestamps of the first subtext, determine the lip synchronization accuracy rate corresponding to the first subtext based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; and if only one sub-video exists between the start and end timestamps of the first subtext, use the lip synchronization accuracy rate corresponding to the one sub-video as the lip synchronization accuracy rate corresponding to the first subtext.

[0069] In some embodiments, when determining the lip synchronization accuracy corresponding to the first sub-text based on the lip synchronization accuracy corresponding to multiple sub-videos, the processing module 402 is specifically used to: obtain the accuracy calculation method input by the user; based on the accuracy calculation method, process the lip synchronization accuracy corresponding to the multiple sub-videos to obtain the lip synchronization accuracy corresponding to the first sub-text.

[0070] In some embodiments, the multiple subtexts include a second subtext. In this case, after outputting the lip synchronization accuracy rates corresponding to each subtext, the processing module 402 is further configured to: adjust the digital human's mouth shape, bone points, or expression base at the time corresponding to the second subtext, or replace one or more items in the digital human's driving algorithm, wherein the lip synchronization accuracy rate corresponding to the second subtext is lower than a preset accuracy threshold.

[0071] In some embodiments, both acquisition module 401 and processing module 402 shown in FIG4 can be implemented via software or hardware. For example, the implementation of acquisition module 401 will be described below using acquisition module 401 as an example. Similarly, the implementation of processing module 402 can refer to the implementation of acquisition module 401.

[0072] As an example of a software functional unit, the module acquisition module 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the acquisition module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0073] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0074] As an example of a hardware functional unit, acquisition module 401 may include at least one computing device, such as a server. Alternatively, acquisition module 401 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0075] The multiple computing devices included in acquisition module 401 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0076] It should be noted that, in other embodiments, the acquisition module 401 can be used to execute any step in the digital human lip synchronization evaluation method described in the above embodiment, and the processing module 402 can be used to execute any step in the digital human lip synchronization evaluation method described in the above embodiment. The steps that the acquisition module 401 and the processing module 402 are responsible for implementing can be specified as needed. By respectively implementing different steps in the digital human lip synchronization evaluation method described in the above embodiment, the full functions of the digital human lip synchronization evaluation device 400 shown in Figure 4 are realized.

[0077] This application also provides a computing device 500. As shown in Figure 5, computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. Computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 500.

[0078] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG5 shows a single line, but this does not imply a single bus or type of bus. Bus 504 may include a path for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, and communication interface 508).

[0079] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0080] The memory 506 may include volatile memory, such as random access memory (RAM). The processor 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0081] Memory 506 stores executable program code, which processor 504 executes to implement the functions of acquisition module 401 and processing module 402 shown in FIG. 4 , thereby implementing the digital human lip synchronization evaluation method described in the above embodiment. In other words, memory 506 stores instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0082] Alternatively, the memory 506 stores executable code, and the processor 504 executes the executable code to implement the functions of the digital human lip synchronization evaluation device 400 shown in FIG. 4 , thereby implementing the digital human lip synchronization evaluation method described in the above embodiment. That is, the memory 506 stores instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0083] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0084] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0085] As shown in Figure 6, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0086] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the digital human lip synchronization evaluation method described in the above embodiment. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0087] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster can store different instructions, each for executing part of the functions of the digital human lip synchronization evaluation device 400 shown in FIG. 4 . In other words, the instructions stored in the memory 506 in different computing devices 500 can implement the functions of one or more of the acquisition module 401 and the processing module 402.

[0088] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), among others. FIG. 7 illustrates a possible implementation. As shown in FIG. 7 , two computing devices 500A and 500B are connected via a network. Specifically, the connection to the network is achieved via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the acquisition module 501. Simultaneously, the memory 506 in the computing device 500B stores instructions for executing the functions of the processing module 502.

[0089] It should be understood that the functionality of the computing device 500A shown in FIG7 may also be implemented by multiple computing devices 500. Similarly, the functionality of the computing device 500B may also be implemented by multiple computing devices 500.

[0090] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between computing device clusters described in Figures 6 and 7. However, the memory 506 in one or more computing devices 500 in this computing device cluster can store the same instructions for executing the method described in the above embodiment.

[0091] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store some instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for executing the aforementioned data processing method.

[0092] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid-state drive), etc.

[0093] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment, or, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above embodiment.

[0094] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0095] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0096] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0097] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating lip synchronization of a digital human, characterized in that: The method comprises: Get the digital human video input by the user; Converting the audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; Determine the matching degree between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; Based on the timestamp of the subtext and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, aligning each of the subtexts with each of the lip synchronization accuracy rates in time to obtain the lip synchronization accuracy rates corresponding to each of the subtexts; The lip synchronization accuracy rate corresponding to each of the sub-texts is output.

2. The method according to claim 1, characterized in that The method further comprises: Determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each of the sub-texts; The lip synchronization accuracy of the digital human video is output.

3. The method according to claim 1 or 2, characterized in that: Outputting the lip synchronization accuracy corresponding to each of the sub-texts specifically includes: A curve graph is output, wherein the vertical axis of the curve graph is the lip synchronization accuracy rate, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy rates corresponding to the sub-texts, and the sub-texts corresponding to the data points are displayed on the curve graph.

4. The method according to any one of claims 1 to 3, characterized in that: The segmenting of the first text to obtain a plurality of sub-texts specifically includes: Acquire the text segmentation method input by the user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; Based on the text segmentation method, the first text is segmented to obtain the multiple sub-texts.

5. The method according to any one of claims 1 to 4, characterized in that: The plurality of subtexts include a first subtext; Obtaining the lip synchronization accuracy rate corresponding to the first subtext includes: If there are multiple sub-videos between the timestamp at which the first sub-text starts and the timestamp at which the first sub-text ends, determining the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; In the case that there is a sub-video between the start timestamp and the end timestamp of the first sub-text, the lip synchronization accuracy rate corresponding to the sub-video is used as the lip synchronization accuracy rate corresponding to the first sub-text.

6. The method according to claim 5, characterized in that The determining, based on the lip synchronization accuracy rates corresponding to the multiple sub-videos, the lip synchronization accuracy rate corresponding to the first sub-text specifically includes: Obtaining a method for calculating the accuracy of the user input; Based on the accuracy calculation method, the lip synchronization accuracy rates corresponding to the multiple sub-videos are processed to obtain the lip synchronization accuracy rates corresponding to the first sub-text.

7. The method according to any one of claims 1 to 6, characterized in that: The plurality of subtexts include a second subtext; After outputting the lip synchronization accuracy rate corresponding to each of the sub-texts, the method further includes: Adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second sub-text, or replace one or more of the driving algorithms of the digital human, wherein the lip synchronization accuracy corresponding to the second sub-text is lower than a preset accuracy threshold.

8. A digital human lip synchronization evaluation device, characterized in that: include: An acquisition module, used to acquire a digital human video input by a user; A processing module, used for converting the audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; The processing module is further used to determine the matching degree between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; The processing module is further configured to align each of the subtexts with each of the lip synchronization accuracy rates in time based on the timestamp of the subtext and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, so as to obtain the lip synchronization accuracy rate corresponding to each of the subtexts; The processing module is further used to output the lip synchronization accuracy corresponding to each of the sub-texts.

9. The device according to claim 7, characterized in that The processing module is further used for: Determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each of the sub-texts; The lip synchronization accuracy of the digital human video is output.

10. The device according to claim 7 or 8, characterized in that When the processing module outputs the lip synchronization accuracy rate corresponding to each of the sub-texts, it is specifically used to: A curve graph is output, wherein the vertical axis of the curve graph is the lip synchronization accuracy rate, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy rates corresponding to the sub-texts, and the sub-texts corresponding to the data points are displayed on the curve graph.

11. The device according to any one of claims 8 to 10, characterized in that: When the processing module segments the first text to obtain a plurality of sub-texts, the processing module is specifically used to: Acquire the text segmentation method input by the user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; Based on the text segmentation method, the first text is segmented to obtain the multiple sub-texts.

12. The device according to any one of claims 8 to 11, characterized in that: The plurality of subtexts include a first subtext; When obtaining the lip synchronization accuracy rate corresponding to the first subtext, the processing module is specifically used to: If there are multiple sub-videos between the timestamp at which the first sub-text starts and the timestamp at which the first sub-text ends, determining the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; In the case that there is a sub-video between the start timestamp and the end timestamp of the first sub-text, the lip synchronization accuracy rate corresponding to the sub-video is used as the lip synchronization accuracy rate corresponding to the first sub-text.

13. The device according to claim 12, characterized in that When the processing module determines the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos, the processing module is specifically used to: Obtaining a method for calculating the accuracy of the user input; Based on the accuracy calculation method, the lip synchronization accuracy rates corresponding to the multiple sub-videos are processed to obtain the lip synchronization accuracy rates corresponding to the first sub-text.

14. The device according to any one of claims 8 to 13, characterized in that: The plurality of subtexts include a second subtext; After outputting the lip synchronization accuracy rate corresponding to each of the sub-texts, the processing module is further used to: Adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second sub-text, or replace one or more of the driving algorithms of the digital human, wherein the lip synchronization accuracy corresponding to the second sub-text is lower than a preset accuracy threshold.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster comprises at least one computing device.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster includes at least one computing device.

Citation Information

Patent Citations

  • Lip synchronization detection device and method

    CN103313089A

  • Output method and system for virtual human multi-modal interaction

    CN107808191A

  • Video and audio lip synchronization synthesis method and system

    CN110493613A

  • Sound and lip consistency judgment method based on multi-key sound joint score fusion

    CN115861881A

  • Systems and methods for textual content creation from sources of audio that contain speech

    US20160179831A1