Voice and lip synchronous evaluation method and device of digital human and computing equipment cluster

Through automatic speech recognition and text segmentation technology, the accuracy of sound and lip synchronization of digital human videos is calculated and displayed, and the problem of time-consuming and labor-consuming evaluation of the existing technology of mid-sound lip synchronization is solved, and the rapid positioning and optimization of sound and lip synchronization is achieved. Effect.

CN119071468APending Publication Date: 2024-12-03HUAWEI CLOUD COMPUTING TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310840758.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-06-01
Filing Date
2023-07-10
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

In the prior art, the lip synchronous evaluation of digital human videos mainly relies on manual scoring, which is time-consuming and labor-intensive and difficult to give word-level, word-level, and sentence-level accuracy, making it difficult for digital human producers to quickly locate the parts that need to be improved.

Method used

By automatically recognizing the audio in digital human videos, dividing the text, calculating the sound and lips synchronization accuracy of each sub-video, and based on the timestamp alignment, the sound and lips synchronization accuracy of the output sub-text is provided to provide an intuitive display of the curve graph. Users can optimize the production process based on the results.

Benefits of technology

The parts that need to be improved when positioning the sound and lips synchronization are achieved are improved, the accuracy of sound and lips synchronization in digital human videos is improved, and the time and human error of manual evaluation are reduced.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119071468A_ABST
    Figure CN119071468A_ABST
Patent Text Reader

Abstract

The invention discloses a sound and lip synchronization evaluation method for a digital person. The method comprises the following steps: acquiring a digital person video input by a user; converting an audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; determining the matching degree between the mouth shape of the digital human in each sub-video included in the digital human video and the audio in the corresponding sub-video, and obtaining a plurality of audio and lip synchronization accuracy rates, one audio and lip synchronization accuracy rate being associated with one sub-video; on the basis of the timestamps of the sub-texts and the timestamps of the sub-videos corresponding to the audio and lip synchronization accuracy rates, all the sub-texts are aligned with all the audio and lip synchronization accuracy rates in time, and the audio and lip synchronization accuracy rates corresponding to all the sub-texts are obtained; and outputting the sound and lip synchronization accuracy rate corresponding to each sub-text. Therefore, a user can quickly position which parts of the audio lip synchronization in the digital human video need to be improved through the audio lip synchronization accuracy rate corresponding to each sub-text, and the digital human video production process is optimized.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application claims the priority of the Chinese patent application filed with the State Intellectual Property Office of China on June 1, 2023, with application number 202310644516.9 and application name “A Method for Evaluating Digital Human Lip Synchronization”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of information technology (IT) technology, and in particular to a method, device and computing device cluster for evaluating lip synchronization of a digital human. Background Art

[0003] Digital humans are virtual characters or artificial intelligence entities generated through technologies such as artificial intelligence (AI). At present, the main ways to generate digital humans include: voice-driven and visual-driven. Voice-driven is to use voice information to drive a given picture or video to generate a digital human. Video-driven is to use video information to drive a given picture or video to generate a digital human. Regardless of whether it is voice-driven or visual-driven, a core point is to consider lip synchronization, that is, to focus on making the voice information match the digital human's lip shape, so that the digital human can be as close to a real person as possible. Therefore, how to evaluate the lip synchronization of digital humans is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, device, computing device cluster, computer storage medium and computer product for evaluating the lip synchronization of a digital human, which can evaluate the lip synchronization of a digital human.

[0005] In a first aspect, the present application provides a method for evaluating lip synchronization of a digital human, comprising: obtaining a digital human video input by a user; converting the audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; determining the degree of matching between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, to obtain a plurality of lip synchronization accuracy rates, wherein a lip synchronization accuracy rate is associated with a sub-video; based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, aligning each sub-text with each lip synchronization accuracy rate in time to obtain the lip synchronization accuracy rate corresponding to each sub-text; and outputting the lip synchronization accuracy rate corresponding to each sub-text.

[0006] In this way, by calculating the lip synchronization accuracy of each sub-text corresponding to the audio contained in the digital human video, the user can quickly locate which parts of the digital human video need to be improved in lip synchronization through the lip synchronization accuracy corresponding to each sub-text, so that the user can optimize the digital human video production process based on the parts that need to be improved to improve the lip synchronization accuracy of the digital human video.

[0007] In a possible implementation, the method may further include: determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each subtext; and outputting the lip synchronization accuracy of the digital human video. In this way, the user can understand the overall lip synchronization accuracy of the digital human video.

[0008] In a possible implementation, outputting the lip synchronization accuracy rate corresponding to each sub-text specifically includes: outputting a curve graph, the vertical axis of the curve graph is the lip synchronization accuracy rate, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy rates corresponding to each sub-text, and the sub-text corresponding to each data point is displayed on the curve graph. In this way, the user can intuitively understand the lip synchronization accuracy rate corresponding to each sub-text through the curve graph.

[0009] In a possible implementation, segmenting the first text to obtain multiple sub-texts specifically includes: obtaining a text segmentation method input by a user, the text segmentation method including: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; segmenting the first text based on the text segmentation method to obtain multiple sub-texts. In this way, the user can segment the text based on the segmentation granularity required by the user, so that the sub-texts obtained by segmentation can meet the user's needs.

[0010] In a possible implementation, the multiple subtexts include the first subtext. In this case, obtaining the lip synchronization accuracy rate corresponding to the first subtext includes: when there are multiple sub-videos between the timestamp at which the first subtext starts and the timestamp at which the first subtext ends, determining the lip synchronization accuracy rate corresponding to the first subtext based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; and when there is one sub-video between the timestamp at which the first subtext starts and the timestamp at which the first subtext ends, using the lip synchronization accuracy rate corresponding to the one sub-video as the lip synchronization accuracy rate corresponding to the first subtext.

[0011] In a possible implementation, based on the lip synchronization accuracy rates corresponding to the multiple sub-videos, determining the lip synchronization accuracy rate corresponding to the first sub-text specifically includes: obtaining the accuracy calculation method input by the user; based on the accuracy calculation method selected by the user, processing the lip synchronization accuracy rates corresponding to the multiple sub-videos to obtain the lip synchronization accuracy rate corresponding to the first sub-text. In this way, the user can select the corresponding accuracy calculation method based on his own needs, so that the calculation result can meet the user's needs.

[0012] In a possible implementation, the second subtext is included in the plurality of subtexts. At this time, after outputting the lip sync accuracy corresponding to each subtext, the method further includes: adjusting the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second subtext, or replacing one or more of the driving algorithms of the digital human, wherein the lip sync accuracy corresponding to the second subtext is lower than a preset accuracy threshold. In this way, the production process of the digital human video can be optimized, and the lip sync accuracy of the subsequently generated digital human video can be improved.

[0013] In a second aspect, the present application provides a lip-sync evaluation device for a digital human, comprising: an acquisition module and a processing module. The acquisition module is used to acquire a digital human video input by a user. The processing module is used to convert the audio in the digital human video into a first text, and segment the first text to obtain a plurality of sub-texts; and, determine the degree of match between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain a plurality of lip-sync accuracy rates, wherein a lip-sync accuracy rate is associated with a sub-video. The processing module is also used to align each sub-text with each lip-sync accuracy rate in time based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip-sync accuracy rate, obtain the lip-sync accuracy rate corresponding to each sub-text, and output the lip-sync accuracy rate corresponding to each sub-text.

[0014] In a possible implementation, the processing module is further used to: determine the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each sub-text; and output the lip synchronization accuracy of the digital human video.

[0015] In one possible implementation, when the processing module outputs the lip synchronization accuracy corresponding to each sub-text, it is specifically used to: output a curve graph, the vertical axis of the curve graph is the lip synchronization accuracy, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy corresponding to each sub-text, and the sub-text corresponding to each data point is displayed on the curve graph.

[0016] In one possible implementation, when the processing module segments the first text to obtain multiple sub-texts, it is specifically used to: obtain a text segmentation method input by a user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; based on the text segmentation method, segment the first text to obtain multiple sub-texts.

[0017] In a possible implementation, the multiple subtexts include the first subtext. At this time, when the processing module obtains the lip synchronization accuracy rate corresponding to the first subtext, it is specifically used to: when there are multiple sub-videos between the timestamp at the start and the timestamp at the end of the first subtext, determine the lip synchronization accuracy rate corresponding to the first subtext based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; when there is one sub-video between the timestamp at the start and the timestamp at the end of the first subtext, use the lip synchronization accuracy rate corresponding to the one sub-video as the lip synchronization accuracy rate corresponding to the first subtext.

[0018] In a possible implementation, when the processing module determines the lip synchronization accuracy corresponding to the first sub-text based on the lip synchronization accuracy corresponding to multiple sub-videos, the processing module is specifically used to: obtain the accuracy calculation method input by the user; based on the accuracy calculation method, process the lip synchronization accuracy corresponding to the multiple sub-videos to obtain the lip synchronization accuracy corresponding to the first sub-text.

[0019] In a possible implementation, the plurality of subtexts include a second subtext. At this time, after outputting the lip synchronization accuracy corresponding to each subtext, the processing module is further used to: adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second subtext, or replace one or more of the driving algorithms of the digital human, wherein the lip synchronization accuracy corresponding to the second subtext is lower than a preset accuracy threshold.

[0020] In a third aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster performs the method described in the first aspect or any possible implementation of the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when the computer program instructions are executed by a computing device, the computing device executes the method described in the first aspect or any possible implementation of the first aspect; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0022] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device, causes the computing device to perform the method described in the first aspect or any possible implementation of the first aspect, or, when executed by a computing device cluster, causes the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect. Exemplarily, the computing device cluster may include one or more computing devices.

[0023] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a schematic diagram of the architecture of a digital human lip synchronization evaluation system provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of a lip synchronization evaluation result of a digital human provided in an embodiment of the present application;

[0026] Figure 3 It is a flowchart of a method for evaluating lip synchronization of a digital human provided in an embodiment of the present application;

[0027] Figure 4 It is a structural schematic diagram of a digital human lip synchronization evaluation device provided in an embodiment of the present application;

[0028] Figure 5 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0029] Figure 6 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0030] Figure 7 It is a structural diagram of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0032] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0033] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0034] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0035] Generally speaking, in a digital human video, only when the digital human's lip shape matches the audio information in the video can the digital human be as close to a real person as possible, thereby improving the use effect of the digital human. However, in related technologies, lip synchronization evaluation of digital human videos is often performed through manual scoring. This method is time-consuming and labor-intensive, and lacks objective basis. In addition, this manual scoring method generally gives the lip synchronization accuracy of the entire digital human video, but it is difficult to give the lip synchronization accuracy at the word level, word level, sentence level or paragraph level, making it difficult for digital human producers to quickly locate which parts of the lip synchronization need to be improved.

[0036] In view of this, the embodiment of the present application provides a method for evaluating the lip synchronization of a digital human, by calculating the lip synchronization accuracy corresponding to each sub-video contained in the digital human video, and aligning the lip synchronization accuracy corresponding to each sub-text in the text corresponding to the audio contained in the digital human video in time. Thus, the user can view the lip synchronization accuracy corresponding to each sub-text in the text corresponding to the audio contained in the digital human video, so as to quickly locate which parts of the lip synchronization in the digital human video need to be improved, and optimize the production process corresponding to the parts that need to be improved.

[0037] For example, Figure 1 FIG. 1 is a schematic diagram showing the architecture of a digital human lip synchronization evaluation system provided by an embodiment of the present application. Figure 1As shown, the digital human lip synchronization evaluation system 100 may include: an audio and video extraction module 110 , a frame-level lip synchronization identification module 120 , a speech recognition module 130 , a text segmentation module 140 and a time alignment module 150 .

[0038] The audio and video extraction module 110 is mainly used to extract the video frames and audio contained in the digital human video to obtain the audio and each video frame contained in the digital human video. Exemplarily, the audio and video extraction module 110 can use a specific function or method to extract audio and video frames from the digital human video. For example, the audio and video extraction module 110 can use the AudioFileClip class in the moviepy library to load the digital human video to extract audio from it; the audio and video extraction module 110 can use the get_frame method of the VideoFileClip class in the moviepy library to extract video frames from the digital human video.

[0039] The frame-level lip synchronization discrimination module 120 is mainly used to calculate the matching degree between the mouth shape of the digital person in each frame window and the audio corresponding to the corresponding frame window, so as to obtain the lip synchronization accuracy of the digital person corresponding to each frame window. Exemplarily, when the minimum of the digital person in a frame window completely matches the audio corresponding to the frame window (that is, the matching degree is 100%), the lip synchronization accuracy of the digital person in the frame window is 100%; when the minimum of the digital person in a frame window does not completely match the audio corresponding to the frame window (that is, the matching degree is 0), the lip synchronization accuracy of the digital person in the frame window is 0; when the minimum of the digital person in a frame window and the audio corresponding to the frame window match 60%, the lip synchronization accuracy of the digital person in the frame window is 60%. Exemplarily, a frame window may include at least one video frame, and a frame window may also be referred to as a sub-video. Exemplarily, the frame-level lip synchronization discrimination module 120 may be a neural network model, which can be obtained by training a large number of real-person speaking videos.

[0040] The speech recognition module 130 is mainly used to recognize the audio in the digital human video through an automatic speech recognition (ASR) algorithm to convert the audio into text.

[0041] The text segmentation module 140 is mainly used to segment the text recognized by the speech recognition module 130 to obtain multiple sub-texts. Exemplarily, the text segmentation module 140 may include any one of: phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation. Exemplarily, the text segmentation module 140 may be, but is not limited to, a neural network model. In some embodiments, before the text segmentation module 140 segments the text, the digital human lip synchronization evaluation system 100 may also display a text segmentation method selection interface through a display module (not shown in the figure) thereon. Through the text segmentation method selection interface, the user can select the corresponding text segmentation method based on the segmentation granularity of his own concern. After the user selects the text segmentation method, the text segmentation module 140 can obtain the text segmentation method selected by the user and perform text segmentation according to the text segmentation method selected by the user. In addition, when outputting each sub-text obtained by segmentation, the text segmentation module 140 may also output the start timestamp and the end timestamp of each sub-text, so as to align them with each sub-video in time later.

[0042] The time alignment module 150 is mainly used to align each sub-text with each sub-video in time based on the timestamp of each sub-text (such as the start timestamp and the end timestamp, etc.) and the timestamp of each sub-video, so as to obtain the lip synchronization accuracy corresponding to each sub-text. Since the lip synchronization accuracy corresponding to each sub-video has been calculated by the frame-level lip synchronization discrimination module 120, after aligning each sub-text with each sub-video in time, each sub-text can be aligned with each calculated lip synchronization accuracy. Among them, when a sub-text is aligned with multiple lip synchronization accuracy rates, the lip synchronization accuracy corresponding to the sub-text can be calculated by the multiple lip synchronization accuracy rates, such as average calculation, weighted average, etc. When a sub-text is aligned with a lip synchronization accuracy rate, this lip synchronization accuracy rate can be used as the lip synchronization accuracy rate corresponding to the sub-text. For example, the text after segmentation is "today", "weather", and "sunny". There are two sub-videos in the duration corresponding to the text "today", and the lip sync accuracy rates of the two sub-videos are both 100%. There are also two sub-videos in the duration corresponding to the text "weather", and the lip sync accuracy rates of the two sub-videos are 20% and 70% respectively. There is one sub-video in the duration corresponding to the text "sunny", and the lip sync accuracy rates of the sub-videos are both 40%. If the average calculation method is used, the lip sync accuracy rates of the text "today" are 100%, the lip sync accuracy rates of the text "weather" are 45%, and the lip sync accuracy rates of the text "sunny" are 40%. In some embodiments, when the time alignment module 150 performs time alignment, the digital human lip sync evaluation system 100 can also display a lip sync accuracy calculation method selection interface through the display module thereon (not shown in the figure). Through the lip sync accuracy calculation method selection interface, the user can select the corresponding lip sync accuracy calculation method based on his own needs, such as average calculation or weighted average calculation. After the user selects the lip synchronization accuracy calculation method, the time alignment module 150 may obtain the lip synchronization accuracy calculation method selected by the user, and perform lip synchronization accuracy calculation using the lip synchronization accuracy calculation method selected by the user.

[0043] Furthermore, after calculating the lip synchronization accuracy rate corresponding to each sub-text, the time alignment module 150 may output the lip synchronization accuracy rate corresponding to each sub-text. In some embodiments, the time alignment module 150 may output a table, which may record the lip synchronization accuracy rate corresponding to each sub-text. In other embodiments, such as Figure 2As shown, the output of the time alignment module 150 may be a curve graph. The vertical axis of the curve graph is the lip synchronization accuracy, and the horizontal axis is time; the data points in the curve graph are the lip synchronization accuracy corresponding to each sub-text, and the sub-text corresponding to each data point may be displayed on the curve graph. In this way, the digital human video production staff can intuitively know which parts of the lip synchronization need to be improved through the curve graph. In some embodiments, continue to refer to Figure 2 In addition to outputting the lip sync accuracy rate corresponding to each subtext, the time alignment module 150 can also output the audio sync accuracy rate of the entire digital human video, so that the producer of the digital human video can know the overall audio sync accuracy rate of the digital human video. The overall audio sync accuracy rate of the digital human video can be obtained by averaging the lip sync accuracy rates corresponding to each subtext, but is not limited to.

[0044] In some embodiments, the above-mentioned digital human lip synchronization evaluation system 100 can be, but is not limited to, configured in a cloud computing platform. The cloud computing platform refers to an Internet-based computing model that provides various computing resources and services through the network, including computing power, storage space, databases, network functions, etc. The cloud computing platform can aggregate and centrally manage computing resources through virtualization technology, and users can obtain and use these resources on demand through the cloud computing platform without owning and maintaining their own physical equipment and infrastructure. Cloud computing platforms are usually composed of large data centers, which have high-performance computing devices (such as servers, etc.), storage devices and network devices, as well as reliable power and network connections.

[0045] The above is an introduction to the digital human lip synchronization evaluation system provided by the embodiment of the present application. Next, based on the digital human lip synchronization evaluation system described above, a digital human lip synchronization evaluation method provided by the embodiment of the present application is introduced.

[0046] For example, Figure 3 The flowchart of a method for evaluating the lip sync of a digital human provided by an embodiment of the present application is shown. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, it can be executed by a cloud computing platform or a terminal device. For ease of description, the following is an introduction using the cloud computing platform as an example. Figure 3 As shown, the digital human lip synchronization evaluation method may include the following steps:

[0047] S301: Obtain a digital human video input by a user.

[0048] In this embodiment, the producer of the digital human video can purchase the digital human lip synchronization evaluation service provided by the cloud computing platform to use the service. When the producer of the digital human video uses the digital human lip synchronization evaluation service, the producer of the digital human video can transmit the digital human video to be evaluated to the cloud computing platform through a client (such as a desktop application, a mobile application, a Web application, or a Web-based application, etc.). After the producer of the digital human video completes uploading the digital human video, the cloud computing platform can obtain the digital human video input by the producer of the digital human video.

[0049] S302: Convert the audio in the digital human video into a first text, and segment the first text to obtain a plurality of sub-texts.

[0050] In this embodiment, the cloud computing platform can extract audio from the digital human video and convert the audio into a first text through an ASR algorithm. Then, the cloud computing platform can segment the first text through a preset text segmentation method to obtain multiple sub-texts. For example, when the first text is "It's sunny today", the segmented sub-texts can be "today", "weather", and "sunny". In some embodiments, the cloud computing platform can also present a text segmentation method selection interface to the user through the client so that the user can select the corresponding text segmentation method. After the user completes the text segmentation method selection through the client, the cloud computing platform can obtain the text segmentation method selected by the user, and segment the first text through the text segmentation method selected by the user. Exemplarily, the text segmentation method for segmenting the first text may include: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation, or paragraph segmentation.

[0051] S303: Determine the matching degree between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video.

[0052] In this embodiment, the cloud computing platform can extract audio and video frames from the digital human video, and calculate the matching degree between the digital human's mouth shape in each sub-video contained in the digital human video and the audio in the corresponding sub-video through the aforementioned frame-level lip sync discrimination module 120 to obtain multiple lip sync accuracy rates. Among them, a sub-video can include at least one video frame. In addition, a lip sync accuracy rate can be associated with a sub-video.

[0053] S304: Based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, align each sub-text with each lip synchronization accuracy rate in time to obtain the lip synchronization accuracy rate corresponding to each sub-text.

[0054] In this embodiment, after completing the segmentation of the first text and the calculation of the lip synchronization accuracy corresponding to each sub-video, the cloud computing platform can align each sub-text with each lip synchronization accuracy in time based on the timestamps of each sub-text and the timestamps of the sub-videos corresponding to each lip synchronization accuracy, so as to calculate the lip synchronization accuracy corresponding to each sub-text. Wherein, in the case where there are multiple sub-videos between the timestamp at the beginning and the timestamp at the end of a sub-text, the lip synchronization accuracy corresponding to the sub-text can be determined based on the lip synchronization accuracy corresponding to the multiple sub-videos, for example, the lip synchronization accuracy corresponding to the multiple sub-videos can be averaged or weighted averaged. In the case where there is a sub-video between the timestamp at the beginning and the timestamp at the end of a sub-text, the lip synchronization accuracy corresponding to the sub-video can be used as the lip synchronization accuracy corresponding to the sub-text. In some embodiments, the cloud computing platform can also present an accuracy calculation method selection interface to the user through the client, so that the user selects the corresponding accuracy calculation method. After the user completes the accuracy calculation method selection through the client, the cloud computing platform can obtain the accuracy calculation method selected by the user (such as average calculation or weighted average calculation, etc.), and calculate the lip synchronization accuracy corresponding to each sub-text according to the accuracy calculation method selected by the user.

[0055] S305: Output the lip synchronization accuracy corresponding to each sub-text.

[0056] In this embodiment, after calculating the lip synchronization accuracy rate corresponding to each sub-text, the cloud computing platform can output the lip synchronization accuracy rate corresponding to each sub-text. Figure 2 The curve graph shown in FIG. 1 shows the lip synchronization accuracy of each sub-text. Figure 2 The overall accuracy of the digital human video shown in can be output based on actual needs, and this embodiment does not impose any mandatory restrictions.

[0057] In this way, by calculating the lip synchronization accuracy of each sub-text corresponding to the audio contained in the digital human video, the digital human video producer can quickly locate which parts of the digital human video need to be improved in lip synchronization through the lip synchronization accuracy corresponding to each sub-text, so that the digital human video producer can optimize the digital human video production process based on the parts that need to be improved to improve the lip synchronization accuracy of the digital human video.

[0058] In some embodiments, when the digital human video production staff optimizes the digital human video production process, when the lip sync accuracy rate corresponding to a certain subtext in the digital human video is lower than the preset accuracy threshold, the digital human video production staff can manually adjust the digital human's mouth shape at the time corresponding to the subtext, or adjust the digital human's skeleton points and / or expression base at the time corresponding to the subtext. Of course, it can also be adjusted in an automatic way. For example, the digital human video production staff or cloud computing platform can input the lip sync accuracy rate corresponding to each subtext in the digital human video into the device or application (application, APP) used to produce the digital human video. After that, the device or APP can automatically adjust the digital human's mouth shape at the time corresponding to the subtext with low lip sync accuracy in the digital human video, or adjust the digital human's skeleton points and / or expression base at the time corresponding to the subtext, or replace the driving algorithm at the time corresponding to the subtext. In this way, the lip sync accuracy rate corresponding to each subtext meets the requirements.

[0059] Understandably, Figure 3 The lip sync evaluation method for a digital human shown is introduced with the cloud computing platform as the execution subject. The process of the lip sync evaluation method for a digital human when other devices or clients are the execution subjects can refer to the execution process of the cloud computing platform, which will not be described here. In addition, the lip sync evaluation method for a digital human when other devices or clients are the execution subjects is still within the protection scope of this application.

[0060] It is understandable that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0061] The above is an introduction to the lip sync evaluation method for a digital human provided in an embodiment of the present application. Next, based on the method in the above embodiment, a lip sync evaluation device for a digital human provided in an embodiment of the present application is introduced.

[0062] For example, Figure 4 FIG. 1 is a schematic diagram showing the structure of a digital human lip synchronization evaluation device provided in an embodiment of the present application. Figure 4As shown, the lip sync evaluation device 400 for a digital human may include: an acquisition module 401 and a processing module 402. The acquisition module 401 is used to acquire a digital human video input by a user. The processing module 402 is used to convert the audio in the digital human video into a first text, and segment the first text to obtain a plurality of sub-texts, and determine the degree of matching between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video to obtain a plurality of lip sync accuracy rates, wherein a lip sync accuracy rate is associated with a sub-video. The processing module 402 is also used to align each sub-text with each lip sync accuracy rate in time based on the timestamp of the sub-text and the timestamp of the sub-video corresponding to the lip sync accuracy rate, obtain the lip sync accuracy rate corresponding to each sub-text, and output the lip sync accuracy rate corresponding to each sub-text.

[0063] In some embodiments, the processing module 402 is further used to: determine the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each sub-text; and output the lip synchronization accuracy of the digital human video.

[0064] In some embodiments, when the processing module 402 outputs the lip synchronization accuracy corresponding to each sub-text, it is specifically used to: output a curve graph, the vertical axis of the curve graph is the lip synchronization accuracy, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy corresponding to each sub-text, and the sub-text corresponding to each data point is displayed on the curve graph.

[0065] In some embodiments, when the processing module 402 segments the first text to obtain multiple sub-texts, it is specifically used to: obtain a text segmentation method input by the user, the text segmentation method including: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; based on the text segmentation method, segment the first text to obtain multiple sub-texts.

[0066] In some embodiments, the first subtext is included in the plurality of subtexts. At this time, when the processing module 402 obtains the lip synchronization accuracy rate corresponding to the first subtext, it is specifically used to: determine the lip synchronization accuracy rate corresponding to the first subtext based on the lip synchronization accuracy rates corresponding to the plurality of subvideos when there are a plurality of subvideos between the timestamp at the start and the timestamp at the end of the first subtext; and use the lip synchronization accuracy rate corresponding to the one subvideo as the lip synchronization accuracy rate corresponding to the first subtext when there is a subvideo between the timestamp at the start and the timestamp at the end of the first subtext.

[0067] In some embodiments, when the processing module 402 determines the lip synchronization accuracy corresponding to the first sub-text based on the lip synchronization accuracy corresponding to multiple sub-videos, it is specifically used to: obtain the accuracy calculation method input by the user; based on the accuracy calculation method, process the lip synchronization accuracy corresponding to the multiple sub-videos to obtain the lip synchronization accuracy corresponding to the first sub-text.

[0068] In some embodiments, the plurality of subtexts include a second subtext. At this time, after outputting the lip sync accuracy rates corresponding to each subtext, the processing module 402 is further used to: adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second subtext, or replace one or more of the driving algorithms of the digital human, wherein the lip sync accuracy rate corresponding to the second subtext is lower than a preset accuracy threshold.

[0069] In some embodiments, Figure 4 The acquisition module 401 and the processing module 402 shown in the figure can be implemented by software or by hardware. For example, the implementation of the acquisition module 401 is described below by taking the acquisition module 401 as an example. Similarly, the implementation of the processing module 402 can refer to the implementation of the acquisition module 401.

[0070] As an example of a software functional unit, the acquisition module 401 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with close geographical locations. Among them, usually a region may include multiple AZs.

[0071] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.

[0072] As an example of a hardware functional unit, the acquisition module 401 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0073] The multiple computing devices included in the acquisition module 401 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 401 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 401 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0074] It should be noted that, in other embodiments, the acquisition module 401 can be used to execute any step in the lip synchronization evaluation method for a digital human described in the above embodiment, and the processing module 402 can be used to execute any step in the lip synchronization evaluation method for a digital human described in the above embodiment. The steps that the acquisition module 401 and the processing module 402 are responsible for implementing can be specified as needed, and the acquisition module 401 and the processing module 402 respectively implement different steps in the lip synchronization evaluation method for a digital human described in the above embodiment to achieve Figure 4 The digital human lip synchronization evaluation device 400 shown has all the functions.

[0075] The present application also provides a computing device 500. Figure 5 As shown, computing device 500 includes: bus 502, processor 504, memory 506 and communication interface 508. Processor 504, memory 506 and communication interface 508 communicate through bus 502. Computing device 500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 500.

[0076] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus 504 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 504 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).

[0077] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0078] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0079] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the aforementioned Figure 4 The functions of the acquisition module 401 and the processing module 402 shown in the above embodiment are implemented to realize the lip synchronization evaluation method for digital human described in the above embodiment. That is, the memory 506 stores instructions for executing the lip synchronization evaluation method for digital human described in the above embodiment.

[0080] Alternatively, the memory 506 stores executable codes, and the processor 504 executes the executable codes to respectively implement the aforementioned Figure 4 The functions of the digital human lip synchronization evaluation device 400 shown in the figure are implemented to realize the digital human lip synchronization evaluation method described in the above embodiment. That is, the memory 506 stores instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0081] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.

[0082] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0083] like Figure 6 As shown, the computing device cluster includes at least one computing device 500. The memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0084] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also respectively store some instructions for executing the digital human lip synchronization evaluation method described in the above embodiment. In other words, the combination of one or more computing devices 500 can jointly execute the instructions for executing the digital human lip synchronization evaluation method described in the above embodiment.

[0085] It should be noted that the memory 506 in different computing devices 500 in the computing device cluster may store different instructions, which are respectively used to execute the aforementioned Figure 4 The illustrated part of the functions of the digital human lip synchronization evaluation device 400. That is, the instructions stored in the memory 506 in different computing devices 500 can implement the functions of one or more modules in the acquisition module 401 and the processing module 402.

[0086] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Figure 7 A possible implementation is shown. Figure 7 As shown, two computing devices 500A and 500B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 506 in the computing device 500A stores instructions for executing the functions of the acquisition module 501. At the same time, the memory 506 in the computing device 500B stores instructions for executing the functions of the processing module 502.

[0087] It should be understood that Figure 7The functions of the computing device 500A shown in FIG. 5A may also be completed by multiple computing devices 500. Similarly, the functions of the computing device 500B may also be completed by multiple computing devices 500.

[0088] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Figure 6 and Figure 7 The connection mode of the computing device cluster is different in that the memory 506 in one or more computing devices 500 in the computing device cluster may store the same instructions for executing the method in the above embodiment.

[0089] In some possible implementations, the memory 506 of one or more computing devices 500 in the computing device cluster may also store partial instructions for executing the aforementioned data processing method. In other words, the combination of one or more computing devices 500 may jointly execute instructions for executing the aforementioned data processing method.

[0090] Based on the method in the above embodiment, the embodiment of the present application provides a computer-readable storage medium, including computer program instructions. When the computer program instructions are executed by a computing device, the computing device executes the method in the above embodiment; or, when the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above embodiment. Exemplarily, the computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0091] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product containing instructions, which, when executed by a computing device, enables the computing device to execute the method in the above embodiment, or, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above embodiment.

[0092] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0093] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0094] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0095] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for evaluating lip synchronization of a digital human, characterized in that: The method comprises: Get the digital human video input by the user; Converting the audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; Determine the matching degree between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; Based on the timestamp of the subtext and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, aligning each of the subtexts with each of the lip synchronization accuracy rates in time to obtain the lip synchronization accuracy rates corresponding to each of the subtexts; The lip synchronization accuracy rate corresponding to each of the sub-texts is output.

2. The method according to claim 1, characterized in that The method further comprises: Determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each of the sub-texts; The lip synchronization accuracy of the digital human video is output.

3. The method according to claim 1 or 2, characterized in that: Outputting the lip synchronization accuracy corresponding to each of the sub-texts specifically includes: A curve graph is output, wherein the vertical axis of the curve graph is the lip synchronization accuracy rate, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy rates corresponding to the sub-texts, and the sub-texts corresponding to the data points are displayed on the curve graph.

4. The method according to any one of claims 1 to 3, characterized in that: The segmenting of the first text to obtain a plurality of sub-texts specifically includes: Acquire the text segmentation method input by the user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; Based on the text segmentation method, the first text is segmented to obtain the multiple sub-texts.

5. The method according to any one of claims 1 to 4, characterized in that: The plurality of subtexts include a first subtext; Obtaining the lip synchronization accuracy rate corresponding to the first subtext includes: If there are multiple sub-videos between the timestamp at which the first sub-text starts and the timestamp at which the first sub-text ends, determining the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; In the case that there is a sub-video between the start timestamp and the end timestamp of the first sub-text, the lip synchronization accuracy rate corresponding to the sub-video is used as the lip synchronization accuracy rate corresponding to the first sub-text.

6. The method according to claim 5, characterized in that The determining, based on the lip synchronization accuracy rates corresponding to the multiple sub-videos, the lip synchronization accuracy rate corresponding to the first sub-text specifically includes: Obtaining a method for calculating the accuracy of the user input; Based on the accuracy calculation method, the lip synchronization accuracy rates corresponding to the multiple sub-videos are processed to obtain the lip synchronization accuracy rates corresponding to the first sub-text.

7. The method according to any one of claims 1 to 6, characterized in that: The plurality of subtexts include a second subtext; After outputting the lip synchronization accuracy rate corresponding to each of the sub-texts, the method further includes: Adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second sub-text, or replace one or more of the driving algorithms of the digital human, wherein the lip synchronization accuracy corresponding to the second sub-text is lower than a preset accuracy threshold.

8. A digital human lip synchronization evaluation device, characterized in that: include: An acquisition module, used to acquire a digital human video input by a user; A processing module, used for converting the audio in the digital human video into a first text, and segmenting the first text to obtain a plurality of sub-texts; The processing module is further used to determine the matching degree between the mouth shape of the digital human in each sub-video contained in the digital human video and the audio in the corresponding sub-video, and obtain multiple lip synchronization accuracy rates, wherein one lip synchronization accuracy rate is associated with one sub-video; The processing module is further configured to align each of the subtexts with each of the lip synchronization accuracy rates in time based on the timestamp of the subtext and the timestamp of the sub-video corresponding to the lip synchronization accuracy rate, so as to obtain the lip synchronization accuracy rate corresponding to each of the subtexts; The processing module is further used to output the lip synchronization accuracy corresponding to each of the sub-texts.

9. The device according to claim 7, characterized in that The processing module is further used for: Determining the lip synchronization accuracy of the digital human video based on the lip synchronization accuracy corresponding to each of the sub-texts; The lip synchronization accuracy of the digital human video is output.

10. The device according to claim 7 or 8, characterized in that When the processing module outputs the lip synchronization accuracy rate corresponding to each of the sub-texts, it is specifically used to: A curve graph is output, wherein the vertical axis of the curve graph is the lip synchronization accuracy rate, the horizontal axis of the curve graph is time, the data points in the curve graph are the lip synchronization accuracy rates corresponding to the sub-texts, and the sub-texts corresponding to the data points are displayed on the curve graph.

11. The device according to any one of claims 8 to 10, characterized in that: When the processing module segments the first text to obtain a plurality of sub-texts, the processing module is specifically used to: Acquire the text segmentation method input by the user, where the text segmentation method includes: any one of phoneme segmentation, single character segmentation, single word segmentation, multiple word segmentation, sentence segmentation or paragraph segmentation; Based on the text segmentation method, the first text is segmented to obtain the multiple sub-texts.

12. The device according to any one of claims 8 to 11, characterized in that: The plurality of subtexts include a first subtext; When obtaining the lip synchronization accuracy rate corresponding to the first subtext, the processing module is specifically used to: If there are multiple sub-videos between the timestamp at which the first sub-text starts and the timestamp at which the first sub-text ends, determining the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos; In the case that there is a sub-video between the start timestamp and the end timestamp of the first sub-text, the lip synchronization accuracy rate corresponding to the sub-video is used as the lip synchronization accuracy rate corresponding to the first sub-text.

13. The device according to claim 12, characterized in that When the processing module determines the lip synchronization accuracy rate corresponding to the first sub-text based on the lip synchronization accuracy rates corresponding to the multiple sub-videos, the processing module is specifically configured to: Obtaining a method for calculating the accuracy of the user input; Based on the accuracy calculation method, the lip synchronization accuracy rates corresponding to the multiple sub-videos are processed to obtain the lip synchronization accuracy rates corresponding to the first sub-text.

14. The device according to any one of claims 8 to 13, characterized in that: The plurality of subtexts include a second subtext; After outputting the lip synchronization accuracy rate corresponding to each of the sub-texts, the processing module is further used to: Adjust the mouth shape, bone points, and expression base of the digital human at the time corresponding to the second sub-text, or replace one or more of the driving algorithms of the digital human, wherein the lip synchronization accuracy corresponding to the second sub-text is lower than a preset accuracy threshold.

15. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster comprises at least one computing device.

17. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7, wherein the computing device cluster includes at least one computing device.