Digital character audio and video data processing system and method and live broadcast system

Abstract

The invention discloses a digital role audio and video data processing system and method and a live broadcast system, and the system comprises a scheduling server which is used for receiving the to-be-explained content of an anchor end, and selecting a specified resource machine in an audio generation server cluster for processing, the audio data generated by the specified resource machine is pulled to the lip language driving and distribution service cluster; a resource machine included in the audio generation service cluster is used for generating audio data according to the received content to be explained; a resource machine in the lip language driving and distribution service cluster is used for generating a digital role driving video at regular time or in real time according to the audio data and a selected digital role in a stream pushing process, and pushing the digital role driving video to a distribution network; and the bottleneck problems that resources are preempted due to single service equipment and the computing power cannot meet the processing requirements are effectively avoided.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more particularly to a digital character audio and video data processing system, and a digital character live broadcast system. The present application also relates to a digital character audio and video data processing method and device, a digital character live broadcast interaction method and device, and a computer storage medium and electronic device.

[0002] The rapid development and popularization of Internet technology has provided the necessary technical foundation for live video streaming technology.

[0003] Live video streaming is a streaming technology, which is a continuous data stream technology that transmits audio and video data from the source media to the client in real time. Early live video services were usually implemented through web plug-ins or third-party software, and the content was relatively simple, such as: games, music, etc. With the continuous advancement of technology and market demand, live video streaming can be implemented not only through third-party application software but also through live streaming exclusive application platforms, and the content is richer. At present, live video streaming has become an important way of information dissemination and entertainment, and has also become a part of daily life.

[0004] Currently, there are two types of live video broadcasting: one is live broadcasting with real people, and the other is live broadcasting with digital characters. In live broadcasting with real people, the host is a real person, who transmits relevant information of the live broadcast in real time through the camera to interact with the user on the other end of the camera. In live broadcasting with digital characters, the host is a virtual digital character, which is generated by computer graphics, artificial intelligence and other technologies, and broadcasts automatically or semi-automatically according to preset programs or algorithms. Live broadcasting with digital characters is not restricted by time and place, and has higher flexibility and scalability. Summary of the invention

[0005] The present application provides a system for processing digital character audio and video data to solve the problems existing in the prior art, such as computing power bottleneck and high cost.

[0006] The present application provides a digital character audio and video data processing system, including: an audio generation service cluster, a lip reading drive and distribution service cluster and a scheduling server; The scheduling server is used to receive the content to be explained by the anchor, select a designated resource machine in the audio generation server cluster for processing, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster; The resource machine included in the audio generation service cluster is used to generate audio data according to the received content to be explained; The resource machine in the lip reading driving and distribution service cluster is used to generate a digital character driving video according to the audio data and the selected digital character in real time or on a scheduled basis during the streaming process, and stream the digital character driving video to the distribution network.

[0007] In some embodiments, the scheduling server is used to select a designated resource machine in the audio generation server cluster for processing according to the content to be explained received from the anchor terminal, including: The scheduling server obtains task statistics data or work status data of the resource machine from the audio generation service cluster according to the obtained content to be explained; According to the task statistical data or work status data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

[0008] In some embodiments, the resource machine is used to send the generated audio data to the scheduling server, and the scheduling server stores the audio data and sends the audio data to the lip reading drive and distribution service cluster.

[0009] In some embodiments, the lip reading drive and distribution service cluster includes a resource machine for obtaining the image data of the explainer as the selected digital character, and matching the extracted lip features in the explainer image data with the audio features in the audio data, driving the lip features to move according to the audio features, and generating the drive video.

[0010] In some embodiments, the resource machine in the audio generation service cluster is used to receive real-time interaction data and generate real-time interaction audio data corresponding to the real-time interaction data.

[0011] In some embodiments, the resource machine in the lip reading drive and distribution service cluster is used to generate real-time interactive drive video of the digital character according to the real-time matching relationship between the interactive audio features of the real-time interactive audio data and the lip features during the timing or streaming process, and stream it to the distribution network.

[0012] The present application also provides a digital character live broadcast system, including: a live broadcast terminal, a client, a scheduling server, an audio generation service cluster, a lip reading driver and a distribution service cluster; The scheduling server is used to select a designated resource machine in the audio generation server cluster for data conversion processing according to the content to be explained of the live broadcast terminal, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster; The audio generation service cluster includes a resource machine, which is used to generate audio data according to the received content to be explained; The resource machine included in the lip reading driving and distribution service cluster is used to generate a driving video according to the audio data and the selected digital character in real time or on a scheduled basis during the streaming process, and to stream the driving video to the distribution network; The live broadcast terminal is used to send the content to be explained to the scheduling server and execute the live broadcast task; The client is used to receive the live video stream of the driving video.

[0013] The present application also provides a method for processing digital character audio and video data, including: Sending the received content to be explained to a designated resource machine selected from the audio generation server cluster; The designated resource machine converts the content to be explained to generate audio data; The resource machine in the lip reading driving and distribution service cluster generates a digital character driving video in real time according to the received audio data and the selected digital character during timing or in the streaming process, and streams the digital character driving video to the distribution network.

[0014] In some embodiments, the step of sending the received content to be explained to a designated resource machine selected from the audio generation server cluster includes: Obtaining task statistics data or work status data of resource machines in the audio generation service cluster; According to the task statistical data or work status data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

[0015] In some embodiments, the designated resource machine converts the content to be explained to generate audio data, including: Converting the text data in the content to be explained into the audio data; The audio data is stored.

[0016] In some embodiments, the resource machine in the audio data to lip reading driving and distribution service cluster generates a driving video in real time according to the received audio data and the selected digital character during a timing or streaming process, and streams the driving video to a distribution network, including: Using the acquired image data of the narrator as the selected digital character; Matching the extracted lip features in the narrator image data with the audio features in the audio data, driving the lip features to move and change according to the audio features, and generating the driving video; The driving video is pushed to a distribution network.

[0017] In some embodiments, it also includes: Converting and processing the received real-time interactive data based on the driving video to generate real-time interactive audio data; The resource machine in the lip reading drive and distribution service cluster performs real-time matching of the interactive audio features in the real-time interactive audio data with the lip features, generates a real-time interactive drive video, and streams it to the distribution network.

[0018] The present application also provides a digital character live interactive method, comprising: In response to a first generation request for the content to be explained, determining a designated resource machine for generating audio data from the audio generation service cluster, and sending the first generation request to the designated resource machine; wherein the first generation request carries the content to be explained; In response to the first generation request, the designated resource machine converts the content to be explained to generate the audio data; In response to a second generation request for the audio data, the lip reading driving and distribution service cluster generates a digital character driving video according to the determined digital character and the audio data and streams the digital character driving video to a distribution network.

[0019] The present application also provides a computer storage medium, characterized in that it includes a computer program. When the computer program is run on an electronic device, the electronic device executes the steps executed by the execution side in the above-mentioned digital character audio and video data processing system; or, executes the steps executed by the execution side in the above-mentioned digital character live broadcast system; or, executes the steps in the above-mentioned digital character audio and video data processing method; or, executes the steps in the above-mentioned digital character live broadcast interaction method.

[0020] The present application also provides an electronic device, comprising: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the steps executed by the execution side in the above-mentioned digital character audio and video data processing system; or, executes the steps executed by the execution side in the above-mentioned digital character live broadcast system; or, executes the steps in the above-mentioned digital character audio and video data processing method; or, executes the steps in the above-mentioned digital character live broadcast interaction method.

[0021] Compared with the prior art, this application has the following advantages: The present application provides a system for processing audio and video data of digital characters, which splits the main data processing processes involved in the processing into two clusters for respective data processing, namely: an audio generation service cluster and a lip reading drive and distribution service cluster. The audio generation service cluster performs data processing of converting text to audio, and the lip reading drive and distribution service cluster executes lip shape according to audio drive to generate a driving video process of the digital character.

[0022] On the one hand, through the mutual cooperation between the audio generation service cluster and the lip reading drive and distribution service cluster, in the process of generating digital character driven videos, it is effectively avoided that the two algorithms of TTS and lip driving occupy resources due to a single service device, as well as the bottleneck problem caused by the computing power being unable to meet the processing needs, while also reducing the cost increase caused by deploying multiple service devices to solve the bottleneck problem.

[0023] On the other hand, the audio generation service cluster and the lip reading drive and distribution service cluster can respectively deploy TTS and lip driving algorithms with larger basic models and stronger processing capabilities to avoid the problem of unsatisfactory processing capabilities due to computing power limitations; and, since the audio generation service cluster and the lip reading drive and distribution service cluster respectively perform corresponding processing, the processing capabilities of either of them can be enhanced.

[0024] On the other hand, the audio data involved in this application can be processed before the specific application scenario is started and stored in the scheduling server, so as to realize the generation of digital character driving video by preloading, and in the case of real-time interactive data, the real-time audio data corresponding to the real-time interactive data is generated by TTS, and the real-time interactive driving video of the digital character is generated by the lip driving algorithm, so as to realize the seamless connection between the preloaded data and the real-time interactive data. In addition, the content to be explained can be pre-written text data, and the corresponding audio data can be generated and sentence-by-sentence processing can be performed before the specific application scenario is started, so as to reduce the computational cost of the execution of the specific application scenario, and support the sentence-level real-time interactive capability in the specific application scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a structural diagram of a digital character audio and video data processing system provided by the present application.

[0026] Figure 2 It is a structural diagram of a digital character live broadcast system provided by this application.

[0027] Figure 3 This is a flowchart of a method for processing digital character audio and video data provided by the present application.

[0028] Figure 4 It is a structural schematic diagram of a digital character audio and video data processing device provided by the present application.

[0029] Figure 5 This is a flow chart of a digital character live interactive method provided by this application.

[0030] Figure 6 It is a timing diagram of an interaction embodiment in a digital character live broadcast interaction method provided by the present application.

[0031] Figure 7 It is a structural schematic diagram of a digital character live interactive device provided by the present application.

[0032] Figure 8 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0033] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0034] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0035] Video live streaming has become a part of daily life. For example, learning, shopping, games, cultural and tourism introductions, etc. can all be used to transmit information in real time through video live streaming. However, in the current live streaming scenario, live streaming has certain limitations because it is restricted by time, venue, characters and other conditions. However, the digital character live streaming method is no longer restricted by time, venue, characters and other conditions, and compared with live streaming, digital character live streaming can reduce the cost of live streaming.

[0036] A digital character refers to a digital person or other digital image. In this embodiment, a digital person is mainly used as an example for explanation, so the following digital character is a digital person. Of course, the digital character is not limited to the digital person example. When different scenarios have different requirements, any digital character is applicable.

[0037] Digital human is a virtual character created by computer technology and graphics algorithms. It simulates human appearance, behavior and performance in a realistic way based on computer-generated models and animation technology.

[0038] Digital human live broadcast refers to the use of virtual characters or virtual anchors or hosts generated by digital artificial intelligence to conduct real-time live broadcast activities. These digital human anchors can disseminate and interact with relevant information on the live broadcast platform through pre-recorded audio, video and animation data combined with real-time speech synthesis and virtual reality technology.

[0039] Based on the above content, the present application provides a digital character audio and video data processing system, which can solve the computing power bottleneck problem and improve the processing speed and efficiency of digital characters in live broadcast scenarios. Before describing in detail a digital character audio and video data processing system provided by the present application, the relevant technical terms involved in the embodiments of the present application are first generally introduced, and the specific contents will be integrated and described in the process of implementing the solution, as follows: Digital human live broadcast: the process of generating audio through text, using the audio to drive the digital human’s movements, and broadcasting live through the digital human image.

[0040] Lip drive: The process of extracting audio features and using artificial intelligence (AI) to drive the lips to open and close while speaking, and to achieve accurate matching of lips with audio features.

[0041] TTS: Text To Speech. In digital human live broadcast, TTS technology may be used to convert text information (such as the lines of the digital human host, barrage comments, etc.) into voice, thereby realizing the voice output of the digital human host.

[0042] CDN: Content Delivery Network. CDN is an intelligent virtual network built on the existing network. It deploys edge servers in various locations and uses the load balancing, content distribution, scheduling and other functional modules of the central platform to enable users to obtain the required content nearby.

[0043] FPS: Frames Per Second. FPS describes the number of frames displayed per second in a video. It is a standard for measuring the amount of dynamic video information. In layman's terms, it is the number of frames that can be updated per second when a video or animation is played. This indicator is directly related to the smoothness and coherence of the visual effect, and is one of the key factors in evaluating video and game experience.

[0044] GRTN: Global Realtime Transport Network. GRTN is based on the central area of ​​the public cloud and edge cloud nodes, integrating the transmission and exchange of audio and video streams in various scenarios such as Internet live broadcast and RTC (real-time audio and video communication). GRTN aims to meet the high requirements of audio and video real-time, interactivity and stability in the mobile Internet era.

[0045] Cluster: A group of computers that connect multiple computing nodes (which can be physical servers or virtual machines) through a network to work together.

[0046] Based on the above content, a digital character audio and video data processing system provided by the present application is described in detail.

[0047] like Figure 1 As shown, Figure 1 It is a structural diagram of a processing system based on digital character audio and video data provided by the present application, the system comprising: an audio generation service cluster 11, a lip reading drive and distribution service cluster 12, and a scheduling server 13.

[0048] The purpose of the audio generation service cluster 11 is to convert text information into audio information.

[0049] The purpose of the lip reading driving and distribution service cluster 12 is to match the audio information with the lips of the digital character, and based on the matching relationship, the lips of the digital character move according to the audio features, thereby generating a driving video of the digital character (also referred to as digital character audio and video data).

[0050] The purpose of the scheduling server 13 is to receive and forward data, and schedule resource machines in the cluster.

[0051] The specific implementation process of the scheduling server 13 may include the following contents: In this embodiment, the scheduling server is used to receive the content to be explained from the anchor, select a designated resource machine from the audio generation service cluster and send the content to be explained to the designated resource machine for processing, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster.

[0052] Among them, the content to be explained can be determined according to the application scenario, for example: text information for introducing commodities, text information for introducing scenic spots, text information for introducing games, text information for introducing learning methods, etc. That is, in different application scenarios, the object to be explained has multiple forms, which can be a physical object with a physical structure, or a virtual object, etc. The purpose is to have a more detailed understanding of the object to be explained through the text information of the object to be explained. Therefore, the dispatch server can receive data information related to the object to be explained, including text, images, etc., and of course, it can also be data in the form of voice. In this embodiment, it is mainly described in the form of text information. However, it does not mean that it is limited to this embodiment. With the development of artificial intelligence technology, the relevant data information of the object to be explained can be in a multimodal form. For example, when it is an image, the relevant information in the image can be identified by artificial intelligence and converted into text information. When it is voice information, it can also be converted into text information, so as to ensure the accuracy of subsequent matching. Of course, the voice information can also be directly forwarded. Considering that the content to be explained in the prior art is mostly in the form of text information, therefore, this embodiment is described by taking the text information form as an example, but it does not mean that the content to be explained is limited to the text information form.

[0053] When the scheduling server selects the designated resource machine in the audio generation service cluster, at least two implementation methods may be included: Method 1 is: The scheduling server obtains task statistics of the resource machine from the audio generation service cluster according to the obtained content to be explained; According to the task statistical data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

[0054] The task statistics data may be to count the number of tasks of the resource machines in the audio generation service cluster, and select the resource machine whose number of tasks meets the selection requirements to be determined as the designated resource machine. Among them, the selection requirement may be to select the resource machine with the least number of tasks as the designated resource machine within the task statistics range, and the small number of tasks can also represent that the resource machine has few tasks waiting in line. It may also be to select the resource machine whose number of tasks is less than or equal to the number threshold to be determined as the designated resource machine. When there are two or more resource machines less than the number threshold, any one may be randomly selected as the designated resource machine, or the resource machine with the least number of tasks may be selected from two or more resource machines as the designated resource machine. When there are two or more resource machines equal to the number threshold, any one may be randomly selected as the designated resource machine, or the resource machine with the least number of tasks may be selected according to the start time of the current task execution. Generally, the earlier the start time of the task execution, the earlier the corresponding end time. Therefore, the resource machine with an earlier start time of the task execution may be determined as the designated resource machine. Of course, it can also be determined according to the amount of received data. Therefore, the determination method for the designated resource machine is not limited to the implementation example given above.

[0055] Method 2 is: The scheduling server obtains the working status data of the resource machine from the audio generation service cluster according to the obtained content to be explained; According to the working status data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

[0056] The scheduling server can learn the working status of the resource machine in the audio generation service cluster, and the working status data of the resource machine may include: idle, busy, standby, etc. In this embodiment, the designated resource machine can be selected according to the idle and / or standby working status data as the selection requirement.

[0057] In this embodiment, the scheduling server can determine and schedule the designated resource machine based on the task statistics data or work status data of the resource machines in the audio generation service cluster, thereby avoiding the situation of sending the content to be explained to a resource machine with more execution tasks before processing the content to be explained, thereby providing a basis for improving the processing efficiency of the content to be explained.

[0058] It is understandable that the above-mentioned method 1 and method 2 can be selected alone or in combination with each other, for example, the selection or determination of a designated resource machine can be achieved by setting weights.

[0059] Based on the above, after determining the designated resource machine, the corresponding data needs to be sent to the designated resource machine. The entire driving video (also called digital character audio and video data) generation process requires not only the audio generation service cluster to perform separate data processing, but also the lip reading drive and distribution service cluster to perform separate data processing. The two also need to cooperate with each other to realize the overall data processing process. The audio generation service cluster and the lip reading drive and distribution service cluster will be described below.

[0060] In this embodiment, the audio generation service cluster and the lip reading drive and distribution service cluster correspond to two data processing capabilities respectively. The audio generation service cluster is used to realize the conversion between text and audio through TTS, that is, to convert the text information of the content to be explained into audio data. The lip reading drive and distribution service cluster is used to realize the process of the explainer explaining according to the audio data through the lip shape drive algorithm. In this embodiment, the two algorithms involved in the processing of digital human audio and video data are split and controlled, and the computing tasks are undertaken by different service clusters respectively, and the generation process of digital human audio and video data is realized by combining the two, thereby avoiding the computing bottleneck problem existing in the single computing device in the execution of digital human audio and video data processing.

[0061] Generally, in the digital human live broadcast application scenario, the requirements for the real-time and smoothness of data are relatively high. Therefore, in this embodiment, the resource machines of the audio generation service cluster and the resource machines of the lip reading drive and distribution service cluster can be deployed with relevant basic dependencies to achieve corresponding data processing capabilities.

[0062] CUDA Tooolki, Pytorch, Conda and other basic dependencies can be deployed inside the image of the resource machine of the audio generation service cluster and the resource machine of the lip reading driver and distribution service cluster, so that the resource machine of the audio generation service cluster can undertake a larger TTS algorithm service, and the resource machine of the lip reading driver and distribution service cluster can undertake the lip shape driver algorithm service. Among them, the CUDA is an open source parallel computing platform and programming model that allows developers to use GPU for high-performance computing. CUDA Toolkit is a development toolkit that contains compilers, libraries, tools and documents required for CUDA programming. For the TTS algorithm, it requires a large amount of computing resources for model training and reasoning. CUDA Toolkit provides the ability to use GPU for accelerated computing, which can significantly improve the execution performance of the TTS algorithm and lip shape driver algorithm. For example, when training neural network models such as WaveNet, Tacotron or FastSpeech, using CUDA can greatly shorten the training time. The Pytorch is an open source deep learning framework that provides powerful GPU acceleration functions and supports dynamic calculation graphs, which can be more flexible for model development and debugging. Pytorch is one of the commonly used frameworks for building and training TTS models. In PyTorch, various neural network models (such as WaveNet, Tacotron, FastSpeech, etc.) can be used to implement TTS functions. The Conda is an open source package management system and environment manager, mainly used for scientific computing package management in the Python language. It allows users to easily create and manage multiple environments, thereby avoiding dependency conflicts between different projects. In the development and deployment process of TTS and lip-driven, Conda can help users manage different versions of Python and dependent libraries, ensure that TTS and lip-driven projects have correct dependencies, thereby helping to simplify the setup and management process of TTS and lip-driven algorithm projects and improve development efficiency. In addition, Conda can also be used to install dependencies such as PyTorch and CUDA Toolkit to provide necessary support for the development and operation of TTS and lip-driven algorithms.

[0063] It is understandable that deploying the above basic dependencies inside the resource machine image in the audio generation service cluster can improve the processing performance of TTS. Similarly, deploying the above basic dependencies inside the resource machine image in the lip reading drive and distribution service cluster can also provide the processing performance of the lip shape drive algorithm. Of course, according to the requirements of different application scenarios for digital human audio and video, the basic dependencies deployed inside the resource machine of the audio generation service cluster and the resource machine image of the lip reading drive and distribution service cluster may also be different. For example: when the performance of the resource machine of the audio generation service cluster and the resource machine of the lip reading drive and distribution service cluster itself meets the scene processing requirements, other basic dependencies can be appropriately deployed. For example: the deep learning framework can be TensorFlow or MXNet, and the programming model can also be replaced by other alternatives, etc., and examples will not be given one by one here.

[0064] It should be noted that when deploying basic dependencies within the resource machine and lip reading driver of the audio generation service cluster and the resource machine image of the distribution service cluster, actual needs can be combined to meet the computing power performance requirements of the resource machine and lip reading driver of the audio generation service cluster and the resource machine of the distribution service cluster, and it is not limited to the above.

[0065] The specific implementation processes of the audio generation service cluster and the lip reading drive and distribution service cluster are described below.

[0066] The specific implementation process of the audio generation service cluster 11 may include the following contents: The audio generation service cluster 11 includes at least two resource machines, and the designated resource machine is used to generate and send audio data according to the received content to be explained.

[0067] In this embodiment, the audio generation service cluster is a TTS computing power cluster, including at least two resource machines that can respectively execute the TTS algorithm. The designated resource machine needs to convert the text data of the content to be explained into audio data. The specific implementation process may include: The text data is preprocessed to determine the target text data, including: removing irrelevant characters, text normalization (such as standardized expression of dates and numbers), vocabulary analysis and segmentation, and other text data preprocessing processes.

[0068] Perform language analysis on the target text data, including part-of-speech tagging, syntactic analysis, semantic understanding, etc., to understand the intent and context of the text and to provide basic processing for generating natural and fluent speech.

[0069] ‌Convert the results of language analysis into actual speech output, including the construction of a sound database, the selection and connection of sound units, and the generation of the final speech through a synthesis algorithm. ‌ The TTS algorithm may include: a splicing method and a parameter method.

[0070] The splicing method is to select the required basic units from a large number of pre-recorded speech and splice them. The basic unit can be a syllable, a phoneme, etc. In order to pursue the coherence of the synthesized speech, a diphone (from the middle of one phoneme to the middle of the next phoneme) is often used as a unit.

[0071] The parameter method is to generate speech parameters (including fundamental frequency, formant frequency, etc.) at every moment according to a statistical model, and then convert these parameters into waveforms.

[0072] Since the TTS algorithm belongs to the prior art, it will not be introduced in detail here.

[0073] It should be noted that the training of the TTS algorithm model in this embodiment is not limited to the language type, and can be Chinese, English, or a mixture of Chinese and English. Of course, it is not limited to the above language type examples, so it can expand the scope of digital character driven video generation.

[0074] In this embodiment, the designated resource machine converts the text data of the content to be explained into audio data. The audio data can be sent in the PCM audio file format. PCM audio file is a pulse code modulation file, which is an audio file with streaming media characteristics, that is, it can be read and played at the same time. Of course, in this embodiment, only the PCM audio file format is used as an example for description, and the actual audio data can also include other formats, such as: WAV files, MP3 files, etc. Therefore, the file format in which the converted audio data is stored can be determined according to specific needs, and is not limited to the above examples.

[0075] In this embodiment, the audio data may be sent to the scheduling server, or may be sent to a designated resource machine in the lip reading driving and distribution service cluster.

[0076] In order to improve the response speed, reduce network delay and optimize resource utilization, this embodiment gives the audio data a pre-loading capability in the PCM audio file format, that is, the audio data is stored in the PCM audio file format, and can be read from the cache in real time according to subsequent processing requirements, or sent in real time. In this embodiment, the designated resource machine of the audio generation service cluster is used to send the generated audio data to the scheduling server, and the scheduling server stores the audio data. The scheduling server can send the audio data to the designated resource machine of the lip reading drive and distribution service cluster for subsequent data processing.

[0077] It can be understood that in the live broadcast application scenario, the designated resource machine of the audio generation service cluster in this embodiment can generate corresponding PCM audio materials for the content to be explained before the live broadcast starts. In this process, the sentence segmentation of the data to be explained is realized, thereby reducing the computing cost during the live broadcast. Moreover, because the content to be explained has been segmented into sentences before the live broadcast, the audio data is also sentence-level audio data, thus providing a basis for real-time interaction at the sentence level during the subsequent live broadcast.

[0078] It should be noted that in this embodiment, the pre-generation mode can be used for the generation of audio data, and the audio data can be sent to the scheduling server through the designated resource machine of the audio generation service cluster, and the scheduling server forwards the data to the designated resource machine of the lip reading drive and distribution service cluster for subsequent lip driving processing. In some other embodiments, the designated resource machine of the audio generation service cluster can also send the generated audio data to the designated resource machine of the lip reading drive and distribution service cluster for subsequent data processing.

[0079] The specific implementation process of the lip reading driving and distribution service cluster 12 may include the following contents: The resource machine in the lip reading drive and distribution service cluster is used to generate a drive video based on the audio data and the selected digital character in real time or on a scheduled basis during the streaming process, and stream the drive video to the distribution network. The lip reading drive and distribution service cluster may include at least two resource machines, and the selected resource machine may be used as the designated resource machine to generate the drive video, that is, to drive the lips of the digital character to move according to the received audio data, and stream the digital character drive video to the distribution network.

[0080] In this embodiment, the lip reading drive and distribution service cluster is a lip shape or lip driving computing power cluster, including at least two resource machines capable of executing a lip shape driving algorithm. In this embodiment, the method for determining the designated resource machine can also be the same as the method for determining the designated resource machine in the above-mentioned audio generation service cluster, using the task statistics data and / or working status data of the resource machine in the lip reading drive and distribution service cluster to achieve the determination of the designated resource machine in the lip reading drive and distribution service cluster.

[0081] It is understandable that, when combined with specific application scenarios, such as in a live broadcast scenario, the designated resource machine in the lip reading drive and distribution service cluster not only assumes the ability of lip shape driving but also assumes the ability of real-time streaming. At this time, the designated resource machine in the lip reading drive and distribution service cluster can be determined based on the binding relationship between the resource machine and the live broadcast platform. However, in the driving video generation stage, the determination of the designated resource machine in the lip reading drive and distribution service cluster is not limited to the binding relationship. In this embodiment, when the application scenario is a live broadcast scenario, the resource machine in the lip reading drive and distribution service cluster includes the relevant execution capabilities of lip shape driving and streaming. In fact, in other embodiments, the resource machine in the lip reading drive and distribution service cluster can be limited to executing lip shape driving, and the resource machine in the deployed streaming service cluster can perform related streaming tasks. This application is mainly described by taking the resource machine in the lip reading drive and distribution service cluster including the relevant execution capabilities of lip shape driving and streaming as an example.

[0082] Whether it is the resource machine in the audio generation service cluster or the resource machine in the lip reading drive and distribution service cluster, it can also be understood as computing power resources, such as computing power servers, etc.

[0083] Referring to the above description of the audio generation service cluster, the resource machine in the lip reading drive and distribution service cluster here will also deploy related basic dependencies to achieve lip shape driving and streaming. In this embodiment, the process of executing lip shape driving by the designated resource machine in the lip reading drive and distribution service cluster may include: The designated resource machine in the lip-reading driving and distribution service cluster is used to obtain the image data of the lecturer; wherein, the image data of the lecturer may include an image of the lecturer or a video of the lecturer, which may be used as reference data (reference image of the lecturer or reference video of the lecturer, and including facial data). In this embodiment, a digital character refers to a virtual character image created by computer technology (such as 3D modeling, motion capture, artificial intelligence, etc.) in the field of technology and science. These digital people may be completely fictitious or recreated based on real people. The image data of the lecturer may include the above two situations, namely: a virtual character image (including cartoon images, etc.) and a real character image. The image data of the lecturer in this embodiment may be a pre-recorded video of the lecturer, a real-time collected video of the lecturer, or an image of the lecturer provided by the live broadcast end. It is understandable that the acquisition method of the lecturer image data is not limited to the acquisition method from the live broadcast end, and the system may also automatically acquire it from other links, for example: from the lecturer image database, etc. The acquired lecturer image data needs to be identified to determine the lip position of the lecturer. In this embodiment, the digital character can be selected through the image data of the explainer, that is, the digital character (for example, digital anchor) in the digital character driving video is determined based on the image data of the explainer.

[0084] It should be noted that, in this embodiment, the image data of the explainer (selection of the digital character) involved are all data obtained after the explainer authorizes or authorizes the explainer. Of course, in specific application scenarios, such as live broadcast scenarios, when the live broadcast end sends the explainer image data, it is authorized by default.

[0085] The designated resource machine of the lip reading driving and distribution service cluster needs to match the extracted lip features in the narrator image data with the audio features in the audio data during the lip shape driving process, so as to drive the lip features to move and change according to the audio features, and generate the digital character driving video (also called digital human driving video). In other words, the changes in the lip shape of the narrator need to be operated according to the audio information. Therefore, there is a matching relationship between the lip shape features and the audio features.

[0086] The driving of the lip features to perform motion changes according to the audio features may adopt a Hallo2 algorithm or a wav2lip algorithm.

[0087] The Hallo2 algorithm is an audio-driven portrait image animation technology, the core of which is to convert audio features into realistic facial animation. The specific implementation process may include: Preprocess the input audio data to extract key audio features, such as pitch, volume, speaking speed, etc.

[0088] According to the extracted audio features, the corresponding facial animation parameters are generated. These parameters include the shape of the lips, the degree of opening and closing, the position of the tongue, etc., which are used to describe the changes in the mouth shape synchronized with the audio content, and of course, facial expressions.

[0089] The Generative Adversarial Network (GAN) structure is adopted, which consists of two parts: the generator and the discriminator. The generator is responsible for generating realistic facial animation images based on audio features, while the discriminator is used to evaluate the authenticity of the generated images and the synchronization between the generated images and the audio content.

[0090] The Hallo2 algorithm also includes time alignment, for example by analyzing the timing relationship between audio features and facial animation parameters to ensure that the generated facial animations are temporally consistent with the audio content.

[0091] The Hallo2 algorithm also includes detailed processing of the generated facial animation, such as adjusting the texture and gloss of the lips to improve its realism and visual effects.

[0092] The wav2lip algorithm is a lip shape driving algorithm based on deep learning. The specific implementation process may include: The input audio is preprocessed to extract audio features. These speech features usually include Mel-frequency cepstral coefficients (MFCCs), spectrograms, etc., which are used to describe the acoustic characteristics of speech.

[0093] Extract lip shape features from the speaker's reference video (which may only contain the face part). This can be achieved through deep learning models such as convolutional neural networks (CNNs) to capture features such as the shape and color of the lip area.

[0094] Through the pre-trained phoneme-to-lip shape mapping model, the correspondence between audio features and lip shape features is learned, so that lip shape animations that match the audio features in the audio data are generated.

[0095] The wav2lip algorithm can also perform post-processing on the generated lip animation, such as adjusting the smoothness of the lip animation, removing noise, etc., so as to improve its realism and visual effects.

[0096] The above two algorithm examples are only used to describe lip drive. And the above two algorithms also belong to the existing technology of lip drive. The Hallo2 algorithm can improve the clarity of digital human audio and video data, and pay more attention to the overall generation and detail processing of facial animation, while the wav2lip algorithm focuses on the generation of lip animation and the synchronization of voice content. Therefore, different algorithms can be used in combination with the specific application scenarios of lip drive. For example: This embodiment mainly involves the live video scene, so it focuses more on the overall generation of facial animation, so the Hallo2 algorithm can be used.

[0097] It can be understood that the lip driving in this embodiment is not limited to the examples given above, and any method that can achieve synchronization between lips and audio can be used as the implementation of lip driving in this application.

[0098] In this embodiment, the resource machine in the lip reading drive and distribution service cluster can perform lip driving and push streaming distribution, that is, the lip reading drive and distribution service cluster is a lip driving and distribution cluster. For push streaming, the resource machine in the lip reading drive and distribution service cluster can use the AI ​​audio and video cross-platform software development toolkit to realize the push streaming of digital character driven video, that is, distribution. Taking into account the real-time requirements for video streaming in live broadcast application scenarios and user experience, in this embodiment, ARTC SDK (Ali Real-Time Communication SDK: Real-time Communication Development Toolkit) can be used for push streaming to achieve low-latency streaming push and pull streaming.

[0099] In this embodiment, the digital character driving video is sent to RTS NetSDK through the push streaming ARTC SDK, and the RTSNetSDK transmits the digital character driving video based on the global real-time transmission network GRTN (Global Real Time Network). GRTN is a low-latency, fully distributed and sinking communication-level streaming media transmission network deployed on edge cloud nodes, which is used to support video conferencing, live video, real-time audio and video and other services. ARTC SDK applies WebRTC (Web Real-Time Communication) real-time communication technology, such as bandwidth estimation (BWE), jitter buffer, network equalizer (NetEQ), forward error correction (FEC), negative acknowledgment (NACK) and other QoS (Quality of Service) mechanisms, which can optimize transmission delays and improve real-time performance, and ensure low-latency push streaming or distribution of video frames in digital human-driven videos.

[0100] It is understandable that the designated resource machine in the lip reading drive and distribution service cluster can select the streaming method according to the requirements of the specific application scenario. In this embodiment, the main application scenario is video live broadcast, so the real-time communication development toolkit is used to achieve low-latency streaming. In scenarios where the real-time requirements for video streaming are not high, other methods can be used for streaming or distribution.

[0101] In the digital human live broadcast application scenario, the designated resource machine in the audio generation service cluster can also receive real-time interactive data and generate real-time interactive audio data corresponding to the real-time interactive data; the resource machine in the lip reading drive and distribution service cluster is used to generate a digital character real-time interactive drive video according to the real-time matching relationship between the interactive audio features of the received real-time interactive audio data and the lip features, and distribute it. That is to say, in this embodiment, after the digital character drive video is generated and distributed, it is also possible to receive real-time interactive data returned from the push stream end (for example: user end or live broadcast viewing end), such as bullet screen text data, so that the corresponding interactive audio data can be generated according to the bullet screen text data, and the digital character real-time interactive drive video can be generated by the designated resource machine in the lip reading drive and distribution service cluster, thereby realizing data interaction, avoiding the computing power bottleneck problem in the prior art, and improving the user experience of real-time interaction.

[0102] It is understandable that when the processing system for digital character audio and video data provided by the present application performs related processing in combination with specific application scenarios, it takes into account that different application scenarios have different requirements for data processing. Therefore, the audio generation service cluster and the lip reading drive and distribution service cluster can be elastically scaled to achieve expansion and / or reduction in combination with the requirements of specific application scenarios for data processing. In this way, the data processing efficiency and user experience can be improved in scenarios with higher data processing requirements; in scenarios with lower data processing requirements, computing resource overhead can be saved.

[0103] The above is a description of an embodiment of a processing system for digital human audio and video data provided by the present application. The system splits the main data processing processes involved in the generation process into two clusters for respective data processing, namely: an audio generation service cluster and a lip reading drive and distribution service cluster. The audio generation service cluster performs data processing of text to audio, and the lip reading drive and distribution service cluster executes the process of generating digital character driven video by lip shape according to audio drive.

[0104] On the one hand, through the mutual cooperation between the audio generation service cluster and the lip reading drive and distribution service cluster, in the process of generating digital character driven videos, it is effectively avoided that the two algorithms of TTS and lip driving occupy resources due to a single service device, as well as the bottleneck problem caused by the computing power being unable to meet the processing needs, while also reducing the cost increase caused by deploying multiple service devices to solve the bottleneck problem.

[0105] On the other hand, the audio generation service cluster and the lip reading drive and distribution service cluster can respectively deploy TTS and lip driving algorithms with larger basic models and stronger processing capabilities to avoid the problem of unsatisfactory processing capabilities due to computing power limitations; and, since the audio generation service cluster and the lip reading drive and distribution service cluster respectively perform corresponding processing, the processing capabilities of either of them can be enhanced.

[0106] On the other hand, the audio data involved in this application can be processed before the specific application scenario is started and stored in the scheduling server, so as to realize the generation of digital character driving video by preloading, and in the case of real-time interactive data, the real-time audio data corresponding to the real-time interactive data is generated by TTS, and the real-time interactive driving video of the digital character is generated by the lip driving algorithm, so as to realize the seamless connection of the digital character driving video between the preloaded data and the real-time interactive data. In addition, the content to be explained can be pre-written text data, and the corresponding audio data can be generated and sentence-by-sentence processing can be performed before the specific application scenario is started, so as to reduce the computational cost of the execution of the specific application scenario, and support the sentence-level real-time interactive capability in the specific application scenario.

[0107] Based on the above, the present application also provides a digital character live broadcast system, which is described in the form of an application scenario, and mainly focuses on the implementation process of the above-mentioned digital character audio and video data processing system in a video live broadcast scenario.

[0108] like Figure 2 As shown, Figure 2 This is a structural diagram of a digital character live broadcast system provided by the present application, which includes: an anchor terminal, a client terminal, a scheduling server, an audio generation service cluster, a lip reading driver and a distribution service cluster.

[0109] The scheduling server is used to select a designated resource machine in the audio generation server cluster for data conversion processing according to the content to be explained of the live broadcast terminal, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster; in this embodiment, the content to be explained can be an object that needs to be explained and described during the video live broadcast, such as: goods, scenic spots, games, real estate, etc. For the specific content of the scheduling server, please refer to the relevant description in the above-mentioned digital character audio and video data processing system, which will not be described in detail here.

[0110] The resource machine included in the audio generation service cluster is used to generate audio data according to the received content to be explained; in this embodiment, the audio data can be sent to the scheduling server and / or the resource machine in the lip reading drive and distribution service cluster through the designated resource machine in the audio generation service cluster. In the specific process, the sending to the scheduling server is described as an example, but it is not limited to this method. Similarly, the specific content of the audio generation service cluster can refer to the relevant description in the above-mentioned digital character audio and video data processing system, which will not be described in detail here.

[0111] The resource machine included in the lip-reading drive and distribution service cluster is used to generate a drive video based on the audio data and the selected digital character in real time or on a scheduled basis during the streaming process, and stream the drive video to the distribution network; in this embodiment, the selected digital character can be a digital character that generates a digital character anchor video through the image data of the explainer, and the image data of the explainer can be an image or a video. Regarding the ability of the lip-reading drive and distribution service cluster to generate a digital character drive video and stream the digital character drive video to the client, the specific content can also be referred to the relevant description in the above-mentioned digital character audio and video data processing system, which will not be described in detail here.

[0112] The anchor end is used to send the content to be explained to the scheduling server and perform the live broadcast task; the anchor end mainly performs the trigger operation of starting the live broadcast, and can also provide the image data of the explainer. Of course, the image data of the explainer can also be provided by the live broadcast service platform, and the anchor end can also be understood as the live broadcast service platform or other service platform to implement the live broadcast. In this embodiment, the content to be explained can be sent to the scheduling server in advance according to the explanation order of the object to be explained before the video live broadcast starts or before the live broadcast starts, thereby providing a basis for the audio generation service cluster to pre-generate audio data.

[0113] The client is used to receive the live video stream of the digital character driving video. The client can be understood as the end side for watching the live broadcast, for example: a live broadcast service application platform based on electronic devices such as mobile phones. The client and the anchor end can directly interact in real time to generate real-time interactive data. In this embodiment, the client can pull the low-latency digital human driving video through the lip reading drive and the designated resource machine in the distribution service cluster to improve the viewing experience of the live video.

[0114] The above is a description of a digital character live broadcast system provided by the present application. For the specific implementation process of the system, please refer to the relevant content of the above-mentioned digital character audio and video data processing system, which will not be described in detail here.

[0115] Based on the above, the present application also provides a method for processing digital character audio and video data, such as Figure 3 As shown, Figure 3 This is a flowchart of a method for processing digital character audio and video data provided by the present application; the method comprises: Step S301: sending the received content to be explained to a designated resource machine selected from the audio generation server cluster; Step S302: the designated resource machine converts the content to be explained to generate audio data; Step S303: The resource machine in the lip reading driving and distribution service cluster generates a driving video in real time according to the received audio data and the selected digital character at a scheduled time or during the streaming process, and streams the driving video to the distribution network.

[0116] The step S301 may be to obtain the content to be explained, such as product copy data, from the scheduling server, and send the content to be explained to a designated resource machine selected by the scheduling server from the audio generation server cluster.

[0117] The step S302 is to convert the content to be explained into audio data through the TTS algorithm. The content to be explained can be obtained in sequence according to the order of the explanation of the explanation object. In the video live broadcast scenario, the content to be explained can be obtained in advance before the live broadcast starts, so as to pre-generate audio data before the broadcast starts. The specific implementation process may include: according to the obtained content to be explained, converting the text data in the content to be explained into the audio data; sending the audio data to the scheduling server for storage.

[0118] The step S303 can establish a matching relationship between the lip features of the digital character and the audio features in the audio data according to the selected digital character and the audio data before the broadcast, and generate a digital character driving video in which the lip features move according to the audio features according to the matching relationship. After the broadcast is triggered, the video stream of the digital character driving video can be distributed to the live broadcast receiving end side in real time, that is, the digital character driving video can be distributed according to the real-time transmission network.

[0119] During the live video broadcast, you can also include: Converting and processing the received real-time interactive data based on the driving video to generate real-time interactive audio data; The interactive audio features in the real-time interactive audio data are matched with the lip features in real time to generate a real-time interactive driving video, which is then pushed to the distribution network.

[0120] For the specific contents of the above steps S301 to S303, reference may be made to the related description of the above-mentioned digital character audio and video data processing system, which will not be described in detail here.

[0121] The above is a detailed description of an embodiment of a method for processing digital character audio and video data provided by the present application. Corresponding to the embodiment of a method for processing digital character audio and video data provided above, the present application also discloses an embodiment of a device for processing digital character audio and video data. Figure 4 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0122] like Figure 4 As shown, Figure 4 : is a structural diagram of a device for processing digital character audio and video data provided by the present application, the device comprising: The sending unit 401 sends the received content to be explained to a designated resource machine selected from the audio generation server cluster; Generating unit 402, the designated resource machine converts the content to be explained to generate audio data; The generation / streaming unit 403, a resource machine in the lip reading drive and distribution service cluster generates a digital character driving video in real time according to the received audio data and the selected digital character during timing or during the streaming process, and streams the digital character driving video to the distribution network.

[0123] The sending unit 401 includes: An acquisition subunit, used to acquire task statistics data or work status data of resource machines in the audio generation service cluster; A selection subunit is used to select the designated resource machine from the audio generation service cluster according to the task statistical data or the work status data.

[0124] The generating unit 402 may include: A conversion subunit, used for converting the text data in the content to be explained into the audio data; The storage subunit is used to store the audio data.

[0125] The generating / streaming unit 403 may include: A selected subunit is used to use the acquired narrator image data as a selected digital character; A generating subunit, used for matching the lip features extracted from the narrator image data with the audio features in the audio data, driving the lip features to move according to the audio features, and generating the driving video; The streaming subunit is used to stream the driving video to a distribution network.

[0126] Further, it may include: An interactive audio generating unit, used for converting and processing the received real-time interactive data based on the driving video to generate real-time interactive audio data; The interactive video generation unit is used for the resource machine in the lip reading drive and distribution service cluster to match the interactive audio features in the real-time interactive audio data with the lip features in real time, generate a real-time interactive drive video, and push it to the distribution network.

[0127] The above is a description of a digital character audio and video data processing device provided by the present application. The specific content of the device can refer to the relevant content of the above-mentioned system and method, which will not be described in detail here.

[0128] Based on the above, the present application also provides a digital character live interactive method, such as Figure 5 and Figure 6 As shown, Figure 5 is a flow chart of a digital character live interactive method provided by this application, Figure 6This is a timing diagram of an interactive embodiment of a digital character live interactive method provided by the present application. The interactive method includes: Step S501: In response to a first generation request for content to be explained, determine a designated resource machine for generating audio data from an audio generation service cluster, and send the first generation request to the designated resource machine; wherein the first generation request carries the content to be explained; regarding this step, reference may be made to the above-mentioned digital character audio and video data processing system and the digital character live broadcast system embodiment regarding the scheduling server, which will not be described in detail here.

[0129] Step S502: In response to the first generation request, the designated resource machine converts the content to be explained to generate the audio data; regarding this step, please refer to the relevant description of the audio generation service cluster in the above-mentioned digital character audio and video data processing system and digital character live broadcast system embodiment, which will not be described in detail here.

[0130] Step S503: In response to the second generation request for the audio data, the lip reading drive and distribution service cluster generates a digital character drive video according to the selected digital character and the audio data, and streams the digital character drive video to the distribution network; for this step, reference may be made to the relevant description of the lip reading drive and distribution service cluster in the above-mentioned digital character audio and video data processing system and digital character live broadcast system embodiments, which will not be described in detail here.

[0131] Furthermore, it may also include: According to the real-time interactive data obtained during the live broadcast of the digital character-driven video, real-time reply data is generated; the specific content of this step can refer to the relevant content of the above-mentioned scheduling server.

[0132] In response to the third generation request for the real-time reply data, real-time audio data corresponding to the real-time reply data is generated and sent to the scheduling server; the specific content of this step can refer to the above-mentioned relevant content about the audio generation service cluster.

[0133] In response to the fourth generation request for the real-time audio data, a real-time interactive driving video of a digital character is generated according to the real-time matching relationship between the real-time audio features in the received real-time audio data and the lip features, and the video is distributed. For the specific content of this step, please refer to the above-mentioned content about the lip reading driving and distribution service cluster.

[0134] Based on the above, the present application also provides a digital character live interactive device, such as Figure 7 As shown, Figure 7 This is a structural diagram of a digital character live interactive device provided by the present application, which may include: The generating / sending unit 701 is used to determine a designated resource machine for generating the audio data from the audio generating service cluster in response to a first generating request for the content to be explained, and send the first generating request to the designated resource machine; wherein the first generating request carries the content to be explained.

[0135] The audio generating unit 702 is used for, in response to the first generating request, the designated resource machine to convert the content to be explained to generate the audio data.

[0136] The generation / streaming unit 703 is used to respond to the second generation request for the audio data, and the lip reading drive and distribution service cluster generates a digital character driving video according to the determined digital character and the audio data and streams the digital character driving video to the distribution network.

[0137] Based on the above content, the present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the steps executed by the execution side in the above-mentioned digital character audio and video data processing system; or, executes the steps executed by the execution side in the above-mentioned digital character live broadcast system; or, executes the steps in the above-mentioned digital character audio and video data processing method; or, executes the steps in the above-mentioned digital character live broadcast interaction method.

[0138] Based on the above content, the present application also provides an electronic device, such as Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of an electronic device provided by the present application. The electronic device includes: Processor 801; The memory 802 is used to store a program for processing data generated by an electronic device. When the program is read and executed by the processor, it executes the steps executed by the execution side in the above-mentioned digital character audio and video data processing system; or, executes the steps executed by the execution side in the above-mentioned digital character live broadcast system; or, executes the steps in the above-mentioned digital character audio and video data processing method; or, executes the steps in the above-mentioned digital character live broadcast interaction method.

[0139] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0140] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0141] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0142] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0143] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0144] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A system for processing digital character audio and video data, characterized in that: include: Audio generation service cluster, lip reading drive and distribution service cluster and scheduling server; The scheduling server is used to receive the content to be explained by the anchor, select a designated resource machine in the audio generation server cluster for processing, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster; The resource machine included in the audio generation service cluster is used to generate audio data according to the received content to be explained; The resource machine in the lip reading driving and distribution service cluster is used to generate a digital character driving video according to the audio data and the selected digital character in real time at a scheduled time or during the streaming process, and stream the digital character driving video to the distribution network.

2. The digital character audio and video data processing system according to claim 1, characterized in that: The scheduling server is used to select a designated resource machine in the audio generation server cluster for processing according to the content to be explained received from the anchor terminal, including: The scheduling server obtains task statistics data or work status data of the resource machine from the audio generation service cluster according to the obtained content to be explained; According to the task statistical data or work status data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

3. The digital character audio and video data processing system according to claim 1, characterized in that: The resource machine is used to send the generated audio data to the scheduling server, and the scheduling server stores the audio data and sends the audio data to the lip reading drive and distribution service cluster.

4. The digital character audio and video data processing system according to claim 1, characterized in that: The lip reading driving and distribution service cluster includes a resource machine, which is used to obtain the image data of the explainer as the selected digital character, and match the lip features extracted from the explainer image data with the audio features in the audio data, drive the lip features to move according to the audio features, and generate the driving video.

5. The digital character audio and video data processing system according to claim 1, characterized in that: The resource machine in the audio generation service cluster is used to receive real-time interaction data and generate real-time interaction audio data corresponding to the real-time interaction data.

6. The digital character audio and video data processing system according to claim 5, characterized in that: The resource machine in the lip reading drive and distribution service cluster is used to generate real-time interactive drive video of the digital character according to the real-time matching relationship between the interactive audio features of the real-time interactive audio data and the lip features during the timing or in the streaming process, and stream it to the distribution network.

7. A digital character live broadcast system, characterized in that: include: Live broadcast terminal, client, scheduling server, audio generation service cluster, lip reading driver and distribution service cluster; The scheduling server is used to select a designated resource machine in the audio generation server cluster for data conversion processing according to the content to be explained of the live broadcast terminal, and pull the audio data generated by the designated resource machine to the lip reading drive and distribution service cluster; The audio generation service cluster includes a resource machine, which is used to generate audio data according to the received content to be explained; The resource machine included in the lip reading driving and distribution service cluster is used to generate a driving video according to the audio data and the selected digital character in real time or on a scheduled basis during the streaming process, and to stream the driving video to the distribution network; The live broadcast terminal is used to send the content to be explained to the scheduling server and execute the live broadcast task; The client is used to receive the live video stream of the driving video.

8. A method for processing digital character audio and video data, characterized in that: include: Sending the received content to be explained to a designated resource machine selected from the audio generation server cluster; The designated resource machine converts the content to be explained to generate audio data; The resource machine in the lip reading driving and distribution service cluster generates a digital character driving video in real time according to the received audio data and the selected digital character during timing or in the streaming process, and streams the digital character driving video to the distribution network.

9. The method for processing digital character audio and video data according to claim 8, characterized in that: The step of sending the received content to be explained to a designated resource machine selected from the audio generation server cluster includes: Obtaining task statistics data or work status data of resource machines in the audio generation service cluster; According to the task statistical data or work status data, a resource machine that meets the selection requirements is selected from the audio generation service cluster as the designated resource machine.

10. The method for processing digital character audio and video data according to claim 8, characterized in that: The designated resource machine converts the content to be explained to generate audio data, including: Converting the text data in the content to be explained into the audio data; The audio data is stored.

11. The method for processing digital character audio and video data according to claim 8, characterized in that: The resource machine in the audio data to lip reading driving and distribution service cluster generates a driving video in real time according to the received audio data and the selected digital character during the timing or streaming process, and streams the driving video to the distribution network, including: Using the acquired image data of the narrator as the selected digital character; Matching the extracted lip features in the narrator image data with the audio features in the audio data, driving the lip features to move and change according to the audio features, and generating the driving video; The driving video is pushed to a distribution network.

12. The method for processing digital character audio and video data according to claim 8, characterized in that: Also includes: Converting and processing the received real-time interactive data based on the driving video to generate real-time interactive audio data; The resource machine in the lip reading drive and distribution service cluster matches the interactive audio features in the real-time interactive audio data with the lip features in real time, generates a real-time interactive drive video, and pushes the video to the distribution network.

13. A digital character live interactive method, characterized in that: include: In response to a first generation request for the content to be explained, determining a designated resource machine for generating audio data from the audio generation service cluster, and sending the first generation request to the designated resource machine; wherein the first generation request carries the content to be explained; In response to the first generation request, the designated resource machine converts the content to be explained to generate the audio data; In response to a second generation request for the audio data, the lip reading driving and distribution service cluster generates a digital character driving video according to the determined digital character and the audio data and streams the digital character driving video to a distribution network.

14. A computer storage medium, characterized in that: The method comprises a computer program, which, when executed on an electronic device, causes the electronic device to execute the steps executed by the execution side in the processing system for digital character audio and video data according to any one of claims 1 to 6; Alternatively, the steps performed by the execution side in the digital character live broadcast system as claimed in claim 7 are performed; Or, executing the steps in the method for processing digital character audio and video data as claimed in any one of claims 8 to 12; Alternatively, execute the steps in the digital character live interactive method as described in claim 13.

15. An electronic device, characterized in that: include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the steps executed by the execution side in the processing system for digital character audio and video data according to any one of claims 1 to 6; Alternatively, the steps performed by the execution side in the digital character live broadcast system as claimed in claim 7 are performed; Or, executing the steps in the method for processing digital character audio and video data as claimed in any one of claims 8 to 12; Alternatively, execute the steps in the digital character live interactive method as described in claim 13.

Images