Digital character audio and video generation method and digital character live broadcast interaction method

Abstract

The invention discloses a digital role audio and video generation method and a digital role live broadcast interaction method, and the generation method comprises the steps: generating explanation data of a to-be-explained object according to a digital role material generation task of a requester; performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; performing role image digital processing on the obtained role image reference data to obtain a digital role image video; carrying out fusion processing on the audio data and the digital role image video to obtain a digital role driving video for explaining the object to be explained by using a digital role; returning the digital role driving video to the requester, so that the requester processes the digital role driving video before distribution to generate a digital role audio / video; therefore, computing resources are saved, the cost is reduced, and the coverage rate of digital role audio and video playing is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more particularly to a method and device for generating digital character audio and video, a method and device for digital character live interactive broadcast, and a digital character audio and video playback system. The present application also relates to a computer storage medium and an electronic device. Background Art

[0002] Live streaming is a form of real-time interaction and information dissemination that uses Internet technology to present audio and video content to the audience in the form of live streaming.

[0003] With the popularization of the Internet and the continuous development of computer technology, audio and video live broadcast technology has gradually matured and moved towards commercialization. In the early days of the Internet, audio and video live broadcasts were mainly based on text chats and voice calls, and the technology and bandwidth conditions for video live broadcasts were relatively limited. However, with the advancement of technology and the upgrade of the network, video live broadcast has gradually become a regular video playback method and has been widely used in entertainment, education, e-commerce and other fields.

[0004] Currently, there are two types of live video broadcasting: one is live broadcasting with real people, and the other is live broadcasting with digital characters. In live broadcasting with real people, the host is a real person, who transmits relevant information of the live broadcast in real time through the camera to interact with the user on the other end of the camera. In live broadcasting with digital characters, the host is a virtual digital character, which is generated by computer graphics, artificial intelligence and other technologies, and broadcasts automatically or semi-automatically according to preset programs or algorithms. Live broadcasting with digital characters is not restricted by time and place, and has higher flexibility and scalability. Summary of the invention

[0005] The present application provides a method for generating digital character audio and video, which is applied to a cloud service end, including: Generate the task according to the digital character material of the requesting party, and generate the explanation data of the object to be explained; Performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; Performing character image digitization processing on the acquired character image reference data to obtain a digital character image video; The audio data and the digital character image video are fused to obtain a digital character driving video in which the digital character explains the object to be explained; The digital character driving video is returned to the requesting party so that the requesting party can process it before distribution to generate digital character audio and video.

[0006] In some embodiments, the generating task according to the digital character material of the requesting party to generate explanation data of the object to be explained includes: Constructing prompt information for generating structured explanation data of the object to be explained according to the identification information of the object to be explained in the digital character material generation task, wherein the prompt information may include: attribute information of the object to be explained and description information describing the object to be explained; The prompt information is used as input information of the large model to obtain the explanation data including the digital character setting generated by the large model.

[0007] In some embodiments, the performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data includes: Performing sentence analysis on the explanation data to extract feature data used to generate the audio data; The explanation data is converted into the audio data according to the phoneme sequence feature data and the tone feature data in the feature data.

[0008] In some embodiments, the step of performing character image digitization processing on the acquired character image reference data to obtain a digital character image video includes: Acquire the character image reference data according to the digital character material generation task or through network search; The character image in the character image reference data is digitally processed to obtain the digital character image video.

[0009] In some embodiments, the step of digitally processing the character image in the character image reference data to obtain the digital character image video includes: Generating digital character image data including the head features according to the extracted head features of the character image reference data; The digital character image video is obtained by adjusting the digital character facial area and / or the digital character decoration style in the digital character image data.

[0010] In some embodiments, the fusing of the audio data and the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained, comprises: Extracting lip features in the digital character image video and audio features in the audio data; Establishing a matching relationship between the lip feature and the audio feature; According to the matching relationship, a digital character driving video is obtained in which the lip area of ​​the digital character undergoes movement changes according to the audio data.

[0011] In some embodiments, it also includes: Acquiring interaction data based on the distribution of the digital character's audio and video; According to the data conversion processing of the interactive data, interactive audio data corresponding to the interactive data is obtained; The interactive audio data and the digital character image video are fused to obtain a digital character interactive driving video in which the digital character explains according to the interactive audio data; The digital character interaction driving video is returned to the requesting party so that the requesting party can process it before distribution to generate digital character interaction audio and video.

[0012] The present application also provides a digital character live interactive method, comprising: In response to a request for generating audio and video playback in the audio and video playback platform, the digital character driving video obtained by the above-mentioned method for generating digital character audio and video is returned to the audio and video playback platform; Performing pre-distribution processing on the received digital character driving video to generate digital character audio and video; In response to a play trigger operation on the audio and video playback platform, distributing the digital character audio and video; In response to the distributed digital character audio and video, returning the digital character interactive driving video obtained according to the digital character audio and video generation method to the audio and video playback platform; The generated digital character interaction audio and video is distributed according to the pre-distribution processing of the received digital character interaction driving video.

[0013] The present application also provides a digital character audio and video playback system, including: a playback control terminal, a playback output terminal and a cloud service terminal; The playback control end is used to send a request for generating digital character audio and video to the cloud service end; The cloud service end is used to generate a task of the digital character material obtained according to the generation request, generate explanation data of the object to be explained, and perform data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; The cloud service end is further used to perform character image digital processing on the acquired character image reference data to obtain a digital character image video, and to fuse the audio data with the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained; The playback control end is used to perform pre-distribution processing on the digital character driving video returned by the cloud service end, generate digital character audio and video, and distribute the digital character audio and video to the playback output end; The playback output terminal is used to output the audio and video of the digital character.

[0014] In some embodiments, it also includes: The playback output terminal is used to send the interactive data generated based on the audio and video of the digital character to the cloud service terminal; The cloud service end is used to generate interactive reply data according to the interactive data, and perform data conversion processing on the interactive reply data to obtain interactive audio data; and fuse the interactive audio data with the digital character image video to obtain a digital character interactive driving video in which the digital character explains according to the interactive reply data; The playback control end is used to perform pre-distribution processing on the digital character interaction driving video returned by the cloud service end to generate digital character interaction audio and video, and distribute the digital character interaction audio and video to the playback output end.

[0015] In some embodiments, the playback control end is used to perform pre-distribution processing on the digital character driving video returned by the cloud service end, generate digital character audio and video, and distribute the digital character audio and video to the playback output end, including: Based on the triggering operation of the audio and video playback in the audio and video playback platform, the playback control end distributes the digital character audio and video to the playback output end.

[0016] The present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the steps in the above-mentioned method for generating digital character audio and video; or, executes the steps in the above-mentioned digital character live interactive method; or, executes the steps executed by the execution side in the above-mentioned digital character audio and video playback system.

[0017] The present application also provides an electronic device, comprising: processor; The memory is used to store a program for processing data generated by an electronic device. When the program is read and executed by the processor, it executes the steps in the above-mentioned method for generating digital character audio and video; or, it executes the steps in the above-mentioned digital character live interactive method; or, it executes the steps executed by the execution side in the above-mentioned digital character audio and video playback system.

[0018] Compared with the prior art, this application has the following advantages: The present application provides a method for generating digital character audio and video. On the one hand, the digital character audio and video playback is realized by combining the corresponding service capabilities of the cloud with the capabilities of the playback control terminal. The cloud mainly performs tasks such as AI reasoning and calculation in the process of generating digital character-driven videos, and then returns the generated digital character-driven videos to the playback control terminal, and performs relevant processing tasks such as rendering, editing, and distribution of digital character audio and video before distribution through the playback control terminal. The combination of the two can reduce the cloud deployment cost, and the cloud-related processing tasks can be combined with different application scenario requirements to migrate some tasks to the playback control terminal, which can further save computing resources and avoid the waste of computing resources on the playback control terminal, so that the two can achieve task processing balance in the process of combining them.

[0019] On the other hand, the method can build a complete processing chain from the generation of explanation data to audio data conversion, and the generation of digital character image video based on character image data, and then generate digital character driving video based on digital character image video and audio data, and then process the digital character driving video before distribution, which effectively reduces the generation and playback costs of digital character audio and video. At the same time, this process can also reduce the participation cost of the playback control end, thereby improving the coverage of digital character audio and video playback.

[0020] The present application provides a method for interactive live broadcast of digital characters. It only requires the live broadcast control end to make a request for the generation of digital character audio and video, and then obtain the generated digital character audio and video through a complete link, and then based on the start of broadcast triggering, a one-stop live broadcast process is realized. This process does not require the technical links to be separately segmented and processed, so that the process of generating digital character audio and video, starting playback, and then pushing the stream, as well as real-time interaction, can be realized through a complete link, thereby avoiding the high cost of broadcasting in the digital character live broadcast scenario, and the complex operation caused by the connection of various technical links, effectively reducing the cost and threshold of broadcasting, and improving the user experience of the live broadcast control end. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of a method for generating digital character audio and video provided by the present application.

[0022] Figure 2 This is a timing diagram of a method for generating digital character audio and video provided by the present application.

[0023] Figure 3 It is a structural schematic diagram of a device for generating digital character audio and video provided by the present application.

[0024] Figure 4 This is a flow chart of a digital character live interactive method provided by this application.

[0025] Figure 5 This is a timing diagram of a digital character live interactive method provided by the present application.

[0026] Figure 6 It is a structural schematic diagram of a digital character live interactive device provided by the present application.

[0027] Figure 7 It is a structural schematic diagram of a digital character audio and video playback system provided by the present application.

[0028] Figure 8 It is a structural schematic diagram of each end embodiment of a digital character audio and video playback system provided by the present application.

[0029] Fig. 9 It is a structural schematic diagram of an electronic device provided by this application. DETAILED DESCRIPTION

[0030] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0031] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The descriptions used in this application and the appended claims, such as "a", "first", and "second", are not limitations on quantity or sequence, but are used to distinguish the same type of information from each other.

[0032] Video playback types include: live video, long video, short video, etc. Regardless of the playback type, users can obtain information in an intuitive way. In some scenarios such as product introduction and tourist attraction introduction, the information to be conveyed by the video needs to be transmitted to the user through characters, and the video playback elements involved in this process include characters, venues, equipment, etc. Especially in the live video scene, these video playback elements will lead to a series of problems such as increased video playback costs and increased difficulty in personnel control. Therefore, digital characters replace real characters (or anchors) with advantages such as high controllability, low cost, and breaking the time limit during playback. Existing digital characters, as characters, whether in live video scenes or in playback types such as short videos or long videos, need to be separated and processed separately from the entire technical process, for example: from starting playback to digital character generation, the digital character is separated from the playback content (such as the application of speech and repeated use); or, playback and interaction are separated, and real-time interaction cannot be carried out, etc. These separations will obviously have problems such as cost and various technical connections.

[0033] Based on this, the present application provides a method for generating digital character audio and video, which can establish a complete technical chain by combining the video playback control end (for example, the live broadcast end) and the video data processing end (for example, the cloud service end) technologies, thereby achieving ease of operation while reducing costs.

[0034] Before describing in detail the method for generating digital character audio and video provided by this application, some technical terms involved are explained as follows: Digital character: a digitized character, such as a digitized human character, a digitized virtual cartoon character, etc. In this embodiment, a digital human character is used as an example.

[0035] Digital human: A virtual character created through computer technology and graphics algorithms. It simulates human appearance, behavior, and performance in a realistic way based on computer-generated models and animation technology.

[0036] Digital human live broadcast: refers to the use of virtual characters or virtual anchors or hosts generated by digital artificial intelligence to conduct real-time live broadcast activities. These digital human anchors can disseminate relevant information and interact on the live broadcast platform through pre-recorded audio, video and animation data, combined with real-time speech synthesis and virtual reality technology. It is a process that can generate audio through copywriting, and then drive the digital population movement through audio, and broadcast live through the digital human image.

[0037] Lip drive: The process of extracting audio features and using artificial intelligence (AI) to drive the lips to open and close while speaking, and to achieve accurate matching of lips with audio features.

[0038] TTS: Text To Speech. In digital character live broadcasts, TTS technology may be used to convert text information (such as the lines and comments of the digital character host) into speech, thereby realizing the voice output of the digital character host.

[0039] CDN: Content Delivery Network. CDN is an intelligent virtual network built on the existing network. It deploys edge servers in various locations and uses the load balancing, content distribution, scheduling and other functional modules of the central platform to enable users to obtain the required content nearby.

[0040] FPS: Frames Per Second. FPS describes the number of frames displayed per second in a video. It is a standard for measuring the amount of dynamic video information. In layman's terms, it is the number of frames that can be updated per second when a video or animation is played. This indicator is directly related to the smoothness and coherence of the visual effect, and is one of the key factors in evaluating video and game experience.

[0041] GRTN: Global Realtime Transport Network. GRTN is based on the central area of ​​the public cloud and edge cloud nodes, integrating the transmission and exchange of audio and video streams in various scenarios such as Internet live broadcast and RTC (real-time audio and video communication). GRTN aims to meet the high requirements of audio and video real-time, interactivity and stability in the mobile Internet era.

[0042] SaaS: Software as a Service (SaaS) is a model that provides software application services through the Internet. Users do not need to install and maintain software locally, and can access or use application software (or applications) through the Internet.

[0043] PaaS: Platform as a Service (PaaS) provides a complete development platform, including development tools, databases, and server resources. Users can design, develop, test, and deploy applications on the PaaS platform.

[0044] IaaS: Infrastructure as a Service (IaaS) provides underlying infrastructure resources, such as servers, storage, and networks. Users can obtain computing resources, storage resources, and network resources through IaaS, and configure and manage them according to their own needs.

[0045] SaaS, PaaS and IaaS may belong to the cloud service end. In this embodiment, these three service modes may together constitute the three main service types of cloud computing.

[0046] Based on the above content, a method for generating digital character audio and video provided by the present application is described in detail.

[0047] like Figure 1 As shown, Figure 1 This is a flow chart of a method for generating digital character audio and video provided by the present application; the generation method is mainly described by taking the application to a cloud server as an example, and specifically includes: Step S101: Generate explanation data of the object to be explained according to the digital character material generation task of the requesting party; Step S102: performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; Step S103: performing character image digitization processing on the acquired character image reference data to obtain a digital character image video; Step S104: fusing the audio data with the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained; Step S105: returning the digital character driving video to the requesting party so that the requesting party can process it before distribution to generate digital character audio and video.

[0048] Before describing the above steps S101 to S105 in detail, it should be noted that the following embodiments are described in terms of the combination of the cloud service end and the playback end, but in fact, they are not limited to the combination defined in the above steps. For example, when the performance of the playback control end meets some requirements, part of the processing power of the cloud service end can be migrated to the playback control end. On the one hand, it can improve processing efficiency and reduce computing resource costs; on the other hand, it can also achieve balance in the entire playback link, avoiding waste of computing resources while also effectively utilizing local computing resources. On the other hand, data security can also be improved in some scenarios.

[0049] Combine the following Figure 2 , the above steps S101 to S105 are described in detail, Figure 2 This is a timing diagram of a method for generating digital character audio and video provided by the present application. The implementation process of step S101 to step S105 can be completed on the cloud service end, and the pre-distribution processing in the implementation process of step S105 can be completed on the playback control end, for example, when the specific application scenario is a live broadcast scenario, it can be a live broadcast control end.

[0050] Regarding step S101: generating explanation data of the object to be explained according to the digital character material generation task of the requesting party.

[0051] pass Figure 2 It can be seen that in this embodiment, the playback end is mainly responsible for issuing the digital character material generation task and the end-side processing of the digital character driving video, etc. Therefore, the purpose of step S101 is to generate explanation data.

[0052] In this embodiment, the big model can be used to implement prompt information for interacting with the big model for the digital character material generation task, thereby obtaining the explanation data output by the big model. Of course, the explanation data that already exists can also be directly used.

[0053] In this embodiment, the object to be explained may be a commodity, a tourist attraction, a teaching content, etc., that is, a certain item or thing that needs to be explained, which may be different according to different application scenarios, and the object to be explained is not specifically limited here.

[0054] The explanation data is data used to describe or characterize the object to be explained, and may be separate text data or multimodal data, including multiple types of data such as text and images.

[0055] The prompt information can be understood as a prompt of the large model, which is used to interact with the large model in order to obtain the required explanation data.

[0056] The large model can be a large language model (LLM) or a multimodal large model. A large language model is a model based on machine learning and natural language processing technology. It learns language understanding and generation capabilities by training on a large amount of text data. A multimodal large model refers to a machine learning model that can process a variety of information from different modalities (such as images, voice, text, etc.). Of course, it can also be a picture and text generation model, so there is no specific limitation on the large model.

[0057] In this embodiment, the type of large model can be selected according to the information related to the object to be explained provided in the digital character material generation task. For example, if only text information is provided, a large language model can be selected; if multiple types of information including images, text, and voice are provided, a multimodal large model can be selected. Of course, the construction of the corresponding prompt can also be realized by deploying a multimodal large model.

[0058] In this embodiment, the large model is deployed on the PaaS service of the cloud service end to realize the generation of explanation data. It is understandable that the large model can also be deployed on the playback end if the hardware performance requirements and / or data security requirements are met on the playback end.

[0059] Of course, the large model in this embodiment can also perform corresponding processing on the interactive data of the video playback feedback, which will be described in detail later.

[0060] The specific implementation process of step S101 may include: Step S101-11: Based on the identification information of the object to be explained in the task of generating digital character materials, construct prompt information for generating structured explanation data of the object to be explained, wherein the prompt information may include: attribute information of the object to be explained and description information describing the object to be explained; for example, structured information of the commodity, such as: attribute information such as price and selling point, and description information of the characteristics of the commodity itself. The identification information may be the ID of the commodity, or other information that can characterize the characteristics of the object to be explained, such as images, texts, and other information.

[0061] Step S101 - 12: using the prompt information as input information of the large model, and obtaining the explanation data output by the large model including the digital character setting.

[0062] It is understandable that the explanation data output by the large model can be generated based on one or more rounds of interaction. Therefore, prompt can also include multiple items.

[0063] In this embodiment, the prompt information may include relevant data describing the explanation object, as well as one or more data describing video style data and digital character setting data, so as to generate data of the object to be explained with character settings, for example: text data and / or image data including task settings.

[0064] The above process can be processed through a large model, thereby improving the accuracy of the explanation data corresponding to the object to be explained, and saving the manpower, material resources, and time costs invested in the early preparation of the explanation data for the object to be explained. Correspondingly, the efficiency of digital character audio and video generation can also be improved.

[0065] Regarding step S102: performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data.

[0066] In this embodiment, step S102 can be implemented by a first task model deployed on the cloud service end, the purpose of which is to convert the explanation data into audio data. The first task model mainly provides data conversion capabilities, and the specific implementation process may include: Step S102-11: inputting the explanation data as first input data into the first task model; Step S102-12: the first task model performs sentence analysis on the explanation data to extract feature data for generating the audio data; Step S102-13: Convert the explanation data into the audio data according to the phoneme sequence feature data and the tone feature data in the feature data.

[0067] In this embodiment, the first task model is a TTS model. The TTS model can first preprocess the explanation data to determine the target explanation data, that is, the target text data. The preprocessing may include: removing irrelevant characters, text normalization (such as standardized expression of dates and numbers), vocabulary analysis and segmentation, and other text data preprocessing processes.

[0068] Perform sentence analysis on the target text data, including part-of-speech tagging, syntactic analysis, semantic understanding, etc., to understand the intent and context of the text and to provide basic processing for generating natural and fluent speech.

[0069] ‌Convert the results of language analysis into actual speech output, including the construction of a sound database, the selection and connection of sound units, and the generation of the final speech through a synthesis algorithm. ‌

[0070] The TTS algorithm may include: a splicing method and a parameter method.

[0071] The splicing method is to select the required basic units from a large number of pre-recorded speech and splice them. The basic unit can be a syllable, a phoneme, etc. In order to pursue the coherence of the synthesized speech, a disyllabic unit (a unit consisting of the end of the current pronunciation unit and the beginning of the next pronunciation unit) is often used as a unit.

[0072] The parameter method is to generate speech parameters (including fundamental frequency, formant frequency, etc.) at every moment according to a statistical model, and then convert these parameters into waveforms.

[0073] Since the TTS algorithm belongs to the prior art, it will not be introduced in detail here, and only a brief description is given here.

[0074] It should be noted that the training of the TTS model in this embodiment is not limited to the language type, and can be Chinese, English, or a mixture of Chinese and English. Of course, it is not limited to the above language type examples, so as to expand the scope of digital character audio and video frame data generation.

[0075] In this embodiment, the first task model converts the text data of the explanation data into audio data to generate audio data. The audio data can be sent in the PCM audio file format. PCM audio file is a pulse code modulation file, which is a multimedia audio file with streaming media characteristics, that is, it can be read and played at the same time. Of course, in this embodiment, only the PCM audio file format is used as an example for description, and the actual audio data can also include other formats, such as: WAV file, MP3 file, etc. Therefore, the file format in which the converted audio data is stored can be determined according to specific needs, and is not limited to the above examples.

[0076] It is understandable that when the explanation data includes digital character settings or style data, etc., the audio data generated in this embodiment can be explanation audio data with a specified timbre (such as a PCM file).

[0077] In order to increase the generation speed of audio data, the first task model can be optimized and model distilled, for example, the model size and stability can be optimized to increase the model generation speed. The model distillation can significantly reduce the scale of the model while retaining most of the performance, thereby reducing the consumption of computing resources and providing support for more practical application scenarios. In this embodiment, the model distillation is not only applicable to the first task model, but also to large models, second task models, third task models, etc., and is adjusted in real time according to actual data processing.

[0078] In TTS application scenarios, the performance of TTS can usually be measured by RTF. The RTF is the Real-Time Factor, which is used to evaluate the efficiency of the model in the speech generation process. RTF represents the ratio of the processing time required to generate speech to the speech duration. Its formula is: Among them, the The time taken to generate speech, is the duration of the generated speech.

[0079] When RTF<1, it indicates real-time or faster generation, that is, the time required to generate speech is less than the duration of the speech; when RTF>1, it indicates non-real-time generation, that is, the time required to generate speech is greater than the duration of the speech; that is, the lower the RTF, the faster the TTS system generates speech.

[0080] Therefore, in this embodiment, the first task model can be adjusted in real time through the value of RTF to optimize the performance of the model, thereby meeting the requirements of the TTS algorithm in different scenarios, thereby fully utilizing computing resources and avoiding the waste of computing resources in some scenarios with low performance requirements.

[0081] It can be understood that the first task model in this embodiment is a TTS model, the core of which is to convert text data into audio data (or voice data). When the explanation data includes not only text data, but also images or other data, or is only other types of data other than text data, it can be converted into text data by first identifying other types of data and then converting them. Therefore, there is no specific limitation on the type of explanation data in this embodiment. Of course, the first task model can also be a conversion model for other data types according to conversion requirements, and is not limited to the TTS model. In this embodiment, it is just combined with scenarios such as live broadcasts, where it is necessary to convert data into example descriptions of commodities or other items that need to be introduced through TTS.

[0082] In this embodiment, by optimizing and distilling the first task model, the model stability can be improved, and the model size can be adjusted in real time based on different application scenarios, which can reduce the data collection cost. In addition, the first task model can use historical playback audio data as the training sample data of the model during the training phase, which can also effectively improve the authenticity of audio data generation and improve the restoration of timbre when there are timbre requirements.

[0083] Regarding step S103: performing character image digitization processing on the acquired character image reference data to obtain a digital character image video.

[0084] In this embodiment, the step S103 may determine the digital character image video by the second task model, and the second task model may be a model for implementing digital processing. The specific implementation process may include: Step S103-11: Generate the character image reference data according to the digital character material generation task or network search; Step S103-12: digitally process the character image in the character image reference data to obtain the digital character image video.

[0085] The character image reference data in step S103 may be data carried in the digital character material generation task, or may be data obtained through a network or local search, i.e., reference data for generating a digital character. In this embodiment, the character image reference data carried in the digital character material generation task is used as an example for explanation, such as the image reference data of a narrator or the image reference data of a host.

[0086] It should be noted that, whether the character image reference data is the data carried in the digital character material generation task, or the data obtained through network or local search, it is the data obtained after the role authorization or role delegation authorization. Of course, in specific application scenarios, such as: in the live broadcast scenario, when the live broadcast end sends the digital character data generation task and carries the character image reference data, it is the default authorization. In this embodiment, the character is described by taking the human character as an example, that is, the digital human.

[0087] The character image reference data may include a character image or a character video, which may be used as reference data (a character reference image or a character reference video, and including a specified area, such as facial data). In this embodiment, a digital human refers to a virtual character image created by computer technology (such as 3D modeling, motion capture, artificial intelligence, etc.) in the fields of technology and science. These digital humans may be completely fictional or recreated based on real people. The character image data may include the above two situations, namely: a virtual character image (including a cartoon image, etc.) and an image based on a real character. The character image data in this embodiment may be a character video recorded in advance, a character video collected in real time, a character image provided by a live broadcast terminal, etc. There is no specific limitation on the character image data. The above is only an example for easy understanding.

[0088] The second task model may be an image generation model, for example, image generation is performed through a diffusion model (StableDiffusion). In this embodiment, the digital human image may also be understood as an AI character image. The IP-Adapter image prompt adapter may be used to convert the narrator image data into the digital human image data based on the diffusion model. The IP-Adapter is an image prompt adapter that introduces text prompts into the image diffusion model. The specific implementation process may include: IP-Adapter first uses a pre-trained image encoder (such as CLIP image encoder) to extract image features from the image of the lecturer image data. The image features are features that represent relevant information such as the content, style, and details of the image. Then, a decoupled cross-attention mechanism is used to separate the cross-attention layers of the text features in the lecturer image data and the image features in the lecturer image data, and the text features and the image features are processed separately. Then, the text features and the image features are combined, and a digital human image video is generated through a diffusion model. IP-Adapter can be used together with other control modules such as ControlNet. Through different preprocessors and models, as well as by adjusting weight values, the generation effect of the digital human image can be further controlled, and more diverse style conversion, posture adjustment, detail addition and other functions can be achieved, making the generated digital human image video more realistic and personalized.

[0089] In this embodiment, in order to improve the authenticity of the digital human image video, StyleGAN can also be superimposed on the IP-Adapter to generate highly realistic digital human facial images. StyleGAN is a deep learning model based on the generative adversarial network (GAN). StyleGAN relies on the interaction between the generator and the discriminator. The generator generates the digital human facial image according to the input style vector, and the discriminator evaluates the authenticity of the digital human facial image.

[0090] In this embodiment, the StyleGAN model is superimposed based on IP-Adapter, and the basic data of the digital human output by the text-to-image extension model based on IP-Adapter, such as the image data of the digital human head and face, can be used as the input data of the StyleGAN model to obtain the digital human image data output by the StyleGAN model, and then generate the digital human image video according to the digital human image data. The text-to-image extension model based on IP-Adapter and the StyleGAN model can be regarded as the second task model.

[0091] In order to improve the diversity of digital human decoration styles and meet the needs of different application scenarios for digital human images, in this embodiment, the second task model may also include: IDM-VTON model is a virtual try-on technology based on a diffusion model, which uses a visual encoder and a parallel U-Net network (GarmentNet) to extract high-level semantic information and detail features of clothing respectively, and generates more natural and realistic virtual fitting images through multi-semantic fusion and text prompts. In this embodiment, the digital human image data output by the IP-Adapter superimposed StyleGAN model can be used as the input data of the IDM-VTON model, or the digital human image data output by the StyleGAN model can be used as the input data of the IDM-VTON model to generate digital human image data with diverse decoration styles, and digital human image videos are generated based on this.

[0092] To improve the accuracy and efficiency of computer vision, the second task model may also include the Sapiens model, i.e., a human vision model. This model is based on deep learning technology and combines a large amount of image data and advanced algorithms to simulate human visual perception. In this embodiment, the accuracy of digital human image generation can be improved in terms of posture estimation, body parts, etc. during the digital human image video generation process.

[0093] Therefore, the specific implementation process of step S103-12 in this embodiment may include: Step S103-121: the second task model generates digital character image data including the head features according to the extracted head features of the character image data; Step S103-122: adjusting the digital character facial area and / or the digital character decoration style in the digital character image data to obtain the digital character image video.

[0094] In this embodiment, in the process of generating a digital character image video using the character image data as reference data, a digital character image generation link of the head, face, decoration style (which can also be understood as clothing style, background decoration, etc.) is established, which can effectively improve the generation efficiency and accuracy of the digital character image video. Since the digital human image collection in the prior art requires models, venues, recording equipment, etc., all of which have high collection costs, this embodiment can reduce the training collection costs of each model in the second task model and the collection costs involved in the generation link based on the above-mentioned generation link.

[0095] It should be noted that, because the models involved in the second task model are all existing models and the model training has been completed in the corresponding application scenario, the task processing can be performed directly. The above is only a summary description of the above models.

[0096] Regarding step S104: the audio data and the digital character image video are fused to obtain a digital character driving video in which the digital character explains the object to be explained.

[0097] The purpose of step S104 is to drive the digital character in the digital character image video to explain according to the explanation data, that is, to make the digital character's lip movements match and synchronize with the audio data. In this embodiment, the driving can be implemented through the third task model, and the specific implementation process may include: Step S104-1: the third task model extracts lip features in the digital character image video and audio features in the audio data respectively; Step S104-2: establishing a matching relationship between the lip feature and the audio feature; Step S104-3: Based on the matching relationship, a digital character driving video is obtained in which the lip region in the digital character image video moves according to the audio data.

[0098] In this embodiment, it is necessary to match the extracted lip features in the image data of the speaker with the audio features in the audio data, so as to drive the lip features to move and change according to the audio features, and generate the digital human audio and video driving video data. In other words, the changes in the shape of the speaker's lips need to be operated according to the audio information. Therefore, there is a matching relationship between the lip shape features and the audio features.

[0099] The lip features are driven to move according to the audio features, and a Hallo2 algorithm or a wav2lip algorithm may be used.

[0100] The Hallo2 algorithm is an audio-driven portrait image animation technology, the core of which is to convert audio features into realistic facial animation. The specific implementation process may include: Preprocess the input audio data to extract key audio features, such as pitch, volume, speaking speed, etc.

[0101] According to the extracted audio features, the corresponding facial animation parameters are generated. These parameters include the shape of the lips, the degree of opening and closing, the position of the tongue, etc., which are used to describe the changes in the mouth shape synchronized with the audio content, and of course, facial expressions.

[0102] The Generative Adversarial Network (GAN) structure is adopted, which consists of two parts: the generator and the discriminator. The generator is responsible for generating realistic facial animation images based on audio features, while the discriminator is used to evaluate the authenticity of the generated images and the synchronization between the generated images and the audio content.

[0103] The Hallo2 algorithm also includes time alignment, for example by analyzing the timing relationship between audio features and facial animation parameters to ensure that the generated facial animations are temporally consistent with the audio content.

[0104] The Hallo2 algorithm also includes detailed processing of the generated facial animation, such as adjusting the texture and gloss of the lips to improve its realism and visual effects.

[0105] The wav2lip algorithm is a lip shape driving algorithm based on deep learning. The specific implementation process may include: The input audio is preprocessed to extract audio features. These speech features usually include Mel-frequency cepstral coefficients (MFCCs), spectrograms, etc., which are used to describe the acoustic characteristics of speech.

[0106] Extract lip shape features from the speaker's reference video (which may only contain the face part). This can be achieved through deep learning models such as convolutional neural networks (CNNs) to capture features such as the shape and color of the lip area.

[0107] Through the pre-trained phoneme-to-lip shape mapping model, the correspondence between audio features and lip shape features is learned, so that lip shape animations that match the audio features in the audio data are generated.

[0108] The wav2lip algorithm can also perform post-processing on the generated lip animation, such as adjusting the smoothness of the lip animation, removing noise, etc., so as to improve its realism and visual effects.

[0109] The above two algorithm examples are only used to describe lip drive. And the above two algorithms also belong to the existing technology of lip drive. The Hallo2 algorithm can improve the clarity of digital human audio and video data, and pay more attention to the overall generation and detail processing of facial animation, while the wav2lip algorithm focuses on the generation of lip animation and the synchronization of voice content. Therefore, different algorithms can be used in combination with the specific application scenarios of lip drive. For example: This embodiment mainly involves the live video scene, so it focuses more on the overall generation of facial animation, so the Hallo2 algorithm can be used.

[0110] It can be understood that the lip driving in this embodiment is not limited to the examples given above, and any method that can achieve synchronization between lips and audio can be used as the implementation of lip driving in this application.

[0111] Regarding step S105: returning the digital character driving video to the requesting party so that the requesting party can process it before distribution to generate digital character audio and video.

[0112] The purpose of step S105 is to return the digital character driving video to the requesting party.

[0113] The requesting party processes the digital character driving video before distribution, and distributes it after processing. In this embodiment, the rendering of the digital character driving video can be performed at the playback control end.

[0114] In this embodiment, the processing of the digital character driving video before distribution may include: performing audio and video preprocessing, video rendering, element arrangement, etc. on the digital character driving video, thereby generating digital character audio and video frame data, encoding the digital character audio and video frame data, and distributing it.

[0115] In this embodiment, the distribution can provide corresponding distribution output capabilities based on different application scenarios. For example, in a live broadcast scenario, a live streaming output method can be adopted, and in a video playback scenario, a video synthesis output method can be adopted.

[0116] It is understandable that when there are multiple output scenario requirements, multiple output methods can be used for output.

[0117] In different output scenarios, interactive data can also be obtained for the distribution of the output digital character audio and video. For example, in the live broadcast scenario, user feedback interactive data on the digital character audio and video, such as barrage data, can be obtained. In the video synthesis output scenario, there can also be user feedback interactive data on the digital character audio and video. One is real-time and the other is non-real-time. For real-time, it can also include: Acquiring interaction data based on the distribution of the digital character's audio and video; Input the interaction data as fourth input data into the first task model to obtain interaction audio data corresponding to the interaction data; The interactive audio data and the digital character image video are used as fifth input data and input into the third task model to obtain a digital character interactive driving video in which the digital character explains according to the interactive audio data; Based on the pre-distribution processing of the digital character interaction driving video, the generated digital character interaction audio and video is distributed.

[0118] The interactive data may also be filtered and screened to ensure the accuracy and standardization of the interactive audio data generated.

[0119] In this embodiment, the interactive data can be generated based on the barrage data generated by the audio and video of the digital character, that is, the interactive data can be a reply interactive data for the barrage data. Similarly, a large model can be used to generate it according to the barrage data, or it can be related data found in a pre-established reply database. The interactive data can be obtained in different ways according to different scenarios. For example, when the explanation object is a relatively fixed information object such as a commodity or a tourist attraction, the reply database method can be used, and of course the large model acquisition method can also be used.

[0120] The above is a description of an embodiment of a method for generating digital character audio and video provided by the present application. On the one hand, the method realizes the playback of digital character audio and video by combining the corresponding service capabilities of the cloud with the capabilities of the playback control terminal, wherein the cloud mainly performs tasks such as AI reasoning and calculation in the process of digital character-driven video generation, and the playback control terminal mainly performs tasks such as rendering, editing, and distribution of digital character audio and video. The combination of the two can reduce the cloud deployment cost, and the cloud-related processing tasks can be combined with different application scenario requirements to migrate some tasks to the playback control terminal, which can further save computing resources and avoid waste of computing resources on the playback control terminal, so that the two can achieve task processing balance in the process of combining the two.

[0121] On the other hand, the method can build a complete processing chain from the generation of explanation data to audio data conversion, and the generation of digital character image video based on character image data, and then generate digital character driving video based on digital character image video and audio data, and then process the digital character driving video before distribution, which effectively reduces the generation and playback costs of digital character audio and video. At the same time, this process can also reduce the participation cost of the playback control end, thereby improving the coverage of digital character audio and video playback.

[0122] The above is a detailed description of an embodiment of a method for generating digital character audio and video provided by the present application. Corresponding to the embodiment of a method for generating digital character audio and video provided above, the present application also discloses an embodiment of a device for playing digital character audio and video. Figure 3 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0123] like Figure 3 As shown, Figure 3 It is a structural schematic diagram of a digital character audio and video playback device provided by the present application, and the device may include: a first acquisition unit 301, a second acquisition unit 302, a third acquisition unit 303, a fourth acquisition unit 304 and a return unit 305.

[0124] The first acquisition unit 301 is used to generate explanation data of the object to be explained according to the digital character material generation task of the requesting party; The second acquisition unit 302 is used to perform data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; The third acquisition unit 303 is used to perform character image digitalization processing on the acquired character image reference data to obtain a digital character image video; The fourth acquisition unit 304 is used to merge the audio data and the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained. The returning unit 305 is used to return the digital character driving video to the requesting party, so that the requesting party can process it before distribution to generate digital character audio and video.

[0125] In this embodiment, the specific implementation process of the first acquisition unit 301 may include: A construction subunit is used to construct prompt information for generating structured explanation data of the object to be explained according to the identification information of the object to be explained in the task of generating the digital character material acquired, wherein the prompt information may include: attribute information of the object to be explained and description information describing the object to be explained; The acquisition subunit is used to use the prompt information as input information of the large model to acquire the explanation data output by the large model including the setting of the digital character.

[0126] The specific implementation process of the second acquisition unit 302 may include: An input subunit, used for inputting the explanation data as first input data into the first task model; An extraction subunit, configured to use the first task model to perform sentence analysis on the explanation data and extract feature data for generating the audio data; The conversion subunit is used to convert the explanation data into the audio data according to the phoneme sequence feature data and the tone feature data in the feature data.

[0127] The specific implementation process of the third acquisition unit 303 may include: An acquisition subunit, used to generate tasks according to the digital character material or to acquire the character image reference data through network search; The obtaining subunit is used to input the character image reference data as second input data into the second task model for digital processing to obtain the digital character image video.

[0128] The obtaining subunit may specifically include: An image data generating subunit, used for the second task model to generate digital character image data including the head features according to the extracted head features of the character image data; The image video obtaining subunit is used to obtain the digital character image video by adjusting the digital character facial area and / or the digital character decoration style in the digital character image data.

[0129] The specific implementation process of the fourth acquisition unit 304 may include: An extraction subunit, used for the third task model to respectively extract lip features in the digital character image video and audio features in the audio data; Establishing a subunit, used to establish a matching relationship between the lip feature and the audio feature; The obtaining subunit is used to obtain, according to the matching relationship, a digital character driving video in which the lip area in the digital character image video moves according to the audio data.

[0130] This embodiment may further include: An interactive data acquisition unit, used to acquire interactive data according to the distribution of the digital character audio and video; An interactive audio obtaining unit, used to input the interactive data as fourth input data into the first task model for data conversion processing to obtain interactive audio data corresponding to the interactive data; An interactive driving video acquisition unit, used to input the interactive audio data and the digital character image video as fifth input data into the third task model for fusion processing, and obtain a digital character interactive driving video in which the digital character explains according to the interactive audio data; The interactive return unit is used to return the digital character interactive driving video to the requesting party so that the requesting party can process it before distribution to generate digital character interactive audio and video.

[0131] The above is a description of an embodiment of a digital character audio and video playback device provided by the present application. For the specific content of the device embodiment, reference can be made to the relevant content of steps S101 to S105 in the above method embodiment, which will not be described in detail here.

[0132] Based on the above, the present application also provides a digital character live interactive method, such as Figure 4 As shown, Figure 4 This is a flow chart of a digital character live interactive method provided by the present application, the method comprising: Step S401: in response to a request for generating audio and video playback in an audio and video playback platform, returning a digital character driving video obtained according to the above-mentioned method for generating digital character audio and video to the audio and video playback platform; Step S402: performing pre-distribution processing on the received digital character driving video to generate digital character audio and video; Step S403: in response to a play triggering operation on the audio and video play platform, distributing the digital character audio and video; Step S404: in response to the distributed digital character audio and video, returning the digital character interactive driving video obtained according to the digital character audio and video generation method to the audio and video playback platform; Step S405: distributing the generated digital character interaction audio and video according to the pre-distribution processing of the received digital character interaction driving video.

[0133] The specific implementation process of this interaction method can be referred to Figure 5 As shown, Figure 5 This is a timing diagram of a digital character live interactive method provided by the present application.

[0134] In this embodiment, the audio and video playback platform can be a live playback control terminal or a video playback control terminal, for example: based on the terminal side of the application service platform with video playback control, it can be understood as the anchor terminal in the live application scenario.

[0135] The audio and video playback platform may have a trigger operation for implementing a live broadcast request, i.e., a live broadcast trigger request. The operation of the live broadcast trigger request may be implemented through a provided trigger control or through voice triggering, etc., without limitation.

[0136] In this embodiment, the digital character audio and video can be pre-generated, and then distributed or pushed in real time when the live broadcast is triggered, so as to complete one-stop live broadcast. Therefore, when a generation request for the digital character audio and video is generated in step S401, the image data of the digital character anchor to be generated will be sent to the cloud service end in the generation request. The cloud service end can generate the digital character driving video according to the relevant steps of steps S101 to S104 in the above-mentioned digital character audio and video generation method embodiment, and return the digital character driving video to the live broadcast control end.

[0137] The play request in step S402 may be implemented through a live broadcast trigger control, which may be a manual operation by the user or a voice operation, or may be automatically triggered according to the set live broadcast start time. Therefore, the triggering of the play request in this embodiment may exist in a variety of ways, and may be set in combination with specific application scenarios and needs. Based on the live broadcast start request (play request), the digital character audio and video may be streamed in real time, i.e. distributed. In this embodiment, a live broadcast scenario is used as an example for description, such as a product purchase service, a tourist attraction introduction service, an educational service, etc. In fact, it may also be a description process of resource information in scenarios such as short videos and long videos.

[0138] In this embodiment, a real-time communication development toolkit (ARTC SDK) can be used for streaming to achieve low-latency streaming and pulling. The digital character audio and video is sent to the RTS NetSDK through the streaming ARTC SDK, and the RTS NetSDK transmits the digital character audio and video based on the global real-time transmission network GRTN (Global Real Time Network). GRTN is a low-latency, fully distributed, sinking communication-level streaming media transmission network deployed on edge cloud nodes to support video conferencing, live video, real-time audio and video and other services. ARTC SDK applies WebRTC (Web Real-Time Communication) real-time communication technology, such as bandwidth estimation (BWE), jitter buffer, network equalizer (NetEQ), forward error correction (FEC), negative acknowledgment (NACK) and other QoS (Quality of Service) mechanisms, which can optimize transmission delays and improve real-time performance, ensuring low-latency streaming or distribution of digital character audio and video.

[0139] It is understandable that different streaming methods can be selected according to the needs of specific application scenarios. In this embodiment, live video is mainly used as the application scenario, so the real-time communication development toolkit is used to achieve low-latency streaming. In scenarios where the real-time requirements for video streaming are not high, other methods can be used for streaming or distribution.

[0140] Step S403 is mainly for the interaction between the pushed digital character audio and video and the video playback client, for example, the barrage data generated by the client is the interaction data. Therefore, the digital character can be interactively driven by the digital character based on the interaction data through the process of processing the interaction data in the above-mentioned digital character audio and video generation method embodiment.

[0141] The step S404 is mainly to drive the video to perform corresponding pre-distribution processing according to the digital character interaction, such as rendering, element arrangement, scene switching, editing code, etc., so as to push the processed digital character interaction audio and video to the client (i.e., the playback output end, the client watching the live broadcast or video).

[0142] For the specific implementation process of the above steps S401 to S404, reference may be made to the relevant description of the above-mentioned embodiment of the method for generating audio and video of a digital character, which will not be described in detail here.

[0143] The above is a description of an embodiment of a digital character live interactive method provided by the present application. In the interactive method, the live control end only needs to make a request for the generation of digital character audio and video to obtain the digital character audio and video generated by the complete link, and then based on the start of broadcast triggering, a one-stop live broadcast process is realized. This process does not involve the separate processing of one or several technical links, so that the process from the generation of digital character audio and video to the start of playback, to the playback push stream, and real-time interaction can be realized through a complete link, thereby avoiding the high cost of broadcasting in the digital character live broadcast scenario, and the complex operation caused by the connection of various technical links, effectively reducing the cost and threshold of broadcasting, and improving the user experience of the live control end.

[0144] It should be noted that the above description is only based on the live broadcast scenario. For some short video or long video playback scenarios, there are also corresponding problems that can be overcome by the above method. The triggering of one-stop playback can achieve the distribution of long videos or short videos. Therefore, the above interaction method is not limited to the live broadcast scenario.

[0145] The above is a specific description of an embodiment of a digital character live interactive method provided by the present application. Corresponding to the aforementioned embodiment of a digital character live interactive method, the present application also discloses an embodiment of a digital character live interactive device. Figure 6 Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0146] like Figure 6 As shown, Figure 6 This is a structural diagram of a digital character live interactive device provided by the present application, the device comprising: The first returning unit 601 is used for returning the digital character driving video obtained in the method for generating the digital character audio and video to the audio and video playing platform in response to the generation request for the audio and video playing in the audio and video playing platform; The generating unit 602 is used to perform pre-distribution processing on the received digital character driving video to generate digital character audio and video; A first distribution unit 603 is used to distribute the digital character driving video in response to a play triggering operation on the audio and video playback platform; A second returning unit 604 is used for returning the digital character interactive driving video obtained according to the method for generating the digital character audio and video to the audio and video playback platform in response to the distributed digital character audio and video; The second distribution unit 605 is used to distribute the generated digital character interaction audio and video according to the pre-distribution processing of the received digital character interaction driving video.

[0147] For the specific implementation process of the above-mentioned interactive device, please refer to the description in the above-mentioned interactive method, as well as the relevant description in the above-mentioned method for generating audio and video of digital characters, which will not be described in detail here.

[0148] Based on the above content, the present application also provides a digital character audio and video playback system, such as Figure 7 As shown, Figure 7 It is a structural diagram of a digital character audio and video playback system provided in the present application, which includes: a playback control terminal 701, a cloud service terminal 702 and a playback output terminal 703.

[0149] The playback control terminal 701 is used to send a request for generating digital character audio and video to the cloud service terminal; and to perform pre-distribution processing based on the digital character driving video returned by the cloud service terminal, and distribute the generated digital character audio and video to the playback output terminal 703 for output; The cloud service end 702 is used to generate a task of generating digital character materials according to the generation request, generate explanation data of the object to be explained, and perform data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; The cloud service end 702 is also used to perform character image digitization processing on the acquired character image reference data to obtain a digital character image video; and to fuse the audio data and the digital character image video to obtain a digital character driving video in which a digital character explains the object to be explained, and return the video to the playback control end; The playback output terminal 703 is used to output the audio and video of the digital character.

[0150] Combine the following Figure 8 , the specific implementation process of the playback control end 701, the cloud service end 702 and the playback output end 703 is described.

[0151] Figure 8 It is a structural diagram of each end embodiment of a digital character audio and video playback system provided by the present application. The cloud service end 702 may include: an infrastructure layer, a large model layer, a software tool layer, an AI engineering service layer, and a task service layer.

[0152] The infrastructure layer provides support for the cloud service end to realize AI computing power. For example, the operating system image can be deployed, and the basic dependencies such as CUDA Tooolki, Pytorch, Conda, etc. can be deployed inside the image to realize TTS algorithm service and lip-shaped driver algorithm service. Among them, the CUDA is an open source parallel computing platform and programming model that allows developers to use GPU for high-performance computing. CUDA Toolkit is a development toolkit that contains compilers, libraries, tools and documents required for CUDA programming. For the TTS algorithm, it requires a lot of computing resources to train and reason about the model. CUDA Toolkit provides the ability to use GPU for accelerated computing, which can significantly improve the execution performance of TTS algorithm and lip-shaped driver algorithm. For example, when training neural network models such as WaveNet, Tacotron or FastSpeech, using CUDA can greatly shorten the training time. The Pytorch is an open source deep learning framework that provides powerful GPU acceleration functions and supports dynamic calculation graphs, which can be more flexible for model development and debugging. Pytorch is one of the commonly used frameworks for building and training TTS models. In Pytorch, various neural network models (such as WaveNet, Tacotron, FastSpeech, etc.) can be used to implement TTS functions. The Conda is an open source package management system and environment manager, mainly used for scientific computing package management in the Python language. It allows users to easily create and manage multiple environments, thereby avoiding dependency conflicts between different projects. In the development and deployment process of TTS and lip-shaped drivers, Conda can help users manage different versions of Python and dependent libraries to ensure that TTS and lip-shaped driver projects have correct dependencies, thereby helping to simplify the setup and management process of TTS and lip-shaped driver algorithm projects and improve development efficiency. In addition, Conda can also be used to install dependencies such as Pytorch and CUDA Toolkit to provide necessary support for the development and operation of TTS and lip-shaped driver algorithms. In this embodiment, the infrastructure layer can provide computing power services for the upper layer to ensure the stability, real-time and accuracy of computing resources in the process of generating digital character audio and video.

[0153] The large model layer deploys interactive large models such as LLM or multimodal, which can support the production of explanation data, human interaction, and auxiliary information of the explanation object (for example, welcome, attention guidance, promotion, and different explanation objects may include different auxiliary information). The large model layer receives the prompt input from the upper module and outputs multimodal results (such as pictures, text, etc.). In this embodiment, the type of large model layer deployment can be adjusted according to the actual application scenario.

[0154] It can be understood that, in this embodiment, the corresponding large model in the large model layer is deployed on the cloud service end, and can actually be deployed on the playback control end according to the performance of the playback control end. This is just an example.

[0155] The software tool layer (LiveCopilot-SDK) is an AI audio and video cross-platform software development kit that can be deployed on both the cloud service end and the playback control end. In this embodiment, the software tool layer includes: infrastructure module, audio and video processing and rendering module, media output module, AI operator module, instruction parsing module, etc. Among them, the infrastructure module: includes services such as logs, UT embedding points, OSS storage, and performance monitoring. The audio and video processing and rendering module: includes plug-in packaging, audio rendering, picture rendering, media editing and other services. The media output module: includes services such as audio and video encoding and decoding, low-latency streaming, and video production. The AI ​​operator module: supports the rapid deployment of AI operators such as lip drive and TTS, and realizes operator scheduling at the same time, and outputs audio, video, graphic results and other services according to input tasks. The instruction parsing module: encapsulates a unified interface, and supports parsing the playback control end and cloud service end instructions (such as: cloud API and end SDK, etc.), receives and issues instruction tasks, and produces multimodal content such as video, live broadcast, picture, and text.

[0156] The AI ​​engineering service layer includes three modules: asset library, intelligent question and answer, and machine management / scheduling. The asset library module includes the uploading and storage of digital human image data, video data, model data, etc. The intelligent question and answer module includes the maintenance and query of the question library and character setting documents. The machine management / scheduling module is responsible for the management and scheduling of the service cluster in the cloud service end, as well as the input data of the task service layer, and converts it into structured tasks and sends it to the LiveCopilot-SDK layer.

[0157] The task service layer includes services related to broadcast room management (for example, services related to live broadcast room management), including services related to live broadcast room creation, broadcast start, pop-up explanation object cards, barrage replies, etc., that is, providing one-stop broadcast services for digital people.

[0158] like Figure 8 As shown, the playback control terminal 701 in this embodiment may include: a terminal-side control module and a terminal-side service module. The terminal-side control module includes: service controls such as role cloning, explanation object data generation, one-click broadcasting, intelligent question and answer, and human interaction, and also includes responding to playback control operations and transparently transmitting relevant instructions to the task server. The terminal-side service module includes explanation object card rendering, material preloading, digital human screen rendering (beautification, makeup, filters, background processing), scene and screen arrangement, digital human video fusion, encoding and streaming services.

[0159] like Figure 8 As shown, the playback output terminal 703 in this embodiment includes an explanation object pop-up module, that is, when the digital human explains a specified explanation object, the card information of the corresponding explanation object will be automatically switched and popped up at the playback output terminal. The barrage interaction module, when the user at the playback output terminal inputs the barrage, can realize real-time interaction between the digital human and the user. Of course, it can also include: a service module, which enables data interaction between the user and the provider of the explanation object, such as: customer service consultation, etc.

[0160] Based on the above, it can be known that in this embodiment, the playback output end is used to send the interactive data generated based on the audio and video of the digital character to the cloud service end; The cloud service end uses the constructed prompt information of the interaction data as the fourth input data of the large model, and obtains the interaction response data output by the large model for the interaction data; and uses the interaction response data as the fifth input data of the first task model, and obtains the interaction audio data output by the first task model after the interaction response data is converted and processed; and uses the interaction audio data and the digital character image video as the sixth input data of the third task model, and obtains the digital character interaction driving video output by the third task model, which drives the digital character to explain according to the interaction response data; The playback control end is used to render the digital character interaction driving video sent by the cloud service end, and send the generated digital character interaction audio and video to the playback output end for output.

[0161] The playback control terminal generates the playback request by triggering the audio and video playback in the audio and video playback platform; the playback request carries the digital character material generation task.

[0162] In the above-mentioned digital character audio and video playback system embodiment, a set of overall playback links can be established, thereby achieving one-stop playback, which is applicable to both live broadcast scenes and video synthesis scenes.

[0163] The above is a description of an embodiment of a digital character audio and video playback system provided in the present application. The specific implementation process of the system embodiment can be combined with the above-mentioned digital character audio and video generation method, as well as the digital character live broadcast interaction method and other related contents, which will not be repeated here.

[0164] Based on the above, the present application also provides a computer storage medium, including a computer program. When the computer program is run on an electronic device, the electronic device executes the steps in the method for generating digital character audio and video; or, executes the steps in the digital character live interactive method; or, executes the steps executed by the execution side in the digital character audio and video playback system.

[0165] Based on the above, the present application also provides an electronic device, such as Fig. 9 As shown, Fig. 9 9 is a schematic diagram of the structure of an electronic device provided by the present application. In this embodiment, the electronic device includes: a processor 901 and a memory 902; The memory 902 is used to store a program for processing data generated by an electronic device. When the program is read and executed by the processor 901, it executes the steps in the method for generating digital character audio and video; or, executes the steps in the digital character live interactive method; or, executes the steps executed by the execution side in the digital character audio and video playback system.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0167] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0168] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0169] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0170] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0171] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A method for generating digital character audio and video, characterized in that: Applied to cloud servers, including: Generate the task according to the digital character material of the requesting party, and generate the explanation data of the object to be explained; Performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; Performing character image digitization processing on the acquired character image reference data to obtain a digital character image video; The audio data and the digital character image video are fused to obtain a digital character driving video in which the digital character explains the object to be explained; The digital character driving video is returned to the requesting party so that the requesting party can process it before distribution to generate digital character audio and video.

2. The method for generating digital character audio and video according to claim 1, characterized in that: The step of generating the explanation data of the object to be explained according to the digital character material of the requesting party includes: Constructing prompt information for generating structured explanation data of the object to be explained according to the identification information of the object to be explained in the digital character material generation task, wherein the prompt information may include: attribute information of the object to be explained and description information describing the object to be explained; The prompt information is used as input information of the large model to obtain the explanation data including the digital character setting generated by the large model.

3. The method for generating digital character audio and video according to claim 1, characterized in that: The step of performing data conversion processing on the explanation data to obtain audio data corresponding to the explanation data includes: Performing sentence analysis on the explanation data to extract feature data used to generate the audio data; The explanation data is converted into the audio data according to the phoneme sequence feature data and the tone feature data in the feature data.

4. The method for generating digital character audio and video according to claim 1, characterized in that: The step of performing character image digitization processing on the acquired character image reference data to obtain a digital character image video includes: Acquire the character image reference data according to the digital character material generation task or through network search; The character image in the character image reference data is digitally processed to obtain the digital character image video.

5. The method for generating digital character audio and video according to claim 4, characterized in that: The step of digitally processing the character image in the character image reference data to obtain the digital character image video comprises: Generating digital character image data including the head features according to the extracted head features of the character image reference data; The digital character image video is obtained by adjusting the digital character facial area and / or the digital character decoration style in the digital character image data.

6. The method for generating digital character audio and video according to claim 1, characterized in that: The step of fusing the audio data with the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained, comprises: Extracting lip features in the digital character image video and audio features in the audio data; Establishing a matching relationship between the lip feature and the audio feature; According to the matching relationship, a digital character driving video is obtained in which the lip area of ​​the digital character undergoes movement changes according to the audio data.

7. The method for generating digital character audio and video according to claim 1, characterized in that: Also includes: Acquiring interaction data based on the distribution of the digital character's audio and video; According to the data conversion processing of the interactive data, interactive audio data corresponding to the interactive data is obtained; The interactive audio data and the digital character image video are fused to obtain a digital character interactive driving video in which the digital character explains according to the interactive audio data; The digital character interaction driving video is returned to the requesting party so that the requesting party can process it before distribution to generate digital character interaction audio and video.

8. A digital character live interactive method, characterized in that: include: In response to a request for generating audio and video playback in an audio and video playback platform, returning a digital character driving video obtained based on the method for generating digital character audio and video according to any one of claims 1 to 7 to the audio and video playback platform; Performing pre-distribution processing on the received digital character driving video to generate digital character audio and video; In response to a play trigger operation on the audio and video playback platform, distributing the digital character audio and video; In response to the distributed digital character audio and video, returning the digital character interactive driving video obtained according to the method for generating digital character audio and video according to claim 7 to the audio and video playback platform; The generated digital character interaction audio and video is distributed according to the pre-distribution processing of the received digital character interaction driving video.

9. A digital character audio and video playback system, characterized in that: include: Playback control terminal, playback output terminal and cloud service terminal; The playback control end is used to send a request for generating digital character audio and video to the cloud service end; The cloud service end is used to generate a task of the digital character material obtained according to the generation request, generate explanation data of the object to be explained, and perform data conversion processing on the explanation data to obtain audio data corresponding to the explanation data; The cloud service end is further used to perform character image digital processing on the acquired character image reference data to obtain a digital character image video, and to fuse the audio data with the digital character image video to obtain a digital character driving video in which the digital character explains the object to be explained; The playback control end is used to perform pre-distribution processing on the digital character driving video returned by the cloud service end, generate digital character audio and video, and distribute the digital character audio and video to the playback output end; The playback output terminal is used to output the audio and video of the digital character.

10. The digital character audio and video playback system according to claim 9, characterized in that: Also includes: The playback output terminal is used to send the interactive data generated based on the audio and video of the digital character to the cloud service terminal; The cloud service end is used to generate interactive reply data according to the interactive data, and perform data conversion processing on the interactive reply data to obtain interactive audio data; and fuse the interactive audio data with the digital character image video to obtain a digital character interactive driving video in which the digital character explains according to the interactive reply data; The playback control end is used to perform pre-distribution processing on the digital character interaction driving video returned by the cloud service end to generate digital character interaction audio and video, and distribute the digital character interaction audio and video to the playback output end.

11. The digital character audio and video playback system according to claim 9, characterized in that: The playback control end is used to perform pre-distribution processing on the digital character driving video returned by the cloud service end, generate digital character audio and video, and distribute the digital character audio and video to the playback output end, including: Based on the triggering operation of the audio and video playback in the audio and video playback platform, the playback control end distributes the digital character audio and video to the playback output end.

12. A computer storage medium, characterized in that: The method comprises a computer program, which, when executed on an electronic device, enables the electronic device to execute the steps in the method for generating digital character audio and video according to any one of claims 1 to 7; Alternatively, executing the steps in the digital character live interactive method as claimed in claim 8; Alternatively, the steps performed by the execution side in the digital character audio and video playback system as claimed in any one of claims 9 to 11 are performed.

13. An electronic device, characterized in that: include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, the program executes the steps of the method for generating digital character audio and video according to any one of claims 1 to 7; Alternatively, executing the steps in the digital character live interactive method as claimed in claim 8; Alternatively, the steps performed by the execution side in the digital character audio and video playback system as claimed in any one of claims 9 to 11 are performed.

Images