Digital human driving method and system, electronic equipment and computer readable storage medium

Abstract

The invention discloses a digital human driving method and system, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a voice text corresponding to a to-be-sent voice; running a pre-trained digital human-driven network based on the digital human-driven process, and through the digital human-driven network and the voice text, generating a digital human audio and video stream of to-be-sent voice sent by the digital human; the digital human-driven process transmits the generated digital human audio and video streams to the audio and video transmission process in real time through a process pipeline, and the digital human-driven process and the audio and video transmission process run in parallel; and transmitting the digital human audio and video streams to downlink equipment in real time based on the audio and video transmission process, wherein the downlink equipment is used for playing the digital human audio and video streams in real time. Through the scheme provided by the invention, the real-time driving of the digital person can be better realized, so that the interaction of live broadcast through the digital person is better, and the user experience in the digital person live broadcast process is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a digital human driving method, system, electronic device and computer-readable storage medium. Background Art

[0002] In the live broadcast industry, the cost of hiring high-value and high-value anchors is high, which makes the return on investment of merchants hiring anchors for live broadcasts low, so some small and medium-sized merchants are less willing to start broadcasting. Compared with real people, digital people have the characteristics of low cost, 24-hour broadcasting, high controllability, and high appearance. They can effectively reduce the cost of starting broadcasts for merchants, increase the number and duration of broadcasts, and enrich the content of live broadcasts.

[0003] In the related technologies, since the process of driving a digital human is relatively complicated, the required computing resources and process are also relatively complicated. Therefore, the driving of a digital human during live broadcast is usually offline, that is, the merchant prepares a live broadcast text in advance as the speech content of the digital human, and the system then performs digital human voice synthesis, lip movement generation, facial expression and body movement generation, etc. based on the live broadcast text, thereby generating a live broadcast video of the digital human. The generation of a live broadcast video of a digital human usually takes a relatively long period of time, and then the generated live broadcast video of the digital human is repeatedly played on the client.

[0004] However, the offline driving method of digital humans cannot respond to new inputs immediately, which means that only pre-generated live videos can be played during live broadcasts. User interactions can only be expressed repeatedly through pre-generated videos, resulting in poor live interactivity and poor user experience during the live broadcast. Summary of the invention

[0005] The present application provides a digital human driving method, system, electronic device and computer-readable storage medium, which can better realize the real-time driving of the digital human, thereby making the live broadcast through the digital human more interactive and improving the user experience during the digital human live broadcast. The specific scheme is as follows:

[0006] In a first aspect, the present application provides a digital human driving method, the method comprising:

[0007] Get the voice text corresponding to the voice to be sent;

[0008] Running a pre-trained digital human driving network based on the digital human driving process, and generating a digital human audio and video stream through the digital human driving network and the voice text, in which the digital human emits the voice to be sent;

[0009] The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is arranged between the digital human driving process and the audio and video transmission process, and the digital human driving process and the audio and video transmission process run in parallel;

[0010] Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time, and the downstream device is used to play the digital human audio and video stream in real time.

[0011] In a second aspect, the present application also provides a digital human driving system, the system comprising a cloud service end, a business service end and a streaming end;

[0012] The streaming end is used to send a streaming request to the business service end in response to a streaming instruction triggered by a user, wherein the streaming request includes the digital human information of the digital human for performing the to-be-sent voice;

[0013] The business service end is used to determine the digital human information according to the streaming request after receiving the streaming request, and send the digital human information to the cloud service end;

[0014] The cloud service end is used to obtain the voice text corresponding to the voice to be sent, and run the pre-trained digital human driving network through the digital human driving process. The digital human driving process generates a digital human audio and video stream through the digital human driving network and based on the voice text and the digital human information. The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline. The digital human driving process runs in parallel with the audio and video transmission process, and transmits the digital human audio and video stream to the downstream device in real time through the audio and video transmission process, wherein the downstream device is used to play the digital human audio and video stream in real time. The target digital human is a digital human that matches the digital human information, and the process pipeline is set between the digital human driving process and the audio and video transmission process.

[0015] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and computer program instructions stored in the memory and executable on the processor; when the processor executes the computer program instructions, it implements the method described in any one of the first aspects.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement any of the methods described in the first aspect.

[0017] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described in the first aspect when executed by a processor.

[0018] Compared with the prior art, this application has the following advantages:

[0019] The digital human driving method provided in the embodiment of the present application obtains the voice text corresponding to the to-be-sent voice, runs the pre-trained digital human driving network based on the digital human driving process, and generates the digital human audio and video stream of the to-be-sent voice through the digital human driving network and the voice text. That is to say, the digital human audio and video stream is generated by the digital human driving process, and the digital human driving process then transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is set between the digital human driving process and the audio and video transmission process, and the process pipeline is used to transmit the generated digital human audio and video stream to the audio and video transmission process in real time. Data is transmitted between the audio and video transmission processes. Since the digital human driving process runs in parallel with the audio and video transmission process, the two processes can independently perform task processing, that is, the generation process and transmission process of the digital human audio and video stream in the present application are independently and parallelly performed by two independent processes. Therefore, after the digital human driving process generates a new digital human audio and video stream, it can send the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, so that the audio and video transmission process can transmit the received digital human audio and video stream to the downstream device in real time, so that the downstream device can play the digital human audio and video stream in real time.

[0020] It can be seen that the solution provided by the present application uses two parallel processes, the digital human driving process and the audio and video transmission process, to respectively generate the digital human audio and video stream and transmit the audio and video stream downstream, and sets a process pipeline between the two processes, so that the process of generating the digital human audio and video stream and the process of transmitting the digital human audio and video stream can be carried out simultaneously. Since the digital human driving process will transmit the digital human audio and video stream to the audio and video transmission process in real time through the process pipeline after generating the digital human audio and video stream, the audio and video transmission process can obtain the digital human audio and video stream generated by the downstream device in real time and transmit it downstream to the downstream device. The downstream device can obtain and play the generated audio and video stream in real time, thereby realizing real-time driving and real-time playback of the digital human. During the live broadcast process, the digital human audio and video stream that meets the user's needs can be driven and played in real time according to the user's real-time needs. For example, the voice text input by the merchant can be obtained in real time and the corresponding digital human audio and video stream can be generated and played in real time according to the voice text. For another example, the reply text can be generated according to the customer's barrage or other forms of questions, and the corresponding digital human audio and video stream can be generated and played in real time according to the reply text, etc., thereby improving the interactivity of the live broadcast and making the user experience better during the live broadcast. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic diagram of the application scenario of the digital human driving solution provided in this application;

[0022] Figure 2 is a flowchart of an example of a digital human driving method provided in an embodiment of the present application;

[0023] Figure 3 is a flowchart of another example of the digital human driving method provided in the embodiment of the present application;

[0024] Figure 4 is a structural diagram of a digital human driving device provided in an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solution of the present application, the present application is described clearly and completely below in conjunction with the drawings in the embodiments of the present application. However, the present application can be implemented in many other ways different from the following description. Therefore, based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "source domain", "third", etc. in the claims, description and drawings of the present application are used to distinguish similar objects and are not used to describe a specific order or sequence. The data used in this way are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including", "having" and their variants are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] In order to facilitate understanding of the various embodiments of the present application, the application background of the embodiments is described.

[0029] In the live broadcast industry, the cost of hiring high-value and high-value anchors is high, which makes the return on investment of merchants hiring anchors for live broadcasts low, so some small and medium-sized merchants are less willing to start broadcasting. Compared with real people, digital people have the characteristics of low cost, 24-hour broadcasting, high controllability, and high appearance. They can effectively reduce the cost of starting broadcasts for merchants, increase the number and duration of broadcasts, and enrich the content of live broadcasts.

[0030] In the related technologies, since the process of driving a digital human is relatively complicated, the required computing resources and process are also relatively complicated. Therefore, the driving of a digital human during live broadcast is usually offline, that is, the merchant prepares a live broadcast text in advance as the speech content of the digital human, and the system then performs digital human voice synthesis, lip movement generation, facial expression and body movement generation, etc. based on the live broadcast text, thereby generating a live broadcast video of the digital human. The generation of a live broadcast video of a digital human usually takes a relatively long period of time, and then the generated live broadcast video of the digital human is repeatedly played on the client.

[0031] However, the offline driving method of digital humans cannot respond to new inputs immediately, which means that only pre-generated live videos can be played during live broadcasts. User interactions can only be expressed repeatedly through pre-generated videos, resulting in poor live interactivity and poor user experience during the live broadcast.

[0032] In order to solve the above problems, the embodiments of the present application provide a digital human driving method, system, electronic device and computer-readable storage medium, which are intended to better realize the real-time driving of the digital human, thereby making the live broadcast through the digital human more interactive and improving the user experience during the digital human live broadcast.

[0033] The digital human driving method provided in this application can be used to generate digital human audio and video streams, specifically used to generate corresponding digital human audio and video streams according to voice text driving. The solution provided in this application can be used in live broadcast platforms, and can also be used in other scenarios with relatively high requirements for real-time driving of digital humans, such as question-and-answer robot scenarios, online teaching and training platforms, customer service platforms, social platforms, etc. This application does not specifically limit the specific application scenarios of the digital human driving solution provided in this application.

[0034] In order to facilitate understanding of the method embodiments of the present application, the application scenarios are introduced. Figure 1 , Figure 1 This is a schematic diagram of the application scenario of the solution provided in the embodiment of the present application, and is also a network structure diagram of the digital human driving system provided in the present application. For the convenience of description, the following describes the embodiment of the present application by taking the live broadcast platform as an example. This application scenario is a schematic illustration and is not intended to be a specific description of its application scenario. Figure 1As shown, in this application scenario, a client 105, a streaming end 101, a business service end 102, a scheduling service end 103, a cloud service end 104, and a content distribution network (Content Delivery Network, CDN) network are set up. In this embodiment, the streaming end 101 and the business service end 102 are connected through network communication, the business service end 102 and the scheduling service end 103 are connected through network communication, the scheduling service end 103 and the business service end 102 are connected to the cloud service end 104 through network communication, the cloud service end 104 and the CDN network are connected through network communication, and the CDN network and the client 105 are connected through network communication, so that data transmission is performed between the end devices that have established connections.

[0035] The streaming end 101 can be an electronic device with display and data processing functions, such as a mobile phone, a tablet computer (pad), a smart watch, a desktop computer, a smart TV, a VR device, a vehicle-mounted device, a wearable device, a laptop computer, etc. The streaming end 101 can be a device used by a merchant. The merchant can trigger a live broadcast on the streaming end 101 and input the voice text to be played, so that the streaming end 101 sends the voice text input by the user to the business service end 102.

[0036] The business service end 102 is used to process the digital human driven business, and the business processing service end has a high computing power. The business processing service end can be a single server or a server cluster. The business processing service end is used to receive voice text from the push stream end 101 and transmit it to the cloud service end 104. It can also be used to obtain the start live broadcast instruction, push stream instruction, digital human information, etc. from the push stream end 101, and send a scheduling request to the dispatching service end based on the start live broadcast instruction and push stream instruction, and send the digital human information to the cloud service end 104. The business service end 102 can also provide other specific services for the push stream end 101 and the client 105, such as user information access, website access, application program access, etc., which are not specifically limited in this application.

[0037] The scheduling server 103 is used to allocate machine resources according to the machine resource information of the cloud server 104 after receiving the start live broadcast instruction or streaming instruction sent by the business server 102, specifically which cloud machine is allocated to generate the digital human audio and video stream.

[0038] The cloud service end 104 is used to generate a corresponding digital human audio and video stream according to the voice text, and send the generated audio and video stream to the CDN network.

[0039] The CDN network is used to forward the digital human audio and video streams to each client 105, so that the client 105 can broadcast the digital human audio and video streams in real time, thereby realizing the digital human live broadcast.

[0040] The client 105 can be a mobile phone, tablet computer (pad), smart watch, desktop computer, smart TV, VR equipment, vehicle-mounted equipment, wearable device, laptop computer and other electronic devices with display function and data processing function. The client 105 is used to play the generated digital human audio and video stream. The client 105 is a device for watching the live broadcast of the digital human.

[0041] Embodiment 1

[0042] The first embodiment of the present application provides a method for driving a digital human, which can be applied to electronic devices, such as servers, laptops, tablet computers, desktop computers, mobile phones, smart TVs, smart handsets, VR devices, clients, and other electronic devices with data processing functions. Alternatively, the method can also be applied to a digital human driving system, which includes various end devices, and each end device executes various steps of the method provided in the present application, thereby realizing the driving of the digital human and obtaining the digital human audio and video stream. The execution subject will be introduced in detail later.

[0043] like Figure 2 As shown, the digital human driving method provided in the first embodiment of the present application includes the following steps S110 to S140. In the embodiment of the present application, the electronic device that executes steps S110 to S140 is an example of a cloud server of a digital human driving system. Of course, the electronic device can also be other devices, such as a server, a laptop computer, etc., which is not specifically limited in the present application.

[0044] Step S110: Acquire the speech text corresponding to the speech to be sent.

[0045] Optionally, an electronic device (such as a cloud server) can obtain voice text from the streaming end. Specifically, a merchant or other person can input or edit the voice text corresponding to the voice to be sent on behalf of the streaming end. The streaming end sends the manually edited voice text to the cloud server, so that the cloud server can obtain the voice text corresponding to the manually pre-edited voice to be sent from the streaming end.

[0046] Alternatively, the cloud server can also generate a voice text corresponding to the voice to be sent according to the voice demand information input by the user. Specifically, the merchant can input the voice demand information that needs to be live or played at the push stream end. For example, the voice demand information may include the main content of the voice, the style of the voice content, the overall voice emotion, etc. The cloud server can generate the corresponding voice text according to the voice demand information input by the user. In this embodiment, the user can only provide a general description of the voice content, and the electronic device can obtain the matching voice text, which improves the convenience of text input for the user.

[0047] Optionally, step S110 may obtain the voice text corresponding to the voice to be sent according to the following steps S111 to S113.

[0048] Step S111: Obtaining the customer's interactive questions in real time.

[0049] The above interactive questions can be the bullet screen content obtained from the client in real time, such as Figure 3 As shown, the customer can input the bullet screen on the client, and the client can send the bullet screen content to the cloud server in real time. The cloud server determines the bullet screen content obtained from the client in real time as an interactive question. The above-mentioned interactive question can also be a question text entered by the customer on the client, or a question voice entered by the customer on the client, etc., which is not specifically limited in this application. In the embodiment of the present application, the cloud server can pull the bullet screen content from the client, and specifically can pull the bullet screen content from the client at a shorter time interval. For example, the bullet screen content can be pulled from the client at a time interval less than a preset time length. The preset time length can be less than 200 milliseconds, but is not limited to this. The time interval for pulling the bullet screen content is set to be relatively small, so that the bullet screen content can be replied to more promptly through the digital human, thereby improving the real-time interaction with the customer, so that the delay within 200 milliseconds can be achieved, and the real-time interaction between the digital human and the user can be better guaranteed.

[0050] Step S112: Determine the reply content corresponding to the interactive question.

[0051] Optionally, the electronic device may call a pre-trained large model and determine the reply content corresponding to the interactive question through the large model.

[0052] The electronic device can specifically call the big model service, that is, the electronic device can access the big model server, which is equipped with a pre-trained big model. The interaction question is sent to the big model server, and the big model server inputs the interaction question into the pre-trained big model, so that the big model outputs the corresponding reply content, and the big model server returns the reply content to the cloud server.

[0053] The above-mentioned large model refers to a large language model (LLM for short), which is a language model trained with a large amount of text data and can perform various natural language processing tasks, such as text generation, translation, question answering, etc. The above-mentioned large model can be a large language model such as BERT, GPT, T5, etc., but is not limited thereto.

[0054] In a specific embodiment, the electronic device can use a streaming call to call a pre-trained large model, and obtain the reply content corresponding to the interactive question in real time through the large model. Specifically, the streaming call to the large model can be implemented based on the Server-SentEvents (SSE) protocol. SSE is a one-way communication protocol that allows the server to actively send data to the receiving end without the receiving end constantly initiating requests. It is a simple and efficient technology suitable for application scenarios that require real-time data updates. Through the SSE protocol, the large model server can transmit the output question reply to the cloud server in real time, so that the cloud server can obtain the corresponding question reply in real time. The embodiment of the present application can also use other protocols to transmit the question reply to the cloud server. The above-mentioned streaming call refers to the large model gradually returning the output when generating the result and sending it to the cloud server in real time, so that the user can obtain part of the result in a shorter time without waiting for the entire processing to be completed, so that the audio and video stream of the digital human can be gradually generated according to the real-time obtained question reply, so that the digital human live broadcast can respond to customer questions more promptly in the live broadcast, improve the interactivity and real-time performance, and the large model reply delay can be reduced to less than 500 milliseconds, effectively improving the response speed and providing a better user experience.

[0055] Alternatively, the cloud server may also be configured with a pre-trained large model to facilitate rapid invocation of the model to generate corresponding reply content.

[0056] The large language model can quickly and accurately obtain the response content corresponding to the interactive questions, so as to quickly respond to customers' barrage questions or other questions.

[0057] Step S113: determining the reply content as the voice text corresponding to the voice to be sent.

[0058] This embodiment generates corresponding reply content as voice text according to the customer's barrage or other questions, so that a digital human audio and video stream matching the reply content can be quickly and in real time and pushed to the client for playback, so that during the live broadcast, the customer's questions can be quickly responded to through the digital human, thereby improving the interactivity with the customer and providing a better user experience.

[0059] Step S120: running a pre-trained digital human driving network based on the digital human driving process, and generating a digital human audio and video stream of the digital human emitting the to-be-sent voice through the digital human driving network and the voice text.

[0060] Among them, the electronic device is configured with a digital human driving process, and the digital human driving process is used to run a digital human driving network to generate a digital human audio and video stream.

[0061] The above-mentioned digital human driving network is used to generate corresponding digital human audio and video streams according to the speech text. The digital human driving network may include various models for generating digital human audio and video, for example, it may include a text-to-speech model, a lip driving model, an image generation model, an audio feature extraction model, an encoding model, etc. It may also include other models, which are not specifically limited in this application.

[0062] The digital human driving network can be trained using at least one of the following training algorithms: supervised learning algorithm, semi-supervised learning algorithm, semi-supervised learning algorithm, reinforcement learning, transfer learning algorithm, and self-supervised learning algorithm in the relevant technology. For example, a training sample can be obtained first, and the training sample can include a sample voice text and a sample digital human audio and video stream. The sample voice text is input into the digital human driving network to be trained to obtain an output digital human audio and video stream, and then the model parameters of each model in the digital human driving network to be trained are adjusted according to the consistency between the output digital human audio and video stream and the sample digital human audio and video stream to obtain a trained digital human driving network that meets the requirements. The specific training process of the digital human driving network can refer to the relevant technology and will not be described in detail here.

[0063] In step S120, the voice text corresponding to the to-be-sent speech can be input into the digital human driving network to output the digital human audio and video stream in which the to-be-sent speech is sent by the digital human. In this case, the digital human driving network can determine the pre-set digital human as the digital human in the digital human audio and video stream, that is, the digital human driving network drives the pre-set digital human according to the voice text, thereby generating the corresponding digital human audio and video stream.

[0064] In a specific embodiment, step S120 can be implemented by the following steps: obtaining target digital human information, and generating a digital human audio and video stream in which the target digital human sends the to-be-sent voice through the digital human driving network, the voice text, and the target digital human information. Specifically, the voice text and the target digital human information can be input into the digital human driving network to generate the above-mentioned digital human audio and video stream.

[0065] Specifically, the electronic device can obtain the target digital human information from the streaming end, and the user can select the target digital human information on the streaming end, and the streaming end transmits the target digital human information to the electronic device (such as the cloud service end). The target digital human information may include the identification information of the target digital human (such as the target digital human ID), the timbre information of the target digital human, the image of the target digital human, the name of the target digital human, and other information. After the cloud service end obtains the target digital human information, it can select the target digital human that meets the target digital human information from the digital human library to drive, thereby generating the corresponding digital human audio and video stream.

[0066] The above-mentioned digital human audio and video stream includes a video stream and a corresponding audio stream. The video stream is a video stream composed of various video frames. The audio stream corresponds to the video stream. The two are played together to display the live picture and sound of the digital human.

[0067] In a specific embodiment, the digital human driving network may include a text-to-speech model and a lip driving model. Accordingly, in the above step S120, the following steps S121 to S124 may be followed to generate a digital human audio and video stream in which the digital human sends the to-be-sent speech.

[0068] Step S121: converting the speech text into corresponding speech audio through the text-to-speech model.

[0069] The above-mentioned text-to-speech (TTS) model can convert written text into speech audio, which is used to form the audio stream in the digital human audio and video stream.

[0070] Step S122: extracting audio features corresponding to the speech audio.

[0071] The above-mentioned audio features may include at least one of pitch, timbre, Mel-frequency cepstral coefficients, phoneme timing, spectrogram, pronunciation features, emotional features, and vocal tract features, but are not limited thereto.

[0072] Specifically, a trained audio feature extraction model can be used to extract audio features corresponding to speech audio, or time domain analysis, frequency domain analysis, peak analysis, etc. can be used to extract audio features corresponding to speech audio. For example, the waveform characteristics of the audio signal can be analyzed to determine the amplitude and time domain shape of the audio, the nature of the signal can be judged by counting the number of times the signal crosses zero in the time domain, and the time domain signal can be converted into the frequency domain using Fourier transform (FFT) to analyze its spectrum, etc. Audio feature extraction can also be carried out in other ways, which are not specifically limited in this application.

[0073] The above-mentioned audio feature extraction model also belongs to the digital human driving network, that is, the above-mentioned digital human driving network also includes the audio feature extraction model.

[0074] Step S123: Generate lip parameters of the digital human through the lip driving model and based on the audio features.

[0075] The above-mentioned lip driving model is a pre-trained model that can generate corresponding lip parameters according to audio features. Lip parameters may include lip opening, mouth corner width, lip shape contour, mouth corner height, upper and lower lip ratio, inner lip visibility, tooth visibility, lip angle, etc., but are not limited to words. According to the lip parameters, the lip shape of the digital human can be accurately determined.

[0076] When the digital person is a target digital person selected by the user, lip parameters of the target digital person may be generated.

[0077] Step S124: generating a digital human audio and video stream in which the digital human emits the to-be-sent voice according to the lip parameters and the voice audio.

[0078] Specifically, a video screen of the digital human can be generated according to the lip parameters, that is, a video frame of the digital human can be generated, and then each video frame is correspondingly merged with the voice audio to generate a digital human audio and video stream in which the digital human emits the to-be-sent voice.

[0079] In a specific embodiment, the digital human driving network may further include a picture generation model, wherein the picture generation model is used to generate a digital human image according to lip parameters, and determine the digital human image as a video picture of the digital human.

[0080] This embodiment can conveniently and efficiently generate speech audio corresponding to speech text and lip parameters corresponding to speech audio features through a text-to-speech model and a lip drive model, thereby efficiently determining a digital human audio and video stream.

[0081] In a specific embodiment, before the step of converting the speech text into corresponding speech audio through the text-to-speech model, the above method may also include: preprocessing the speech text to obtain processed text; converting the speech text into corresponding speech audio through the text-to-speech model includes: converting the processed text into corresponding speech audio through the text-to-speech model.

[0082] Preprocessing the speech text may include at least one of the following: converting numbers in the speech text into text, unifying the upper and lower case in the speech text, expanding abbreviations in the speech text into full names, removing special characters, labeling grammatical roles of each word or character, dividing sentence boundaries, converting the speech text into a phoneme sequence, marking the stress, rhythm and emotion of the speech text, correcting spelling errors, and correcting grammatical errors. By preprocessing the speech text, the text-to-speech model can better understand and parse the input text, thereby generating more natural and high-quality speech audio.

[0083] In one implementation, the digital human driving network can be trained through the following steps S10 to S30.

[0084] Step S10: Obtaining an initial digital human driving network.

[0085] Each model in the above-mentioned initial digital human driving network can be a pre-trained model or a model with model parameter initialization. When each model is a pre-trained model, the subsequent training of the digital human driving network can be faster and more efficient. The model structure and weight of each model in the initial digital human driving network are set to the initial state.

[0086] Optionally, each model in the above-mentioned initial digital human driving network is a lightweight neural network model, wherein the lightweight neural network model can be a model in which the depth and width of the convolutional layer are less than a preset threshold and the weight matrix is ​​a sparse matrix.

[0087] Step S20: performing pruning and acceleration processing on the initial digital human driving network to obtain a pruned digital human driving network.

[0088] Among them, pruning and accelerating the initial digital human driving network may include at least one of the following: removing non-important weights in the initial digital human driving network, converting the weights and / or activation values ​​of each model of the initial digital human driving network from a floating point data type to an integer data type, removing at least one data processing layer of the initial digital human driving network, and performing low-rank decomposition on the weight matrix of the initial digital human driving network. The pruning and accelerating method may also include other methods, which are not specifically limited in this application. The structure of each model in the digital human driving network after pruning is simpler than the model structure of each model of the initial digital human driving network, so that the digital human audio and video stream can be obtained more quickly, so that the present application can better realize the real-time driving of the digital human. Among them, pruning and accelerating the initial digital human driving network means pruning and accelerating each model of the initial digital human driving network.

[0089] The above-mentioned non-important weights may be weights whose absolute value is less than a preset threshold and whose importance score is less than a preset score. The importance score of the weight can be determined by evaluating the contribution of the weight to the model performance. Alternatively, the importance of the weight can be determined in any of the following ways: if the gradient of the weight maintains a low gradient value during model training, the weight is determined to be a non-important weight; the performance of the model is judged by the performance of the model on the validation set before and after the weight is removed. If the decline is less than the preset range, the removed weight is determined to be a non-important weight. Non-important weights can also be determined in other ways, which are not specifically limited in this application. Removing non-important weights can make the structure of the model simpler, thereby making the model's operation speed faster and improving the real-time performance of digital human drive.

[0090] The weights and / or activation values ​​of each model of the initial digital human driving network are converted from floating point data types to integer data types, for example, the weights and / or activation values ​​of each model of the initial digital human driving network of float32 data types are converted to weights and / or activation values ​​of int8 data types. After the weights and / or activation values ​​of each model are converted from floating point data types to integer data types, the data processing efficiency of the model can be greatly improved, thereby improving the digital human generation efficiency of the digital human driving network and improving the real-time performance of digital human driving.

[0091] At least one data processing layer of the initial digital human driving network may be removed, or the non-important data processing layer may be determined first and then removed. The method for determining the non-important data processing layer may refer to the method for determining the non-important weight, which will not be described in detail here.

[0092] Low-rank decomposition of the weight matrix of the initial digital human driving network refers to decomposing a weight matrix of the model into the product of two or more low-rank matrices, thereby reducing the storage requirements and computational complexity of the original matrix, making the model's calculation speed faster and better realizing the real-time nature of digital human driving.

[0093] Step S30: training the pruned digital human driving network to obtain a digital human driving network for generating digital human audio and video streams.

[0094] The process of training the pruned digital human driving network can adopt supervised learning algorithms, unsupervised learning algorithms, etc. in related technologies, which will not be described in detail in this application.

[0095] This implementation method can make the generation efficiency of the digital human audio and video stream of the final digital human driving network higher by pruning and accelerating the initial digital human driving network, thereby further improving the real-time performance of generating the digital human audio and video stream.

[0096] In one implementation, the above-mentioned preprocessing method for preprocessing the speech text may also be optimized to speed up the preprocessing speed and enable the preprocessed text to generate corresponding speech audio more efficiently.

[0097] In one implementation, step S122 may extract the audio features corresponding to the speech audio in the following steps: extract the audio features corresponding to the speech audio based on the principle that the extracted audio features have coherence and consistency. Specifically, the coherence and consistency of the audio features may include: the audio features extracted in different time periods have similar structures, different audio features maintain feature smoothness, etc., but is not limited thereto.

[0098] In one embodiment, the method may further include the following steps: optimizing the audio feature recognition link to optimize the audio feature extraction process so that the extracted audio features have better quality. The audio feature recognition link refers to the entire process of extracting audio features from speech audio.

[0099] Step S130: the digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline.

[0100] Among them, the electronic device is also configured with an audio and video transmission process, and the process pipeline is set between the digital human driving process and the audio and video transmission process. The process pipeline is used to transmit data between the digital human driving process and the audio and video transmission process. The digital human driving process and the audio and video transmission process run in parallel.

[0101] In the embodiment of the present application, the digital human driving process can transmit the new digital human audio and video stream to the audio and video transmission process in real time when detecting that a new digital human audio and video stream is generated.

[0102] Step S140: Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time.

[0103] The downstream device is used to play the digital human audio and video stream in real time. The downstream device can be a live broadcast terminal for watching live broadcasts, or it can be other devices for watching videos. In the embodiment of the present application, the cloud service end can directly transmit the digital human audio and video stream to the downstream device in real time through the audio and video transmission process, or it can forward the digital human audio and video stream to the downstream device through a transfer device. For example, the cloud service end can transmit the digital human audio and video stream to the CDN network in real time through the audio and video transmission process, and then select the optimal path through the CDN network to transmit the digital human audio and video stream to each live broadcast terminal (i.e., the downstream device) in real time. The speed of sending the digital human audio and video stream to each live broadcast terminal can be increased through the CDN network, thereby further improving the real-time distribution of the digital human audio and video stream.

[0104] The digital human driving method provided in the embodiment of the present application obtains the voice text corresponding to the to-be-sent voice, runs the pre-trained digital human driving network based on the digital human driving process, and generates the digital human audio and video stream of the to-be-sent voice through the digital human driving network and the voice text. That is to say, the digital human audio and video stream is generated by the digital human driving process, and the digital human driving process then transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is set between the digital human driving process and the audio and video transmission process, and the process pipeline is used to transmit the generated digital human audio and video stream to the audio and video transmission process in real time. Data is transmitted between the audio and video transmission processes. Since the digital human driving process runs in parallel with the audio and video transmission process, the two processes can independently perform task processing, that is, the generation process and transmission process of the digital human audio and video stream in the present application are independently and parallelly performed by two independent processes. Therefore, after the digital human driving process generates a new digital human audio and video stream, it can send the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, so that the audio and video transmission process can transmit the received digital human audio and video stream to the downstream device in real time, so that the downstream device can play the digital human audio and video stream in real time.

[0105] It can be seen that the solution provided by the present application uses two parallel processes, the digital human driving process and the audio and video transmission process, to respectively generate the digital human audio and video stream and transmit the audio and video stream downstream, and sets a process pipeline between the two processes, so that the process of generating the digital human audio and video stream and the process of transmitting the digital human audio and video stream can be carried out simultaneously. Since the digital human driving process will transmit the digital human audio and video stream to the audio and video transmission process in real time through the process pipeline after generating the digital human audio and video stream, the audio and video transmission process can obtain the digital human audio and video stream generated by the downstream device in real time and transmit it downstream to the downstream device. The downstream device can obtain and play the generated audio and video stream in real time, thereby realizing real-time driving and real-time playback of the digital human. During the live broadcast process, the digital human audio and video stream that meets the user's needs can be driven and played in real time according to the user's real-time needs. For example, the voice text input by the merchant can be obtained in real time and the corresponding digital human audio and video stream can be generated and played in real time according to the voice text. For another example, the reply text can be generated according to the customer's barrage or other forms of questions, and the corresponding digital human audio and video stream can be generated and played in real time according to the reply text, etc., thereby improving the interactivity of the live broadcast and making the user experience better during the live broadcast. Through the pruning acceleration of the digital human driving network and the optimization of the audio feature extraction method provided by this application, the ratio of the time required for text-to-speech Real-Time Factor, TTS-RTF for short) can be reduced to 0.1, thereby fully ensuring the playback requirements, and the lip-driven frame generation speed can be increased to 25FPS (frames per second), thereby reducing the overall streaming delay of the digital human audio and video stream pushed to the client to less than 1 second, thereby better ensuring the real-time live broadcast effect.

[0106] In one implementation, the digital human driving method can be applied to a digital human driving system, such as Figure 3 As shown, the digital human driving system includes a cloud service end, a business service end and a streaming end.

[0107] The above steps S110 to S140 can be executed by the cloud server, that is, the digital human audio and video stream is generated by the cloud server and transmitted to the downstream device. In this way, through the cloud direct push, the digital human generation algorithm environment is deployed in the cloud, and the high computing resources in the cloud can be used to generate digital human audio and video more quickly and efficiently, effectively solving the computing power problem, and reducing the video transmission and rendering delay by about 40 milliseconds.

[0108] Before step S110, the digital human driving method may further include the following steps S110a to S110b.

[0109] Step S110a: the streaming end sends a streaming request to the business service end in response to a streaming instruction triggered by the user, wherein the streaming request includes target digital human information of the target digital human for performing the to-be-sent voice.

[0110] The relevant contents of determining the target digital human information by the streaming end are described above and will not be described in detail here. After the user inputs the target digital human information in the streaming end, the streaming end receives the streaming instruction triggered by the user.

[0111] Step S110b: After receiving the streaming request, the business service end determines the target digital human information according to the streaming request, and sends the target digital human information to the cloud service end.

[0112] Correspondingly, in step S120, the digital human audio and video stream can be generated by the following steps: the digital human drives the network, and based on the voice text and the target digital human information, generates a digital human audio and video stream in which the target digital human sends the to-be-sent voice.

[0113] In this implementation, the business service end processes the digital human driving business at the streaming end and extracts information and sends it to the business service end to trigger the digital human driving program, which can make the entire digital human driving process more efficient and reasonable.

[0114] In a specific embodiment, Figure 3 As shown, the digital human driving system may further include a scheduling service end.

[0115] Before the business server sends the target digital human information to the cloud server, the method further includes the following steps S110c to S110d.

[0116] Step S110c: the business service end sends a scheduling request to the scheduling service end.

[0117] The business service end executes step S110c after receiving the streaming instruction from the streaming end.

[0118] Step S110d: In response to receiving the scheduling request, the scheduling server selects a target cloud machine in an idle state from the cloud server.

[0119] The business server sends the target digital human information to the cloud server, including: the business server sends the target digital human information to the target cloud machine.

[0120] The steps of step S110 to step S140 may be specifically executed by the target cloud machine.

[0121] In this embodiment, the scheduling server allocates idle cloud machines to perform digital human generation services for the digital human driving services of the streaming end. This can reasonably and effectively utilize the resources of the cloud server to generate digital humans, making it possible to more efficiently arrange the digital human generation needs of each streaming end, thereby improving generation efficiency.

[0122] In a specific embodiment, the streaming end, the business service end, the cloud service end and the downlink device all transmit data via the real-time communication RTC protocol. Since the RTC protocol has low latency and efficient data transmission performance, the present application uses the RTC protocol to transmit data between various device ends, which can provide more immediate feedback on the data to be transmitted, thereby better ensuring the real-time transmission performance of the digital human audio and video stream.

[0123] In a specific embodiment, the digital human driving system may further include a content distribution network CDN and a stream pulling end, and the downstream device is the stream pulling end, which is the above-mentioned client.

[0124] In step S140, the digital human audio and video stream can be transmitted to the downstream device in real time according to the following steps: based on the audio and video transmission process, the digital human audio and video stream is transmitted to the content distribution network in real time; the content distribution network transmits the digital human audio and video stream to the stream pulling end in real time.

[0125] Specifically, the cloud service end can compress and encode the digital human audio and video stream to obtain a compressed audio and video stream, and then transmit the compressed audio and video stream to the content distribution network in real time based on the audio and video transmission process, so as to distribute it to the client. The client can render and play the audio and video stream after decoding based on the corresponding decoding method. Specifically, the S265 software encoder can be used to compress the digital human audio and video stream into a compressed audio and video stream in the H.265 / HEVC (High Efficiency Video Coding) format. This embodiment can reduce the audio and video stream before transmitting it through encoding technology, thereby improving the transmission speed of the audio and video stream, making the real-time performance of live broadcasting through digital humans higher.

[0126] The method may further include: the stream pulling end rendering and playing the digital human audio and video stream in real time.

[0127] The CDN network data transmission method and beneficial effects can be referred to the description above and will not be described in detail here. In this embodiment, data can also be transmitted between the cloud service end and the CDN network, and between the CDN network and the client through the RTC protocol to increase the data transmission speed and better ensure the real-time performance of the live broadcast. This application uses full-link RTC streaming to reduce the streaming delay to less than 2 seconds, ensuring the real-time performance of the live broadcast.

[0128] The following uses a live broadcast example to introduce the specific process of the digital human driving system generating digital human audio and video streams. Figure 3 As shown, the digital human driving method in this example may include the following steps 1 to 13.

[0129] Step 1: The streaming end sends a streaming request to start the digital human live broadcast to the business server.

[0130] For example, the merchant can trigger the start live broadcast operation in the central console of the streaming end or the anchor interface of the streaming end or the browser and other software used by the merchant, so that the streaming end sends a streaming request to the business server after receiving the start live broadcast operation.

[0131] Step 2: The business server sends a scheduling request to the scheduling server.

[0132] Step 3: The scheduling server allocates the target cloud machine and triggers the target cloud machine to execute the streaming process.

[0133] Step 4: The target cloud machine sends a request to the business server to obtain business data.

[0134] Step 5: The business server obtains the target digital human information and voice text according to the target digital human information and voice text carried in the streaming request, and sends the target digital human information and voice text to the target cloud machine.

[0135] The target digital person information and voice text in step 5 are Figure 3 Business information in.

[0136] Step 6: The target cloud machine generates a digital human audio and video stream based on the target digital human information and voice text, and sends the digital human audio and video stream to the CDN network.

[0137] Among them, various functional modules can be set in the cloud server, and each functional module is used to implement different steps of generating digital human audio and video streams. For example, the cloud server may include a barrage module, an AI scheduling module, a rendering orchestration module, a full-link RTC module, etc. The barrage module is used to process the barrage, the AI ​​scheduling module is used to call various AI model operators, the rendering orchestration module is used to render and generate the final digital human audio and video stream, and the full-link RTC module is used to control the transmission of data between device ends through RTC. Other functional modules can also be set, which are not specifically limited in this application.

[0138] Step 7: The CDN network sends the digital human audio and video stream to the client, so that the client renders and plays the digital human audio and video stream through the live broadcast room or other playback platform.

[0139] The RTC method is used for streaming, which reduces the time consumption from the original 6 seconds to 2 seconds.

[0140] Step 8: The bullet screen server pulls the bullet screen information from the client.

[0141] The small pull interval makes the time of pulling bullet comments less than 200 milliseconds.

[0142] Step 9: The barrage server sends the barrage information to the target cloud machine.

[0143] Step 10: The target cloud machine sends the bullet message information to the big model service so that the big model service generates a corresponding question response.

[0144] Step 11: The big model service sends the question response to the target cloud machine.

[0145] Specifically, the large model service is called in a streaming manner. During the test, the response time of the large model is reduced from about 3 seconds to about 500 milliseconds, which greatly reduces the response time and improves real-time performance.

[0146] Step 12: The target cloud machine takes the question response as voice text and executes steps 6-7.

[0147] Step 13: The target cloud machine sends the question response to the client, so that the client displays the question response.

[0148] Specifically, the target cloud machine can also use streaming to send the question reply to the client, so that the reply time can be reduced from the original 2 seconds to 400 milliseconds.

[0149] This example introduces the process of driving the digital human at each device end in the digital human driving system during the live broadcast through the overall operation process. The relevant details have been introduced in detail in the previous embodiments and will not be described in detail here.

[0150] Embodiment 2

[0151] The second embodiment of the present application also provides a digital human driving system, which includes a cloud service end, a business service end and a streaming end.

[0152] The streaming end is used to send a streaming request to the business service end in response to a streaming instruction triggered by a user, wherein the streaming request includes the digital human information of the digital human for performing the to-be-sent voice.

[0153] The business service end is used to determine the digital human information according to the streaming request after receiving the streaming request, and send the digital human information to the cloud service end.

[0154] The cloud service end is used to obtain the voice text corresponding to the voice to be sent, and run the pre-trained digital human driving network through the digital human driving process. The digital human driving process generates a digital human audio and video stream through the digital human driving network and based on the voice text and the digital human information. The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline. The digital human driving process runs in parallel with the audio and video transmission process, and transmits the digital human audio and video stream to the downstream device in real time through the audio and video transmission process, wherein the downstream device is used to play the digital human audio and video stream in real time. The target digital human is a digital human that matches the digital human information, and the process pipeline is set between the digital human driving process and the audio and video transmission process.

[0155] The specific contents of the digital human driving system have been described in detail in the first embodiment. For the related contents, please refer to the first embodiment and will not be described in detail here.

[0156] Embodiment 3

[0157] The third embodiment of the present application also provides a digital human driving device corresponding to the digital human driving method embodiment provided in the first embodiment. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple. For the details of the relevant technical features and the effects achieved, please refer to the corresponding description of the digital human driving method embodiment provided above. Figure 4 As shown, the digital human driving device provided in this embodiment includes:

[0158] An acquisition unit 310 is used to acquire a speech text corresponding to the speech to be sent;

[0159] A generating unit 320 is used to run a pre-trained digital human driving network based on the digital human driving process, and generate a digital human audio and video stream through which the digital human sends the to-be-sent voice through the digital human driving network and the voice text;

[0160] The transmission unit 330 is used to enable the digital human driving process to transmit the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is set between the digital human driving process and the audio and video transmission process, and the digital human driving process and the audio and video transmission process run in parallel; based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time, and the downstream device is used to play the digital human audio and video stream in real time.

[0161] The fourth embodiment of the present application also provides an electronic device embodiment corresponding to the digital human driving method provided in the first embodiment. The following description of the electronic device embodiment is only illustrative. The electronic device embodiment is as follows:

[0162] Please refer to Figure 5 Understand the above electronic devices, Figure 5 The electronic device provided in this embodiment includes: a processor 1001, a memory 1002, a communication bus 1003, and a communication interface 1004;

[0163] The memory 1002 is used to store computer instructions for data processing. When the computer instructions are read and executed by the processor 1001, the following steps are performed:

[0164] Get the voice text corresponding to the voice to be sent;

[0165] Running a pre-trained digital human driving network based on the digital human driving process, and generating a digital human audio and video stream through the digital human driving network and the voice text, in which the digital human emits the voice to be sent;

[0166] The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is arranged between the digital human driving process and the audio and video transmission process, and the digital human driving process and the audio and video transmission process run in parallel;

[0167] Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time, and the downstream device is used to play the digital human audio and video stream in real time.

[0168] The fifth embodiment of the present application also provides a computer-readable storage medium for implementing the method described in the first embodiment. The computer-readable storage medium embodiment provided in the present application is described relatively simply. For the relevant parts, please refer to the corresponding description of the above method embodiment. The following described embodiment is only illustrative.

[0169] The computer readable storage medium provided in this embodiment stores computer instructions, and when the instructions are executed by a processor, the following steps are implemented:

[0170] Get the voice text corresponding to the voice to be sent;

[0171] Running a pre-trained digital human driving network based on the digital human driving process, and generating a digital human audio and video stream through the digital human driving network and the voice text, in which the digital human emits the voice to be sent;

[0172] The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is arranged between the digital human driving process and the audio and video transmission process, and the digital human driving process and the audio and video transmission process run in parallel;

[0173] Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time, and the downstream device is used to play the digital human audio and video stream in real time.

[0174] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0175] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0176] 1. Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves.

[0177] 2. Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] Although the present application is disclosed as above in the form of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

Claims

1. A digital human driving method, characterized in that: The method comprises: Get the voice text corresponding to the voice to be sent; Running a pre-trained digital human driving network based on the digital human driving process, and generating a digital human audio and video stream through the digital human driving network and the voice text, in which the digital human emits the voice to be sent; The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline, wherein the process pipeline is arranged between the digital human driving process and the audio and video transmission process, and the digital human driving process and the audio and video transmission process run in parallel; Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the downstream device in real time, and the downstream device is used to play the digital human audio and video stream in real time.

2. The digital human driving method according to claim 1, characterized in that: The digital human driving network is trained by the following steps: Acquire the initial digital human driver network; Performing pruning and acceleration processing on the initial digital human driving network to obtain a pruned digital human driving network; The pruned digital human driving network is trained to obtain a digital human driving network for generating digital human audio and video streams.

3. The digital human driving method according to claim 3, characterized in that: The pruning and acceleration process of the initial digital human driving network includes at least one of the following: Removing non-important weights in the initial digital human driving network; The weights and / or activation values ​​of each model of the initial digital human driving network are converted from floating point data types to integer data types.

4. The digital human driving method according to claim 1, characterized in that: The digital human driving network includes a text-to-speech model and a lip driving model; The step of generating a digital human audio and video stream by driving the network and the voice text through the digital human to send the voice to be sent includes: Converting the speech text into corresponding speech audio by using the text-to-speech model; Extracting audio features corresponding to the speech audio; Generate lip parameters of the digital human through the lip driving model and based on the audio features; A digital human audio and video stream in which the digital human emits the to-be-sent voice is generated according to the lip parameters and the voice audio.

5. The digital human driving method according to claim 4, characterized in that: The digital human driving network also includes an audio feature extraction model; The extracting audio features corresponding to the speech audio includes: The audio features corresponding to the speech audio are extracted through the audio feature extraction model.

6. The digital human driving method according to any one of claims 1 to 5, characterized in that: The step of obtaining the voice text corresponding to the voice to be sent includes: Get the voice text corresponding to the pre-edited voice to be sent; Or, get customer interaction questions in real time; Determine the reply content corresponding to the interactive question; The reply content is determined as the voice text corresponding to the voice to be sent.

7. The digital human driving method according to claim 6, characterized in that: The determining of the reply content corresponding to the interactive question includes: The pre-trained big model is called, and the reply content corresponding to the interactive question is determined by the big model.

8. The digital human driving method according to claim 7, characterized in that: The calling of the pre-trained big model and determining the reply content corresponding to the interactive question by using the big model includes: The pre-trained big model is called in a streaming call manner, and the reply content corresponding to the interactive question is obtained in real time through the big model.

9. The digital human driving method according to any one of claims 1 to 5, characterized in that: The method is applied to a digital human driving system, which includes a cloud service end, a business service end, and a streaming end; The steps of obtaining the voice text corresponding to the to-be-sent voice and transmitting the digital human audio and video stream to the downstream device in the audio and video transmission process are performed by the cloud service end; Before obtaining the voice text corresponding to the voice to be sent, the method further includes: The streaming end sends a streaming request to the business service end in response to a streaming instruction triggered by a user, wherein the streaming request includes digital human information of the digital human for performing the to-be-sent voice; After receiving the streaming request, the business service end determines the target digital human information according to the streaming request, and sends the target digital human information to the cloud service end; The step of generating a digital human audio and video stream by driving the network and the voice text through the digital human to send the voice to be sent includes: The digital human drives the network, and based on the voice text and the target digital human information, generates a digital human audio and video stream that uses the target digital human to send out the voice to be sent.

10. The digital human driving method according to claim 9, characterized in that: The digital human driving system also includes a scheduling service end; Before the business server sends the target digital human information to the cloud server, the method further includes: The business service end sends a scheduling request to the scheduling service end; The scheduling server selects a target cloud machine in an idle state from the cloud server in response to receiving the scheduling request; The sending of the digital human information to the cloud service end includes: sending the digital human information to the target cloud machine; The step of obtaining the voice text corresponding to the to-be-sent voice to the step of transmitting the digital human audio and video stream to the downstream device in the audio and video transmission process is executed by the target cloud machine.

11. The digital human driving method according to claim 9, characterized in that: The streaming end, the business service end, the cloud service end and the downlink device all transmit data via the real-time communication RTC protocol.

12. The digital human driving method according to claim 11, characterized in that: The digital human driving system further includes a content distribution network and a stream pulling end, and the downstream device is the stream pulling end; The real-time transmission of the digital human audio and video stream to the downlink device based on the audio and video transmission process includes: Based on the audio and video transmission process, the digital human audio and video stream is transmitted to the content distribution network in real time; The content distribution network transmits the digital human audio and video stream to the stream pulling end in real time; The method further comprises: The stream pulling end renders and plays the digital human audio and video stream in real time.

13. A digital human driving system, characterized in that: The system includes a cloud service end, a business service end and a streaming end; The streaming end is used to send a streaming request to the business service end in response to a streaming instruction triggered by a user, wherein the streaming request includes the digital human information of the digital human for performing the to-be-sent voice; The business service end is used to determine the digital human information according to the streaming request after receiving the streaming request, and send the digital human information to the cloud service end; The cloud service end is used to obtain the voice text corresponding to the voice to be sent, and run the pre-trained digital human driving network through the digital human driving process. The digital human driving process generates a digital human audio and video stream through the digital human driving network and based on the voice text and the digital human information. The digital human driving process transmits the generated digital human audio and video stream to the audio and video transmission process in real time through the process pipeline. The digital human driving process runs in parallel with the audio and video transmission process, and transmits the digital human audio and video stream to the downstream device in real time through the audio and video transmission process, wherein the downstream device is used to play the digital human audio and video stream in real time. The target digital human is a digital human that matches the digital human information, and the process pipeline is set between the digital human driving process and the audio and video transmission process.

14. An electronic device, characterized in that: include: A processor, a memory, and computer program instructions stored on the memory and executable on the processor; When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 is implemented.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to any one of claims 1 to 8.

Images