Virtual video live broadcast processing method and device, storage medium and electronic equipment

Abstract

The present disclosure provides a virtual video live processing method and device, electronic equipment and computer readable storage medium; it relates to video processing technical field.The method comprises the following steps: obtaining text data and action data of a virtual object; the action data comprises an action identifier of a specified action and a start position identifier of the specified action in the text data; generating audio data and expression data of the virtual object according to the text data, and generating a face image of the virtual object according to the expression data; generating a background image sequence containing the specified action according to the start position identifier and the action identifier; the background image sequence comprises a background image; performing image fusion processing on the face image and the background image to obtain a live video frame; and real-time synthesizing the live video frame and the audio data into a live video stream.The present disclosure can specify the object action of the virtual object in the virtual live according to the personalized needs of the user, and make the object action of the virtual object match the text.

Description

Technical Field

[0001] This disclosure relates to the field of video processing technology, and more specifically, to a virtual video live streaming processing method, a virtual video live streaming processing device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] With the development of communication technology, network communication bandwidth has been greatly improved, and video live streaming technology has become increasingly mature and applied in various fields. Simultaneously, with the development of artificial intelligence technology, text-to-speech (TTS) and image synthesis technologies have become research hotspots. The combination of video live streaming technology and artificial intelligence technology can play a role in many areas, such as replacing real people for virtual news broadcasts and replacing game streamers for virtual game commentary, demonstrating broad application prospects.

[0003] In virtual video live streaming technology, the generation of audio and images requires a significant amount of computation time. To ensure the real-time nature of virtual video live streaming, the implementation of real-time virtual video streaming becomes a crucial factor affecting the final quality of the live video. Most existing real-time video live streaming methods are designed for applications with existing stable audio and image data inputs (such as local video streaming) or where audio and image data can be quickly acquired (such as data obtained from a camera).

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a virtual video live streaming processing method, a virtual video live streaming processing device, an electronic device, and a computer-readable storage medium, thereby overcoming to some extent the problem that existing virtual video live streaming technologies cannot support personalized assignment of actions to virtual objects, and that there is a mismatch between the default actions of virtual characters and the text.

[0006] According to one aspect of this disclosure, a virtual video live streaming processing method is provided, comprising: acquiring text data and motion data of a virtual object; the motion data including an action identifier of a specified action and a start position identifier of the specified action in the text data; generating audio data and facial expression data of the virtual object based on the text data, and generating a facial image of the virtual object based on the facial expression data; generating a background image sequence containing the specified action based on the start position identifier and the action identifier; the background image sequence including a background image; performing image fusion processing on the facial image and the background image to obtain a live video frame; and synthesizing the live video frame and the audio data into a live video stream in real time.

[0007] According to one aspect of this disclosure, a virtual video live streaming processing apparatus is provided, comprising: a data acquisition module for acquiring text data and motion data of a virtual object; the motion data including an action identifier of a specified action and a start position identifier of the specified action in the text data; a face image determination module for generating audio data and facial expression data of the virtual object based on the text data, and generating a face image of the virtual object based on the facial expression data; a background image determination module for generating a background image sequence containing the specified action based on the start position identifier and the action identifier; the background image sequence including a background image; an image fusion module for performing image fusion processing on the face image and the background image to obtain a live video frame; and a live video synthesis module for synthesizing the live video frame and the audio data into a live video stream in real time.

[0008] In one exemplary embodiment of this disclosure, the face image determination module includes: a text determination unit, used to determine the corresponding text to be converted based on text data; an audio data generation unit, used to perform text-to-speech processing on the text data to generate audio data; and an expression data generation unit, used to generate expression data of a virtual object based on the audio data.

[0009] In one exemplary embodiment of this disclosure, the audio data generation unit is configured to: convert the text to be converted into the corresponding target word vector; encode and decode the target word vector to generate audio features of the target word vector; and synthesize the audio features to generate audio data.

[0010] In one exemplary embodiment of this disclosure, the facial expression data generation unit is configured to: determine the number of facial expression frames contained in the facial expression data, and determine the number of facial expression frames as a first quantity; determine the image playback frame rate, and determine the playback duration of the facial expression data based on the first quantity and the image playback frame rate; the playback duration is the same as the audio duration of the audio data.

[0011] In one exemplary embodiment of this disclosure, the background image determination module includes: a default image acquisition unit for acquiring a default background image; an action image acquisition unit for acquiring an action background image corresponding to a specified action based on an action identifier; an image order determination unit for determining the image arrangement order of the default background image and the action background image based on a start position identifier; and an image arrangement unit for sorting multiple background images according to the image arrangement order to generate a background image sequence.

[0012] In one exemplary embodiment of this disclosure, the motion image acquisition unit is configured to: determine the motion sequence contained in the motion data; the motion sequence includes one or more specified actions; and acquire the motion background image corresponding to each specified action from a preset motion library according to the motion identifier.

[0013] In one exemplary embodiment of this disclosure, the image order determination unit further includes: a text length determination subunit, used to determine the text length of the text data; a start identifier determination subunit, used to obtain a first quantity and determine the start frame identifier of the specified action in the background image based on the text length, the start position identifier, and the first quantity; an end identifier determination subunit, used to determine the number of action background images as the number of action images and determine the end frame identifier based on the start frame identifier and the number of action images; and an order determination subunit, used to compare the size of the end frame identifier with the first quantity to determine the image arrangement order based on the comparison result.

[0014] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a first sequence determination subunit, which is configured to: determine the background image before the start frame identifier as the default background image; determine the background image between the start frame identifier and the end frame identifier as the action background image; and determine the background image between the end frame identifier and the first quantity identifier as the default background image.

[0015] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a second sequence determination subunit, which is configured to: determine the current start frame identifier and the current end frame identifier corresponding to the currently specified action; determine the background image between the end frame identifier of the previous specified action and the current start frame identifier as the default background image; determine the background image between the current start frame identifier and the current end frame identifier as the action background image; and determine the background image between the current end frame identifier and the start frame identifier of the next specified action as the default background image.

[0016] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a third sequence determination subunit, which is configured to: configure the background image before the start frame identifier as the default background image; determine the number of filter frames based on a first quantity and the start frame identifier; determine a set of filter action images from multiple action background images based on the number of filter frames; and configure the background image between the start frame identifier and the first quantity identifier as the set of filter action images.

[0017] In one exemplary embodiment of this disclosure, the third sequence determination subunit includes an image filtering subunit, which is configured to: sequentially select a number of filtering frames of action background images from a plurality of action background images as a set of filtering action images; or randomly select a number of filtering frames of action background images from a plurality of action background images as a set of filtering action images.

[0018] In one exemplary embodiment of this disclosure, the image fusion module is configured to: determine a background image that matches the face image as the matching background image; obtain a pre-built image fusion model; and perform image fusion processing on the face image and the matching background image through the image fusion model to generate a live video frame.

[0019] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the method described in any of the preceding methods by executing the executable instructions.

[0020] According to one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.

[0021] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the virtual video live streaming processing method provided in the various embodiments described above.

[0022] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0023] In a virtual video live streaming processing method provided in an exemplary embodiment of this disclosure, audio data and a facial image of a virtual object can be generated based on the acquired text data. A background image containing a specified action can be generated based on the acquired action identifier and start position identifier. The facial image of the virtual object and the background image are then image fused to generate a live video frame. A live video stream can be synthesized in real time based on the live video frame and audio data. On one hand, the action data can be an object action specified by the user for the virtual object. Generating a live video stream based on the action data can achieve the purpose of the user specifying actions for the virtual object in the virtual live video. On the other hand, determining the corresponding playback position of the specified action and text data based on the action identifier and start position identifier, so that the specified action is played at the specified position, can effectively solve the problem of mismatch between the virtual object's action and the text.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 A schematic diagram of an exemplary system architecture for a virtual video live streaming processing method and apparatus to which embodiments of the present disclosure can be applied is shown.

[0027] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0028] Figure 3 A flowchart illustrating a virtual video live streaming processing method according to an embodiment of the present disclosure is shown.

[0029] Figure 4 The diagram illustrates an overall framework of a virtual video live streaming service according to one embodiment of the present disclosure.

[0030] Figure 5 The flowchart illustrating a process for determining a background image sequence based on an action identifier and a start position identifier according to an embodiment of the present disclosure is shown.

[0031] Figure 6 The flowchart illustrating a process for determining the image arrangement order based on a start position identifier according to an embodiment of the present disclosure is shown schematically.

[0032] Figure 7 A flowchart illustrating the determination of the image arrangement order according to one embodiment of the present disclosure is shown.

[0033] Figure 8 A flowchart illustrating the determination of the image arrangement order according to another embodiment of the present disclosure is shown.

[0034] Figure 9 A flowchart illustrating the determination of the image arrangement order according to yet another embodiment of the present disclosure is shown.

[0035] Figure 10 The diagram illustrates an implementation framework of a virtual video streaming service according to an embodiment of the present disclosure.

[0036] Figure 11 The diagram illustrates a structural diagram of a virtual video live streaming processing apparatus according to an embodiment of the present disclosure. Detailed Implementation

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0038] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0040] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] The solutions provided in this application involve technologies such as virtual video live streaming processing using artificial intelligence, and are specifically illustrated through the following embodiments:

[0042] Figure 1 A schematic diagram of the system architecture of an exemplary application environment to which an embodiment of the virtual video live streaming processing method and apparatus of the present disclosure can be applied is shown.

[0043] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables. Terminal devices 101, 102, and 103 may be various electronic devices with displays, including but not limited to desktop computers, laptops, smartphones, and tablets. It should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster consisting of multiple servers.

[0044] The virtual video live streaming processing method provided in this embodiment is generally executed by server 105, and correspondingly, the virtual video live streaming processing device is generally located in server 105. However, it is readily understood by those skilled in the art that the virtual video live streaming processing method provided in this embodiment can also be executed by terminal devices 101, 102, and 103, and correspondingly, the virtual video live streaming processing device can also be located in terminal devices 101, 102, and 103. This exemplary embodiment does not impose any special limitations on this. For example, in one exemplary embodiment, a user may upload text data and action data to server 105 through terminal devices 101, 102, and 103. The server, through the virtual video live streaming processing method provided in this embodiment, generates audio data and a facial image of a virtual object based on the text data, generates a background image containing a specified action based on the action data, performs image fusion processing on the facial image and background image to generate a live video frame, performs real-time synthesis processing on the live video frame and audio data to generate a live video stream, and transmits the live video stream to terminal devices 101, 102, and 103, etc.

[0045] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown.

[0046] It should be noted that, Figure 2 The computer system 200 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0047] like Figure 2As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 202 or programs loaded from storage section 208 into random access memory (RAM) 203. The RAM 203 also stores various programs and data required for system operation. The CPU 201, ROM 202, and RAM 203 are interconnected via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0048] The following components are connected to I / O interface 205: an input section 206 including a keyboard, mouse, etc.; an output section 207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, modem, etc. The communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0049] In particular, according to embodiments of this disclosure, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs various functions defined in the methods and apparatus of this application. In some embodiments, computer system 200 may also include an AI (Artificial Intelligence) processor for processing computational operations related to machine learning.

[0050] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0051] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0052] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware, and the described units may also be disposed in a processor. In some cases, the names of these units do not constitute a limitation on the units themselves.

[0053] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the method described in the following embodiments. For example, the electronic device may implement as Figures 3 to 10 the respective steps and the like as shown.

[0054] The technical solutions of the embodiments of the present disclosure are described in detail below:

[0055] In the virtual video live streaming technology, a corresponding live video stream is mainly generated according to audio and images, and the live video stream is pushed to a client in real time. The generation of audio and images consumes a large amount of calculation time. To ensure the real-time performance of virtual video live streaming, the implementation of real-time virtual video streaming has become an important factor affecting the quality of the final live video.

[0056] On the premise of not affecting the real-time performance, most of the existing virtual live video technologies do not support the function of allowing users to specify actions for virtual characters, and instead use default actions provided by the service. In the virtual live streaming solution using default actions, there may be a problem that the actions of the virtual character in the virtual video cannot match the text and the performance is unnatural. For example, the default action for greeting is all a waving action performed by the virtual character, and when the text corresponding to the greeting is "welcome", a problem that the action does not match the text may occur.

[0057] In addition, since the virtual character in the virtual live broadcast can only use the provided default actions, using only the default actions may not meet the personalized needs of users. For example, the text "cheer up" only corresponds to one default action, and the user cannot specify a self-designed action for the virtual character according to their own will.

[0058] Based on one or more of the above problems, the present exemplary embodiment provides a virtual video live streaming processing method. The virtual video live streaming processing method may be applied to the above server 105, or may be applied to one or more of the above terminal devices 101, 102, and 103, which is not specifically limited in the present exemplary embodiment. With reference to Figure 3 as shown, the virtual video live streaming processing method may include the following steps S310 to S350:

[0059] Step S310. Obtain text data and motion data of virtual objects; motion data includes the motion identifier of the specified motion and the start position identifier of the specified motion in the text data.

[0060] Step S320. Generate audio data and facial expression data of the virtual object based on the text data, and generate a facial image of the virtual object based on the facial expression data.

[0061] Step S330. Generate a background image sequence containing the specified action based on the start position identifier and the action identifier; the background image sequence includes background images.

[0062] Step S340. Perform image fusion processing on the face image and the background image to obtain live video frames.

[0063] Step S350. Combine the live video frames and audio data into a live video stream in real time.

[0064] In the virtual video live streaming processing method provided in this example embodiment, audio data and a facial image of a virtual object can be generated based on the acquired text data. A background image containing a specified action can be generated based on the acquired action identifier and start position identifier. The facial image of the virtual object and the background image are then fused to generate a live video frame. A live video stream can be synthesized in real time based on the live video frame and audio data. On one hand, the action data can be an object action specified by the user for the virtual object. Generating a live video stream based on the action data can achieve the purpose of the user specifying actions for the virtual object in the virtual live video. On the other hand, determining the corresponding playback position of the specified action and text data based on the action identifier and start position identifier, so that the specified action is played at the specified position, can effectively solve the problem of mismatch between the virtual object's action and the text.

[0065] The steps described above in this example implementation will now be explained in more detail.

[0066] In step S310, text data and action data of virtual objects are acquired; the action data includes the action identifier of the specified action and the start position identifier of the specified action in the text data.

[0067] In this example implementation, the text data can be the text spoken by the virtual object in the virtual live video. For example, the user can input the text data themselves through the live streaming client, or the user can select the corresponding text data from a variety of optional texts provided by the live streaming client; the text data can also be text data pre-stored in the server. The virtual object can be a virtual image modeled in the virtual live video. The virtual object can be a virtual object of a person, animal, plant, or other things, and is different from a real object. For example, the virtual object can be a cartoon character, an animal such as a penguin, or a plant such as a sunflower. The motion data can be the data used to drive the virtual object to display corresponding actions. The specified action can be the relevant action generated by the virtual object in the virtual video according to the user's personalized needs. The action identifier can be the identifier of a specific specified action, and one specified action can correspond to one unique identifier. The action identifier can be represented by motion-id. The start position identifier can be the identifier of the relative byte position of the start of the specified action in the text data. The start position identifier can be used to determine the position corresponding to a specified action and specific text data. For example, if the text data includes 10 characters and the specified action starts playing from the 3rd character, then the start position identifier can be set to 3.

[0068] When conducting virtual video live streaming, the virtual video live streaming server can first obtain text data and motion data, and then process the obtained text data and motion data accordingly to generate the virtual live streaming video. (Reference) Figure 4 , Figure 4 This diagram schematically illustrates an overall framework of a virtual video live streaming service according to an embodiment of the present disclosure. The virtual video live streaming service may include a live streaming client 410 and a virtual video live streaming server 420. The virtual video live streaming server 420 may include a text-to-speech module 421, a 3D rendering module 422, and a video streaming module 423. The virtual video live streaming server 420 can obtain text data and motion data from the live streaming client 410; for example, a user can input or select corresponding text data and motion data through the live streaming client. The virtual video live streaming server 420 can also obtain pre-stored text data and motion data from a server database. After obtaining the text data and motion data, the virtual video live streaming server 420 can parse and process the text data and motion data.

[0069] After parsing and processing the text data, the virtual video live streaming server 420 can obtain the text length L, for example, the text length can be 10. After parsing and processing the motion data, the virtual video live streaming server 420 can obtain the motion identifier of the specified motion and the start position identifier of the specified motion in the text data. For example, the start position identifier can be represented by index, index = I; the motion identifier can be: motion-id = D.

[0070] In step S320, audio data and facial expression data of the virtual object are generated based on the text data, and a facial image of the virtual object is generated based on the facial expression data.

[0071] In this example implementation, the facial expression data can be text data corresponding to the facial expressions of a virtual object in a virtual video. The audio data can be simulated human voices used to match the live video output image of the virtual object. The facial image can be the facial image of the virtual object output in the live video.

[0072] refer to Figure 4 After acquiring text and motion data, the virtual video live streaming server 420 can determine the target text corresponding to the text data, and then process the target text into speech using the text-to-speech module 421 to generate audio data and facial expression data for the virtual object. After obtaining the facial expression data, the 3D rendering module 422 can render the facial expression data to generate the face image of the virtual object.

[0073] In step S330, a background image sequence containing a specified action is generated based on the start position identifier and the action identifier; the background image sequence includes a background image.

[0074] In this example implementation, the background image can be an image corresponding to the virtual object's facial image and containing other features of the virtual object; for example, the background image can include the virtual object's body movements and the environmental background information corresponding to the generated video output image. Since the user has specified corresponding actions for certain text data, the background image is an image containing the specified actions of the virtual object.

[0075] After determining the start position identifier and action identifier, the 3D rendering module 422 in the virtual video live streaming server 420 can determine the background image corresponding to the text data based on the text data, start position identifier and action identifier, and the background image contains the specified action so that the virtual objects in the virtual live streaming video can perform the specified action.

[0076] In step S340, the face image and the background image are fused to obtain a live video frame.

[0077] In this example implementation, image fusion processing involves fusing the facial image of a virtual object with its matching background image to obtain a live video frame. The live video frame can be any video frame corresponding to the virtual live video. A virtual live video can be generated based on these live video frames.

[0078] After generating the face and background images of the virtual object, the virtual video live streaming server 420 can perform image fusion processing on the face and background images to form complete live video frames.

[0079] In step S350, the live video frames and audio data are combined in real time to form a live video stream.

[0080] In this example implementation, the live video stream can be a video stream synthesized in real time based on live video frames and audio data. The virtual video live streaming server can push the live video stream to the client, which can be a live streaming client that can receive the live video stream and present the virtual live video in real time based on the live video stream.

[0081] refer to Figure 4 After receiving live video frames and audio data, the video streaming module 423 in the virtual video live streaming server 420 can perform video synthesis processing on the live video frames and audio data, synthesize a live video stream in real time, and push the live video stream to the live streaming client 410 so that the live streaming client 410 can present the virtual live video in real time according to the live video stream. The live streaming client 410 obtains the live video stream and presents the corresponding virtual live video in real time. This can be achieved through the following technical solution: the virtual video live streaming server 420 pushes the live video stream to the streaming media interface of the corresponding streaming media playback address, so that the live streaming client 410 pulls the live video stream based on the streaming media playback address and presents the real-time synthesized virtual live video through the user interface of the live streaming client 410.

[0082] For example, the audio and facial expression data of virtual objects can be generated through the following steps.

[0083] In one example embodiment of this disclosure, the corresponding text to be converted is determined based on the text data; text-to-speech processing is performed on the text to be converted to generate audio data; and facial expression data of a virtual object is generated based on the audio data.

[0084] The text to be converted can be the text itself undergoing text-to-speech processing. The text data to be converted can include the original text and associated text. For example, in some human-computer interaction scenarios, when the original text in the text data is "How to use product A," the associated text could be the text corresponding to the specific usage instructions for product A. Text-to-speech processing is the process of converting the text to be converted into a natural speech stream. The audio data can be the corresponding audio generated after the text to be converted has undergone text-to-speech conversion processing.

[0085] For details, please refer to Figure 4 The virtual video live streaming server 420 can use the text-to-speech module 421 to process the text to be converted into speech. This module can contain a trained text-to-speech model, i.e., a pre-built initial deep learning text-to-speech model is trained using deep learning methods. After training, the trained text-to-speech model is obtained. The text to be converted is then processed using this model to generate audio data.

[0086] In addition, the text-to-speech module 421 can also generate corresponding facial expression data based on the text to be converted. This can be achieved through the following technical solution: predicting the key points of the mouth of the virtual object in the virtual live video based on the audio data of the text to be converted; normalizing the key points of the mouth to adapt them to a standard facial template; performing dimensionality reduction on the normalized key points of the mouth to obtain the lip shape features of the corresponding virtual object; performing semantic analysis on the text to be converted to obtain the semantics represented by the text to be converted; and determining the facial expression data that matches the semantics based on the semantics represented by the text to be converted.

[0087] In one example embodiment of this disclosure, the text to be converted is transformed into the corresponding target word vector, the target word vector is encoded and decoded to generate audio features of the target word vector, and the audio features are synthesized to generate audio data.

[0088] The target word vector can be the corresponding vector generated after vectorizing the text to be converted. The audio features can be the key features corresponding to the audio data converted from the target word vector. For example, audio features can include short-time zero-crossing rate, short-time energy, spectral centroid, etc.

[0089] Specifically, after the virtual video live streaming server 420 determines the text to be converted based on the text data, the text-to-speech module 421 inputs the text to be converted into the text-to-speech model. The text to be converted can first be transformed into corresponding target word vectors. For example, the text to be converted can be a sentence. The text is segmented to determine the word vectors corresponding to each word. After obtaining the target word vectors, they can be encoded to obtain intermediate semantic identifiers. Then, through decoding, the audio features of the target word vectors are obtained. Using an audio synthesis algorithm, the obtained audio features are synthesized to generate the audio data corresponding to the text to be converted.

[0090] In one example embodiment of this disclosure, the number of facial expression frames contained in the facial expression data is determined, and the number of facial expression frames is determined as a first quantity; the image playback frame rate is determined, and the playback duration of the facial expression data is determined based on the first quantity and the image playback frame rate; the playback duration is the same as the audio duration of the audio data.

[0091] The number of facial expression frames can be the number of facial expression image frames that can be generated from the facial expression data; the first number is equal to the number of facial expression frames. The image playback frame rate can be the frequency (rate) at which bitmap images, measured in frames, continuously appear on the display. The playback duration of the facial expression data can be the length of time the facial expression data is played. The audio duration can be the length of time the audio data is played.

[0092] Specifically, after the text-to-speech module 421 converts the text to be converted into corresponding facial expression data, the number of facial expression frames contained in the facial expression data can be determined. For example, after converting the text, the number of facial expression frames obtained can be N, i.e., the first quantity is N. After determining the image playback frame rate, the playback duration of the facial expression data can be determined based on the image playback frame rate and the first quantity. In order to ensure that the lip movements in the final generated virtual live video are aligned with the audio, the audio duration of the audio data must be the same as the playback duration of the facial expression data when generating the audio data and facial expression data.

[0093] In one example embodiment of this disclosure, a default background image is obtained; an action background image corresponding to a specified action is obtained according to an action identifier; the image arrangement order of the default background image and the action background image is determined according to a start position identifier; and multiple background images are sorted according to the image arrangement order to generate a background image sequence.

[0094] The default background image can be a default background image provided in the background image library. For example, the default background image can be an image that only includes the environmental background information of the virtual object without including any actions of the virtual object. Alternatively, the default background image can be an image that includes both the environmental background information and basic actions of the virtual object. The action background image can be a background image that includes a specified action of the virtual object. The image arrangement order can be the order in which all default background images and action background images are arranged.

[0095] Specifically, since users can assign corresponding actions to virtual objects for a portion of the text data, while another portion of the text may not have a specified action, the background image can include two parts: a default background image and an action background image. The 3D rendering module 422 in the virtual video live streaming server 420 can determine the background image corresponding to the text data based on the start position identifier of the specified action in the text data and the action identifier of the specified action.

[0096] refer to Figure 5 , Figure 5 This illustration schematically shows a flowchart of determining a background image based on an action identifier and a start position identifier according to an embodiment of the present disclosure. In step S510, a default background image is obtained. For example, a default background image can be obtained from a background image library. The default background image may be an image including basic actions such as waving, shaking hands, nodding, and shaking the head. For text data where the user has not specified a corresponding action, the obtained default background image can be used as the background image corresponding to this part of the text. In step S520, for text data with a specified action, an action background image corresponding to the specified action can be obtained based on the action identifier of the specified action. After determining the default background image and the action background image, in step S530, the image arrangement order of these default background images and action background images can be determined based on the start position identifier of the specified action. In step S540, multiple background images are sorted according to the image arrangement order, that is, the default background image and the action background image are sorted to generate a background image sequence.

[0097] In one example embodiment of this disclosure, an action sequence contained in the action data is determined; the action sequence includes one or more specified actions; and the action background image corresponding to each specified action is obtained from a preset action library according to the action identifier.

[0098] The action sequence can be a sequence of one or more specified actions. If the action sequence contains multiple specified actions, it also includes the order in which these actions are performed. The preset action library can be an image library that pre-stores the background images corresponding to the specified actions.

[0099] Specifically, after acquiring the motion data, since the motion data includes motion identifiers for specified actions, the sequence of actions contained in the motion data can be determined based on these motion identifiers. Furthermore, the order of the motion identifiers in the motion data can determine the order of the specified actions within the action sequence. The data structure of the motion data acquired by the virtual video live streaming server 420 can be "index:motion-id", where index represents the starting position identifier of the specified action in the text data. The pre-stored motion background images in the preset motion library are all stored in a format where each motion identifier is bound to a set of motion background images. After acquiring the motion identifier of a specified action, the virtual video live streaming server 420 can retrieve a set of motion background images corresponding to that motion identifier from the preset motion library one by one.

[0100] In one example embodiment of this disclosure, the text length of the text data is determined; a first quantity is obtained, and a start frame identifier for a specified action in the background image is determined based on the text length, the start position identifier, and the first quantity; the number of action background images is determined as the number of action images, and an end frame identifier is determined based on the start frame identifier and the number of action images; the end frame identifier is compared with the first quantity to determine the image arrangement order based on the comparison result.

[0101] The text length can be the length of the text data itself. For example, if the text data contains the phrase "Today is a sunny and breezy day," then the text length can be defined as 10. Alternatively, the text data length can be defined based on the number of bytes it occupies. For example, if one Chinese character occupies two bytes, the text length of the aforementioned text data would be 20. This disclosure does not impose any special limitations on the calculation method for text length. The frame identifier can be the identifier of the image frame corresponding to all background images. The start frame identifier can be the position identifier where the background image of a specified action begins to play among all background images. The number of action images can be the number of background images included in a specified action. The end frame identifier can be the frame identifier corresponding to the last background image of a specified action. The end frame identifier can be determined based on the start frame identifier and the number of action images. The comparison result can be the result obtained by comparing the end frame identifier with the first quantity.

[0102] refer to Figure 6 , Figure 6The flowchart illustrating the process of determining the image arrangement order based on a start position identifier according to an embodiment of the present disclosure is shown. After acquiring text data, the virtual video live streaming server 420 can determine the text length corresponding to the text data in step S610. In step S620, a first quantity is obtained. Based on the determined text length, the obtained start position identifier, and the first quantity, the start frame identifier of the specified action in the background image can be determined. Since a virtual object's face image can be rendered and generated based on one frame of facial expression data, the first quantity can be the number of face images corresponding to the virtual object. Simultaneously, since the face images of the virtual object correspond one-to-one with the background images, the number of face images of the virtual object is the same as the number of background images; that is, the first quantity is also the number of background images.

[0103] For example, the start frame identifier for a specified action can be represented as n; the first quantity can be represented as N; the start position identifier can be represented as I; and the text length can be represented as L. The start frame representation can then be calculated according to Formula 1, and 0 <n<N+1。

[0104] n = N * I / L (Formula 1)

[0105] In step S630, the number of action background images is determined as the number of action images. The end frame identifier corresponding to a specified action can be determined based on the start frame identifier and the number of action images. The number of action images can be represented by f, and the end frame identifier can be represented by n-1+f. Since the background images may include a default background image, in step S640, the end frame identifier and the first number are compared, and the image arrangement order of all background images is determined based on the comparison result.

[0106] For example, the image arrangement order of the background image can be determined by the following steps.

[0107] In one example embodiment of this disclosure, if the comparison result is that the end frame identifier is less than or equal to the first quantity, then the background image before the start frame identifier is determined as the default background image; the background image between the start frame identifier and the end frame identifier is determined as the action background image; and the background image between the end frame identifier and the first quantity identifier is determined as the default background image.

[0108] Since the background image consists of two parts, a default background image and an action background image, when the comparison result is that the end frame identifier is less than or equal to the first number, the background image corresponding to the part of the text data for which the user has not set a specified action can be configured as the default background image, while the background image of the text data corresponding to the specified action can be configured as the corresponding action background image.

[0109] refer to Figure 7 , Figure 7A flowchart illustrating the determination of image arrangement order according to an embodiment of the present disclosure is shown. In step S710, the background image corresponding to the text portion before the start frame identifier is determined as the default background image. Since the text data has a corresponding designated action starting from the start frame identifier, in step S720, the background image of the text portion between the start frame identifier and the end frame identifier can be determined as the action background image. The text portion after the end frame identifier does not have a corresponding designated action, and in step S730, the background image between the end frame identifier and the first quantity identifier can be determined as the default background image. That is, when n-1+f<=N, the background image of the first n-1 frames of the N background images is configured as the default background image; the background image between the nth to n-1+fth frames is configured as the action background image corresponding to the designated action; and the background image between the n+fth to Nth frames is configured as the default background image.

[0110] In one example embodiment of this disclosure, if the action data includes a currently specified action, a previous specified action, and a next specified action, then the current start frame identifier and the current end frame identifier corresponding to the currently specified action are determined; the background image between the end frame identifier of the previous specified action and the current start frame identifier is determined as the default background image; the background image between the current start frame identifier and the current end frame identifier is determined as the action background image; and the background image between the current end frame identifier and the start frame identifier of the next specified action is determined as the default background image.

[0111] The currently specified action can be the action being processed. The previous specified action can be the action that precedes the currently specified action in sequence. The next specified action can be the action that immediately follows the currently specified action. There are no other specified actions between the previous and current specified actions. There are also no other specified actions between the current and next specified actions. The current start frame identifier can be the identifier of the start position of the currently specified action in the background image. The current end frame identifier can be the identifier of the end position of the currently specified action in the background image. The end frame identifier of the previous specified action can be the identifier of the end position of the previous specified action in the background image. The start frame identifier of the next specified action can be the identifier of the start position of the next specified action in the background image.

[0112] This disclosure uses an example of motion data containing three specified actions for illustration. (See reference...) Figure 8 , Figure 8A flowchart illustrating the determination of image arrangement order according to another embodiment of the present disclosure is shown. In step S810, the current start frame identifier and the current end frame identifier corresponding to the currently specified action are determined. When the comparison result is that the end frame identifier is less than or equal to a first number, and the action data includes the currently specified action, the previous specified action, and the next specified action, in step S820, the background image between the end frame identifier of the previous specified action and the current start frame identifier is determined as the default background image. In step S830, the background image between the current start frame identifier and the current end frame identifier is determined as the action background image. In step S840, the background image between the current end frame identifier and the start frame identifier of the next specified action is determined as the default background image. The background image between the start frame identifier and the end frame identifier of the previous specified action is the action background image corresponding to the previous specified action. The background image between the start frame identifier and the end frame identifier of the next specified action is the action background image corresponding to the next specified action.

[0113] Additionally, if the start frame marker of the previous specified action contains text data, the background image of that text data is configured as the default background image; similarly, if the end frame marker of the next specified action contains text data, the background image of that text data is configured as the default background image.

[0114] It will be readily understood by those skilled in the art that when the comparison result shows that the end frame identifier is less than or equal to the first number, and the action data contains other numbers (such as 2, 4, 5, etc.) of specified actions, the configuration method of the background image is similar to the above method, and this disclosure will not elaborate further on this.

[0115] In one example embodiment of this disclosure, if the comparison result is that the end frame identifier is greater than the first quantity, the background image before the start frame identifier is configured as the default background image; the number of filter frames is determined according to the first quantity and the start frame identifier; the set of filter action images is determined from multiple action background images according to the number of filter frames; and the background image between the start frame identifier and the first quantity identifier is configured as the set of filter action images.

[0116] The number of frames to be filtered can be the number of background images selected from the background images corresponding to a specified action and played in the virtual video. The set of background images to be filtered can be a subset of background images selected from all background images corresponding to a specified action. The set of background images to be filtered can include one image or multiple images.

[0117] refer to Figure 9 , Figure 9A flowchart illustrating the determination of image arrangement order according to another embodiment of the present disclosure is shown. When the comparison result shows that the end frame identifier is greater than a first number, in step S910, the background image corresponding to the part of text data for which the user has not set a specified action can be configured as the default background image, and the background image of the text data corresponding to the specified action can be configured as the corresponding action background image. However, when the end frame identifier is greater than the first number, if all action background images corresponding to the specified action are played in image order, the end frame of the specified action will exceed the first number. This may result in a situation where the voice broadcast of the virtual object has ended, but the virtual object is still continuing the corresponding action. Therefore, a portion of the action background images can be selected from the action background images of the specified action for playback.

[0118] For the default background image, since the text data before the start frame identifier does not have a corresponding specified action, the background image corresponding to the text portion before the start frame identifier can be configured as the default background image, that is, the background image of the first n-1 frames can be configured as the default background image. In step S920, the number of filter frames is determined according to the first quantity and the start frame identifier. Taking the text data corresponding to one specified action as an example, N-(n-1) frames of action background images can be played between the start frame identifier and the first quantity identifier. Therefore, the number of filter frames can be determined as N-(n-1). In step S930, the set of filter action images is determined from multiple action background images according to the number of filter frames. For example, N-(n-1) action background images can be selected from the action background images of the specified action according to the number of filter frames as the set of filter action images. After determining the set of filter action images, in step S940, the background image between the start frame identifier and the first quantity identifier can be configured as the set of filter action images.

[0119] In one example embodiment of this disclosure, a number of action background images are sequentially selected from a plurality of action background images to form a set of action images to be filtered; or a number of action background images are randomly selected from a plurality of action background images to form a set of action images to be filtered.

[0120] Selecting images sequentially can be done by choosing the action image set according to the order of the action background images. Random selection can be done by randomly selecting the action image set from the action background images.

[0121] Specifically, selecting a set of background images from multiple action images can be done in two ways. After determining the number of frames to filter, a set of background images can be selected sequentially from the multiple background images based on that number of frames. For example, the first N-(n-1) frames can be selected sequentially from the background images of a specified action. Alternatively, N-(n-1) frames can be randomly selected from the background images of a specified action; or N-(n-1) frames can be randomly selected from the background images of a specified action at certain frame intervals.

[0122] In one example embodiment of this disclosure, a background image matching the face image is determined as the matching background image; a pre-constructed image fusion model is obtained; and the face image and the matching background image are fused using the image fusion model to generate a live video frame.

[0123] Among them, the image fusion model can be a model that performs image fusion processing on the face image and background image of a virtual object.

[0124] Specifically, in the virtual video live streaming server 420, the 3D rendering module 422 renders the facial image of the virtual object based on its facial expression data, and obtains the background image based on text data, start position markers, and action markers. This allows for the determination of the background image that matches the facial image of each virtual object. A pre-built image fusion model is then acquired; this model can be a Convolutional Neural Network (CNN) model. The facial image and background image are input into the image fusion model, which performs image fusion processing to generate a complete video output image, i.e., a live video frame.

[0125] refer to Figure 10 , Figure 10 This diagram schematically illustrates an implementation framework of a virtual video streaming service according to an embodiment of the present disclosure. Figure 10 In this context, virtual video streaming services can include text-to-speech modules, 3D rendering modules, and video streaming modules.

[0126] The live streaming client sends text data and action data to the virtual video live streaming server. The virtual video live streaming server can receive and parse the text data and action data, determine the text length of the text data, the action identifier of the specified action in the action data, and the start position identifier of the specified action in the text data.

[0127] After the virtual video live streaming server determines the text to be converted based on the text data, the text-to-speech module can generate corresponding audio and facial expression data based on the text. This audio and facial expression data is then packaged into a single data packet and pushed to the 3D rendering module. Specifically, the text-to-speech module can stream the audio and facial expression data corresponding to the text to be converted. Each time the text-to-speech module obtains a data packet containing audio and facial expression data, it pushes that data packet to the 3D rendering module. Additionally, the virtual video live streaming server can send information including the text length, an action identifier for a specified action, and a start position identifier to the 3D rendering module.

[0128] When the 3D rendering module receives a data packet containing facial expression data and audio data, it can extract the facial expression data and perform 3D rendering to obtain a set of corresponding facial images. The 3D rendering module can determine the background image corresponding to the text data based on the text length, the action identifier of the specified action, and the start position identifier. The background image includes a default background image and an action background image, and these background images are sorted. The virtual video live streaming server can perform image fusion processing on each facial image and the background image matching the facial image to obtain live video frames, and then send the live video frames and audio data to the video streaming module.

[0129] Each time the video streaming module receives a data packet pushed by the 3D rendering module, it extracts the audio data and live video frames. Using the Fast Forward Moving Picture Expert Group (FFMPEG) tool, it synthesizes the audio data and live video frames to generate a live video stream, which is then simultaneously pushed to the live streaming client. Furthermore, the video streaming module can send an end signal to the client to terminate the current video streaming service and disconnect the communication connection with the live streaming client.

[0130] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0131] Furthermore, this example embodiment also provides a virtual video live streaming processing apparatus. This virtual video live streaming processing apparatus can be applied to a server or terminal device. (See reference...) Figure 11As shown, the virtual video live streaming processing device 1100 may include a data acquisition module 1110, a face image determination module 1120, a background image determination module 1130, an image fusion module 1140, and a live video synthesis module 1150. Wherein:

[0132] The data acquisition module 1110 is used to acquire text data and action data of virtual objects; the action data includes the action identifier of the specified action and the start position identifier of the specified action in the text data; the face image determination module 1120 is used to generate audio data and facial expression data of the virtual object based on the text data, and generate a face image of the virtual object based on the facial expression data; the background image determination module 1130 is used to generate a background image sequence containing the specified action based on the start position identifier and the action identifier; the background image sequence includes a background image; the image fusion module 1140 is used to perform image fusion processing on the face image and the background image to obtain a live video frame; the live video synthesis module 1150 is used to synthesize the live video frame and the audio data into a live video stream in real time.

[0133] In one exemplary embodiment of this disclosure, the face image determination module includes: a text determination unit, used to determine the corresponding text to be converted based on text data; an audio data generation unit, used to perform text-to-speech processing on the text data to generate audio data; and an expression data generation unit, used to generate expression data of a virtual object based on the audio data.

[0134] In one exemplary embodiment of this disclosure, the audio data generation unit is configured to: convert the text to be converted into the corresponding target word vector; encode and decode the target word vector to generate audio features of the target word vector; and synthesize the audio features to generate audio data.

[0135] In one exemplary embodiment of this disclosure, the facial expression data generation unit is configured to: determine the number of facial expression frames contained in the facial expression data, and determine the number of facial expression frames as a first quantity; determine the image playback frame rate, and determine the playback duration of the facial expression data based on the first quantity and the image playback frame rate; the playback duration is the same as the audio duration of the audio data.

[0136] In one exemplary embodiment of this disclosure, the background image determination module includes: a default image acquisition unit for acquiring a default background image; an action image acquisition unit for acquiring an action background image corresponding to a specified action based on an action identifier; an image order determination unit for determining the image arrangement order of the default background image and the action background image based on a start position identifier; and an image arrangement unit for sorting multiple background images according to the image arrangement order to generate a background image sequence.

[0137] In one exemplary embodiment of this disclosure, the motion image acquisition unit is configured to: determine the motion sequence contained in the motion data; the motion sequence includes one or more specified actions; and acquire the motion background image corresponding to each specified action from a preset motion library according to the motion identifier.

[0138] In one exemplary embodiment of this disclosure, the image order determination unit further includes: a text length determination subunit, used to determine the text length of the text data; a start identifier determination subunit, used to obtain a first quantity and determine the start frame identifier of the specified action in the background image based on the text length, the start position identifier, and the first quantity; an end identifier determination subunit, used to determine the number of action background images as the number of action images and determine the end frame identifier based on the start frame identifier and the number of action images; and an order determination subunit, used to compare the size of the end frame identifier with the first quantity to determine the image arrangement order based on the comparison result.

[0139] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a first sequence determination subunit, which is configured to: determine the background image before the start frame identifier as the default background image; determine the background image between the start frame identifier and the end frame identifier as the action background image; and determine the background image between the end frame identifier and the first quantity identifier as the default background image.

[0140] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a second sequence determination subunit, which is configured to: determine the current start frame identifier and the current end frame identifier corresponding to the currently specified action; determine the background image between the end frame identifier of the previous specified action and the current start frame identifier as the default background image; determine the background image between the current start frame identifier and the current end frame identifier as the action background image; and determine the background image between the current end frame identifier and the start frame identifier of the next specified action as the default background image.

[0141] In one exemplary embodiment of this disclosure, the sequence determination subunit includes a third sequence determination subunit, which is configured to: configure the background image before the start frame identifier as the default background image; determine the number of filter frames based on a first quantity and the start frame identifier; determine a set of filter action images from multiple action background images based on the number of filter frames; and configure the background image between the start frame identifier and the first quantity identifier as the set of filter action images.

[0142] In one exemplary embodiment of this disclosure, the third sequence determination subunit includes an image filtering subunit, which is configured to: sequentially select a number of filtering frames of action background images from a plurality of action background images as a set of filtering action images; or randomly select a number of filtering frames of action background images from a plurality of action background images as a set of filtering action images.

[0143] In one exemplary embodiment of this disclosure, the image fusion module is configured to: determine a background image that matches the face image as the matching background image; obtain a pre-built image fusion model; and perform image fusion processing on the face image and the matching background image through the image fusion model to generate a live video frame.

[0144] The specific details of each module or unit in the aforementioned virtual video live streaming processing device have been described in detail in the corresponding virtual video live streaming processing method, so they will not be repeated here.

[0145] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0146] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0147] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A virtual video live streaming processing method, characterized in that, include: During virtual video live streaming, text data and action data of virtual objects input by the user through the live streaming client are acquired; the action data includes the action identifier of the specified action generated by the virtual object in the virtual video according to the user's personalized needs, and the starting position identifier of the specified action in the text data. The audio data and facial expression data of the virtual object are generated based on the text data, and the facial image of the virtual object is generated based on the facial expression data; Get the default background image; Obtain the action background image corresponding to the specified action based on the action identifier; Determine the text length of the text data; Obtain the first number of facial images corresponding to the virtual object, where the first number is the number of background images, and according to the formula... Calculate the start frame identifier of the specified action in the background image, where n represents the start frame identifier, N represents the first quantity, I represents the start position identifier, and L represents the text length; The number of background images is determined as the number of motion images. An end frame identifier is determined based on the start frame identifier and the number of motion images. The end frame identifier is represented as n-1+f, where f represents the number of motion images. The image arrangement order of the default background image and the action background image is determined by comparing the size of the end frame identifier with the first quantity. Specifically, if the comparison result includes the end frame identifier being less than or equal to the first quantity, the background image before the start frame identifier is determined as the default background image, the background image between the start frame identifier and the end frame identifier is determined as the action background image, and the background image between the end frame identifier and the first quantity identifier is determined as the default background image. If the comparison result includes the end frame identifier being greater than the first quantity, the background image before the start frame identifier is configured as the default background image, and the number of filter frames is determined to be N-(n-1). A set of filtered action images is determined from the plurality of action background images based on the number of filter frames, and the background image between the start frame identifier and the first quantity identifier is configured as the set of filtered action images. The default background image and the action background image are sorted according to the image arrangement order to generate a background image sequence containing the specified action; The face image and the background image included in the background image sequence are subjected to image fusion processing to obtain live video frames; The live video frames and the audio data are combined in real time to create a live video stream.

2. The method according to claim 1, characterized in that, The step of generating audio data and facial expression data of the virtual object based on the text data includes: The corresponding text to be converted is determined based on the text data; The text data is processed to convert it to speech, thereby generating the audio data; The virtual object's facial expression data is generated based on the audio data.

3. The method according to claim 2, characterized in that, The step of performing text-to-speech processing on the text data to generate the audio data includes: The text to be converted is transformed into the corresponding target word vector, and the target word vector is encoded and decoded to generate the audio features of the target word vector; The audio features are synthesized to generate the audio data.

4. The method according to claim 2, characterized in that, The method further includes: Determine the number of facial expression frames contained in the facial expression data; The image playback frame rate is determined, and the playback duration of the facial expression data is determined based on the number of facial expression frames and the image playback frame rate; the playback duration is the same as the audio duration of the audio data.

5. The method according to claim 1, characterized in that, The step of obtaining the action background image corresponding to the specified action based on the action identifier includes: Determine the action sequence contained in the action data; the action sequence includes one or more specified actions. The background image corresponding to each specified action is obtained from the preset action library based on the action identifier.

6. The method according to claim 1, characterized in that, The action data includes the current specified action, the previous specified action, and the next specified action. Determining the image arrangement order based on the comparison result further includes: Determine the current start frame identifier and the current end frame identifier corresponding to the currently specified action; The background image between the end frame identifier of the previous specified action and the current start frame identifier is determined as the default background image; The background image between the current start frame identifier and the current end frame identifier is determined as the action background image; The background image between the current end frame identifier and the start frame identifier of the next specified action is determined as the default background image.

7. The method according to claim 1, characterized in that, The step of determining the set of action images to be filtered from the plurality of action background images based on the number of filtered frames includes: The number of motion background images selected sequentially from the plurality of motion background images is the number of filter frames as the filter motion image set; or The number of action background images selected randomly from the plurality of action background images is the number of filter frames as the filter action image set.

8. The method according to claim 1, characterized in that, The step of performing image fusion processing on the face image and the background image to obtain live video frames includes: The background image that matches the face image is determined as the matching background image; Obtain a pre-built image fusion model; The face image and the matching background image are fused using the image fusion model to generate the live video frame.

9. A virtual video live streaming processing device, characterized in that, include: The data acquisition module is used to acquire text data and virtual object action data input by the user through the live streaming client during virtual video live streaming; the action data includes the action identifier of the specified action generated by the virtual object in the virtual video according to the user's personalized needs, and the start position identifier of the specified action in the text data. A facial image determination module is used to generate audio data and facial expression data of the virtual object based on the text data, and to generate a facial image of the virtual object based on the facial expression data; The background image determination module is used to generate a background image containing the specified action based on the start position identifier and the action identifier; An image fusion module is used to perform image fusion processing on the face image and the background image to obtain live video frames; A live video synthesis module is used to synthesize the live video frames and the audio data into a live video stream in real time; The background image determination module is further configured to perform the following steps: Get the default background image; Obtain the action background image corresponding to the specified action based on the action identifier; Determine the text length of the text data; Obtain the first number of facial images corresponding to the virtual object, where the first number is the number of background images, and according to the formula... Calculate the start frame identifier of the specified action in the background image, where n represents the start frame identifier, N represents the first quantity, I represents the start position identifier, and L represents the text length; The number of background images is determined as the number of motion images. An end frame identifier is determined based on the start frame identifier and the number of motion images. The end frame identifier is represented as n-1+f, where f represents the number of motion images. The image arrangement order of the default background image and the action background image is determined by comparing the size of the end frame identifier with the first quantity. Specifically, if the comparison result includes the end frame identifier being less than or equal to the first quantity, the background image before the start frame identifier is determined as the default background image, the background image between the start frame identifier and the end frame identifier is determined as the action background image, and the background image between the end frame identifier and the first quantity identifier is determined as the default background image. If the comparison result includes the end frame identifier being greater than the first quantity, the background image before the start frame identifier is configured as the default background image, and a number of filter frames is determined based on the first quantity and the start frame identifier. A set of filtered action images is determined from multiple action background images based on the number of filter frames, and the background image between the start frame identifier and the first quantity identifier is configured as the set of filtered action images. The default background image and the action background image are sorted according to the image arrangement order to generate a background image sequence containing the specified action.

10. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the virtual video live streaming processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the virtual video live streaming processing method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes computer instructions, which are stored in a computer-readable storage medium; When the processor of a computer device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the computer device to perform the virtual video live streaming processing method as described in any one of claims 1 to 8.

Images