Virtual character rendering method, device, equipment, storage medium and program product

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for rendering a virtual character. The method can be applied to real-time driving and animation rendering scenes, and the method includes: obtaining a real-time video screen including a target object; calling a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtaining real-time bone node data of the target object and facial pinching data of the target object; binding the virtual character model to the bone node of the target object, and driving the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone node, and controlling the facial expression changes of the virtual character in real time according to the facial pinching data, and rendering and displaying the virtual character. It can enhance the virtual experience of the target object, and open up the link of obtaining the real-time video screen of the target object in the Unreal Engine to drive the virtual character in real time and render the virtual character.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a virtual character method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of computer technology and Internet technology, virtual characters of various forms can bring people a variety of virtual experiences. Usually, virtual characters are driven by skeletal animation, which is produced and designed by artists. After the key frame data of skeletal animation is produced, the virtual character can be driven to move according to the preset behavior trajectory. This is an offline driving method. In recent years, there has also been a method of allowing users to wear motion capture devices to capture user movements to drive virtual characters in real time. The threshold for use is high, the price is expensive, and the usage scenarios are limited. In addition, the large motion capture devices worn usually make it difficult for users to show violent movements, resulting in a poor user experience. In addition, because the driven virtual character model is pre-set, its appearance and expression are relatively simple, resulting in a relatively simple virtual character experienced by each user and a poor virtual experience. Summary of the invention

[0003] Based on this, it is necessary to provide a virtual character rendering method, apparatus, computer device, computer-readable storage medium and computer program product that can reduce the cost of real-time driving of virtual characters and enhance the user's virtual character experience in response to the above technical problems.

[0004] In a first aspect, the present application provides a method for rendering a virtual character. The method comprises:

[0005] Acquire a real-time video image including a target object;

[0006] Calling a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object and facial pinching data corresponding to the target object;

[0007] Acquire a virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes of the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0008] In a second aspect, the present application also provides a virtual character rendering device. The device comprises:

[0009] An acquisition module, used to acquire a real-time video image including a target object;

[0010] A detection module, used to call a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtain real-time bone node data of the bone nodes of the target object, and facial pinching data corresponding to the target object;

[0011] A rendering module is used to obtain a virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes in facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0012] In one embodiment, the acquisition module is also used to call a video acquisition plug-in in the Unreal Engine, and request permission to call the camera through the camera call logic in the video acquisition plug-in that matches the current operating system. After obtaining permission to call the camera, the real-time video image including the target object captured by the camera is obtained. The video acquisition plug-in integrates the camera call logic of various operating systems.

[0013] In one embodiment, the detection module is also used to determine the original encoding format of the video data corresponding to the real-time video screen; when the original encoding format is the target encoding format processed by the video detection plug-in, the video detection is performed on the video data corresponding to the real-time video screen through the video detection plug-in; when the original encoding format is not the target encoding format processed by the video detection plug-in, the video data in the original encoding format is format-converted to obtain video data in the target encoding format, and video detection is performed on the video data in the target encoding format through the video detection plug-in.

[0014] In one embodiment, the detection module is further used to decode the video data in the original coding format to obtain decoded video data; and re-encode the decoded video data according to the target coding format to obtain video data in the target coding format.

[0015] In one embodiment, the detection module is also used to start a sub-thread in the Unreal Engine, where the sub-thread is a sub-thread created in the main thread, and the main thread is a thread used to obtain real-time video images and render and display virtual characters; the sub-thread is used to convert the video data in the original encoding format to obtain video data in the target encoding format.

[0016] In one embodiment, the detection module is also used to call a graphics drawing interface that matches the current operating system, input the video data in the original encoding format into a graphics processor, and perform format conversion on the video data in the original encoding format in the graphics processor to obtain video data in a target encoding format; after reading the video data in the target encoding format stored in the video memory into the internal memory, the video detection plug-in is used to perform video detection on the video data in the target encoding format.

[0017] In one embodiment, the detection module is also used to call a video detection plug-in in the Unreal Engine to segment a target object area from the real-time video screen, determine the position of each bone node of the target object in the target object area; and output the displacement information and rotation angle information of each bone node based on the position of the same bone node in adjacent real-time video screens.

[0018] In one embodiment, the rendering module is further used to obtain the character features corresponding to each preset virtual character model in the Unreal Engine; obtain the object features of the currently logged-in object as the object features of the target object; and determine the virtual character model that matches the target object from the preset virtual character models based on the degree of match between the character features and the object features.

[0019] In one embodiment, the rendering module is also used to bind the vertices of the virtual character model to at least one bone node of the target object; based on the weights of the bone nodes to which the vertices are bound, perform weighted summation on the real-time bone node data corresponding to each of the bound bone nodes to obtain real-time updated driving data of the vertices; and based on the driving data, drive the movement of the virtual character corresponding to the virtual character model in real time.

[0020] In one embodiment, the device also includes a facial data acquisition module, and the detection module is used to perform video detection on the real-time video screen by calling the video detection plug-in when the video detection plug-in is started, so as to obtain the facial pinching data corresponding to the target object; when the video detection plug-in is not started, obtain the facial pinching data about the target object edited in the facial editing interface.

[0021] In one embodiment, the real-time video screen is the original live screen of the target object; the device also includes: a live data transmission module, which is used to obtain the real scene where the target object is located in the original live screen; merge the rendered virtual character with the real scene to generate a virtual character live screen; and upload the virtual character live screen to the server and then push the video.

[0022] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0023] Acquire a real-time video image including a target object;

[0024] Calling a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object and facial pinching data corresponding to the target object;

[0025] Acquire a virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes of the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0026] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0027] Acquire a real-time video image including a target object;

[0028] Calling a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object and facial pinching data corresponding to the target object;

[0029] Acquire a virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes of the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0030] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0031] Acquire a real-time video image including a target object;

[0032] Calling a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object and facial pinching data corresponding to the target object;

[0033] Acquire a virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes of the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0034] The virtual character rendering method, device, computer equipment, storage medium and computer program product described above, the Unreal Engine can provide high-quality rendering effects, obtain the real-time video screen including the target object through the Unreal Engine, and directly call the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtain the real-time bone node data of the bone node of the target object and the facial pinching data corresponding to the target object. In this way, after binding the acquired virtual character model to the bone node of the target object in the Unreal Engine, the virtual character corresponding to the virtual character model can be driven to move in real time according to the real-time bone node data of the bone node, and the facial expression changes of the virtual character can be controlled in real time according to the real-time changing facial pinching data, so that the virtual character rendered and displayed in high quality and real time based on the Unreal Engine not only has high rendering quality, but also does not require the target object to wear additional motion capture equipment, and the rendered and displayed virtual character's movements and facial expression changes are synchronized with the target object in the real-time video screen, which improves the virtual experience of the target object, and also opens up the link of obtaining the real-time video screen of the target object in the Unreal Engine, outputting the real-time bone node data to drive the virtual character in real time, and rendering the virtual character with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A diagram of an application environment of a virtual character rendering method in an embodiment;

[0036] Figure 2 An overall framework for rendering a virtual character in one embodiment;

[0037] Figure 3 A schematic diagram of a process of rendering a virtual character in one embodiment;

[0038] Figure 4 A framework diagram for obtaining driving data of a virtual character in one embodiment;

[0039] Figure 5 A schematic diagram of a framework of video detection in one embodiment;

[0040] Figure 6 It is a schematic diagram of performing data preprocessing on video data with different encoding formats in one embodiment;

[0041] Figure 7 A schematic diagram of a skeleton node in an embodiment;

[0042] Figure 8 A schematic diagram of binding a virtual character model to a skeleton in one embodiment;

[0043] Fig. 9 A schematic diagram of a virtual character driving in one embodiment;

[0044] Fig.10 is a structural block diagram of a virtual character rendering device in one embodiment;

[0045] Fig.11 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The following are some concepts involved in this application:

[0048] Unreal Engine: An engine with high-quality rendering capabilities, such as UE4, is a game engine.

[0049] Plugins in Unreal Engine: For example, in UE4, plugins are collections of code and data that developers can enable or disable on a per-project basis in the editor. Plugins can add runtime game features, modify built-in engine functionality, create new file types, and extend the functionality of the editor with new menus, toolbar commands, and sub-modes. Many UE4 subsystems can be extended using plugins.

[0050] Editor: It is a visual operation tool for UE4. It integrates UE4 functions into a visual interface to facilitate users to edit quickly. It also integrates various tools and serves as a bridge for developers or users to use the engine.

[0051] Monocular camera: A camera with one camera.

[0052] Mesh: The mesh of a model. Usually a 3D model is made up of polygons. A complex polygon is actually made up of multiple triangular faces. The surface of a 3D model is made up of multiple interconnected triangular faces. The collection of vertices and edges that make up these triangular faces in three-dimensional space is a Mesh.

[0053] Skeleton: A concept abstracted by an animation algorithm, which is physically similar to bones and imitates the human skeleton to control the driving of virtual models.

[0054] Skeletal animation: Animation resources prepared in advance by artists to control the position transformation of bones and indirectly drive the position transformation of the bound Mesh vertices. It is usually used to drive more complex animations, such as running and jumping of characters.

[0055] Bone skinning: After selecting a bone, specify which vertices of the Mesh are driven by it and the corresponding drive weights.

[0056] Human-driven virtual characters in related technologies are either driven by offline data, which cannot achieve real-time driving, or driven by users wearing large motion capture devices. The method of capturing user movements to drive virtual characters in real time has a high threshold for use, is expensive, and has limited usage scenarios. Moreover, the large motion capture devices worn usually make it difficult for users to show violent movements, resulting in a poor user experience. In addition, the virtual characters in traditional game applications are controlled by artists to make skeletal animations to move according to preset trajectories and behaviors. The production cost is high, and its essence is also an offline driving method, which cannot achieve real-time control of the behavior of virtual characters. The present application provides a virtual character driving method, which opens up the link of obtaining real-time video images of the target object in the Unreal Engine, outputting real-time skeletal node data to drive the virtual character in real time, and rendering the virtual character with high quality.

[0057] The virtual character rendering method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other servers.

[0058] In one embodiment, the terminal 102 can directly obtain a real-time video screen including a target object through the Unreal Engine, call a video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtain real-time bone node data of the bone nodes of the target object, as well as facial pinching data corresponding to the target object; obtain a virtual character model provided by the Unreal Engine, bind the virtual character model to the bone nodes of the target object, and according to the real-time bone node data of the bone nodes, drive the movement of the virtual character corresponding to the virtual character model in real time, and according to the facial pinching data, control the changes in the facial expressions of the virtual character in real time, and render and display the virtual character.

[0059] In one embodiment, the terminal 102 is equipped with a camera, which can be a monocular camera or a binocular camera. The terminal 102 can call the camera in the Unreal Engine to shoot the target object and obtain a real-time video picture of the target object. In one embodiment, a target application is running on the terminal 120. In order to obtain a high-quality rendering effect, part or all of the graphics drawing in the target application can be implemented based on the Unreal Engine. The Unreal Engine can be, for example, UE4 or other high-quality game engines. The target application can be, for example, a game application, a live broadcast application, a short video application, a social application, etc. After the terminal 102 runs the target application, when the target application needs to realize the function of driving a virtual character based on the target object, the Unreal Engine can directly call the camera to shoot the target object to obtain a real-time video picture of the target object, call the video detection plug-in in the Unreal Engine to perform video detection on the real-time video picture, obtain the real-time bone node data of the bone node of the target object, and the facial pinching data corresponding to the target object, and drive the real-time virtual character based on the real-time bone node data and the real-time facial pinching data.

[0060] In one embodiment, the target application can also upload the virtual character screen rendered and displayed in real time to the server 104. For example, in a game scene, user A and user B interact with each other through a game application, and the terminal used by user A can upload the virtual character screen driven by user A in real time to the server 104, and the server 104 sends the virtual character screen to the terminal used by user B, or the server 104 updates the game scene screen according to the real-time virtual character screen corresponding to user A, and sends the updated game scene screen to the terminals used by user A and user B respectively. For another example, in a live broadcast scene, a user broadcasts a video through a live broadcast application, and the terminal used by user C collects the original live broadcast screen of user C, obtains the real scene where user C is in the original live broadcast screen, and merges the virtual character screen driven by user C in real time with the real scene to generate a virtual character live broadcast screen, and uploads the virtual character live broadcast screen to the server 104 for live video streaming.

[0061] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0062] like Figure 2 FIG. 1 is an overall framework of virtual character rendering in one embodiment. Figure 2First, it is necessary to obtain the video data corresponding to the real-time video screen. For video data with different encoding formats, it is necessary to preprocess the video data to obtain video data with an encoding format that is compatible with the video detection algorithm, and then perform video detection to drive the virtual character, and finally render and display the driven virtual character.

[0063] In one embodiment, Figure 3 As shown, a virtual character rendering method is provided, which can be executed by the terminal 102 or the server 104 alone, or by the terminal 102 and the server 104 in collaboration. For example, the server 104 can provide a virtual character model to the terminal. Figure 1 The terminal 102 in the example is used as an example to illustrate, and the following steps are included:

[0064] Step 302: Acquire a real-time video image including the target object.

[0065] Among them, the target object usually refers to the user, the virtual character mentioned below can be a virtual image such as a virtual person, a virtual animal or a virtual animation character, and the real-time video screen is a video screen including the target object collected in real time, and the actions and expressions made by the target object are presented through the real-time video screen.

[0066] In one embodiment, obtaining a real-time video image including a target object includes: calling a video acquisition plug-in in the Unreal Engine, requesting permission to call a camera through the camera call logic in the video acquisition plug-in that matches the current operating system, and after obtaining permission to call the camera, obtaining the real-time video image including the target object captured by the camera, wherein the video acquisition plug-in integrates the camera call logic of various operating systems.

[0067] In this embodiment, in order to achieve real-time driving of virtual characters while obtaining high-quality rendering effects, developers can integrate a video acquisition plug-in in the Unreal Engine, call the camera through the video acquisition plug-in, and obtain the real-time video picture including the target object captured by the camera. Among them, the camera can be a monocular camera or a computer eye on the terminal. The video acquisition plug-in integrates the camera call logic of various operating systems and can support various operating systems. The terminal can request permission to call the camera through the camera call logic matching the current operating system in the video acquisition plug-in, and after obtaining the permission to call the camera, obtain the real-time video picture including the target object captured by the camera.

[0068] In this embodiment, the real-time video images captured by the terminal initiate system permission acquisition through a unified video capture plug-in in the Unreal Engine. The plug-in encapsulates the calling logic of cameras and microphones of different operating systems. The user is unaware of the underlying implementation and only needs to simply call the plug-in to complete the permission acquisition and receive the data captured by the local audio and video devices in the Unreal Engine.

[0069] In a specific application scenario, the user is in front of the terminal camera, and the terminal is installed with a target application based on the Unreal Engine. After the target application is started, the camera is called through the video acquisition plug-in in the Unreal Engine to collect real-time images within the camera's shooting range. The user's actions and expressions made to the terminal camera will be collected in real time, thereby obtaining a real-time video image including the target object. The camera can be a front camera or a rear camera.

[0070] In one embodiment, the terminal can also receive real-time video images including the target object collected and transmitted by other computer devices through the network. Optionally, when the video acquisition plug-in in the Unreal Engine is activated, the real-time video images including the target object can be directly captured through the video acquisition plug-in, and when the video acquisition plug-in is not activated, the terminal can receive real-time video images including the target object collected and transmitted by other computer devices through the network.

[0071] Step 304, calling the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtaining the real-time bone node data of the bone nodes of the target object, and the facial pinching data corresponding to the target object.

[0072] In the embodiment of the present application, it is not necessary to drive the virtual character through skeletal animation or offline data. In order to achieve the real-time driving of the virtual character while obtaining high-quality rendering effects, the developer can integrate a video detection plug-in in the Unreal Engine. The terminal performs video detection on the real-time video screen by calling the video detection plug-in of the Unreal Engine to obtain the real-time skeletal node data of the skeletal node of the target object. In addition, in the embodiment of the present application, in order to improve the matching degree of the virtual character displayed by the rendering and the target object, improve the sense of reality of the rendering display, and avoid the problem that the face pinching data is set in advance for the virtual character model to bring the virtual experience to be very dull and single, the terminal can also obtain the real-time face pinching face data of the target object by using the video detection plug-in of the Unreal Engine for video detection, and drive and render the virtual character based on the real-time skeletal node data and the real-time face pinching face data in the Unreal Engine. In addition, the video detection plug-in is directly integrated in the Unreal Engine, which can be used by terminals or applications of different platforms and different operating systems without compatible platforms.

[0073] Among them, the target object's bone nodes can be joints that are highly correlated with the target object's movement determined by video detection. It can be understood that the target object's movement can be determined by the displacement and rotation angle of the bone nodes, so the real-time bone node data refers to the displacement and rotation angle information of each bone node determined by video detection. Similarly, the changes in the target object's facial appearance and facial expression can also be reflected by the virtual character.

[0074] Step 306, obtain the virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes of the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0075] The virtual character model is a model corresponding to the virtual character, also known as a virtual character three-dimensional mesh model, the surface of the model is composed of a plurality of mutually connected triangular faces, and the set of vertices and edges of the triangular faces can represent the model. The virtual character model can be a virtual character model for simulating a person, a virtual animal model for simulating an animal, or a virtual animation character model, etc.

[0076] In addition, the appearance of the virtual character model can be preset, for example, the hair, clothing, and accessories of the virtual character model can be set in advance, and the facial features of the virtual character model can also be set in advance, and when rendered and displayed, the rendering is updated through the above-mentioned facial pinching data. The virtual character model can also be a model with character features, and the character features of the virtual character model obtained by the terminal match the object features of the target object. It should be noted that the terminal can execute the steps of obtaining the virtual character model and binding the virtual character model to the bone nodes of the target object before performing video detection.

[0077] The terminal binds the virtual character model to the skeleton nodes of the target object, that is, records which skeleton nodes affect each vertex in the virtual character model and the corresponding influence weights, thereby establishing a linkage between the virtual character and the target object, so that the virtual character corresponding to the virtual character model can be driven to move according to the skeleton node data.

[0078] After the terminal binds the virtual character model to the bone nodes of the target object, it can drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes. At the same time, it can control the changes in the facial expressions of the virtual character in real time according to the facial pinching data. The high-quality rendering capabilities of the Unreal Engine are directly used to render and display the virtual character, and the virtual character can be presented with the same movement and expression changes as the target object.

[0079] It is understandable that since developers have integrated a video detection plug-in into the Unreal Engine, any application can implement a link in the Unreal Engine from acquiring real-time video footage of the target object, outputting real-time skeletal node data to drive the virtual character in real time, and rendering the virtual character with high quality.

[0080] In the above virtual character rendering method, the Unreal Engine can provide high-quality rendering effects, obtain the real-time video screen including the target object through the Unreal Engine, and directly call the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, so as to obtain the real-time bone node data of the bone node of the target object and the facial pinching data corresponding to the target object. In this way, after binding the acquired virtual character model to the bone node of the target object in the Unreal Engine, the virtual character corresponding to the virtual character model can be driven to move in real time according to the real-time bone node data of the bone node, and the facial expression changes of the virtual character can be controlled in real time according to the real-time changing facial pinching data, so that the virtual character rendered and displayed based on the Unreal Engine with high quality not only has high rendering quality, but also does not require the target object to wear additional motion capture equipment, and the rendered virtual character's movements and facial expression changes are synchronized with the target object in the real-time video screen, which improves the virtual experience of the target object, and also opens up the link of obtaining the real-time video screen of the target object in the Unreal Engine, outputting the real-time bone node data to drive the virtual character in real time, and rendering the virtual character with high quality.

[0081] In one embodiment, a video detection plug-in is called in an Unreal Engine to perform video detection on a real-time video screen, including: determining the original encoding format of video data corresponding to the real-time video screen; when the original encoding format is the target encoding format processed by the video detection plug-in, performing video detection on the video data corresponding to the real-time video screen through the video detection plug-in; when the original encoding format is not the target encoding format processed by the video detection plug-in, performing format conversion on the video data in the original encoding format to obtain video data in the target encoding format, and performing video detection on the video data in the target encoding format through the video detection plug-in.

[0082] Among them, the original encoding format is the encoding format of the video data obtained by the real-time video screen obtained by the terminal. It can be seen that the encoding method of the obtained video data is not unified due to different operating systems. When the real-time video screen is obtained by calling the video acquisition plug-in in the Unreal Engine to obtain the terminal's camera permission, and then the camera is acquired through the camera after obtaining the camera permission, the original encoding format at this time is matched with the current terminal operating system. When the real-time video screen is received by the terminal from other computer devices, the video file corresponding to the real-time video screen may be unified with the operating system of the computer device. Some video data adopts YUV encoding (a video data storage format, where Y represents the brightness of the color of a point, UV represents the brightness of the color of a point, U is the blue component, V is the red component, and YUV together describe the color of a point), and some video data adopts RGB encoding (a video data storage format, R represents the red component, G represents the green component, and B represents the blue component). The video data input for video detection is usually a unified target encoding format, such as RGB encoding. In order to realize cross-platform video detection, each platform can use the video detection plug-in to perform video detection. In some cases, it is necessary to preprocess the video data, that is, decode and then re-encode, convert the video encoding format, convert it to a target encoding format that can be uniformly processed by the video detection plug-in, and perform video detection on the video data in the target encoding format through the video detection plug-in.

[0083] like Figure 4 FIG. 1 is a schematic diagram showing a framework for obtaining driving data of a virtual character in one embodiment. Figure 4 The video data corresponding to the real-time video images captured by different operating system platforms, such as iOS, Android, Windows, etc., have different encoding formats, or the formats of video files transmitted in real time by other computer devices are different, such as MP4 files, WMV files or AVI files, etc. For the video data corresponding to the real-time video images collected by different platforms or the video files received in different formats, the video encoding format is converted to a target encoding format that can be uniformly processed by the video detection plug-in, and then video detection is performed to obtain real-time skeleton node data and facial pinching data.

[0084] Figure 5 FIG. 1 is a schematic diagram of a framework for optimizing video detection to obtain driving data in an embodiment. Figure 5 The video data in the original encoding format is input, and is preprocessed. The detection performance is accelerated during the preprocessing process to obtain the video data in the target encoding format required for video detection. After video detection, the real-time detection result is output, and the downstream virtual character driving module asynchronously reads the real-time detection result to drive the virtual character.

[0085] In one embodiment, format conversion is performed on video data in an original coding format to obtain video data in a target coding format, including: decoding the video data in the original coding format to obtain decoded video data; and re-encoding the decoded video data according to the target coding format to obtain video data in the target coding format.

[0086] When format conversion is required, the terminal needs to preprocess the video data in the original encoding format. The preprocessing includes first decoding to obtain decoded video data, and then re-encoding according to the target encoding format to obtain video data in the target encoding format.

[0087] Usually, decoding and re-encoding will affect other processes, such as the smoothness of video acquisition and virtual character rendering and display. For example, when converting YUV-encoded video data into RGB-encoded video data, the data source is accessed at the CPU logic layer, pixel by pixel is read, and the source-encoded pixel is input through the conversion function, and the target-encoded video data is output. This method will block the current logic thread during the conversion process, and subsequent logic will not be executed until the conversion is completed, affecting other processes.

[0088] To this end, this application also proposes a method to accelerate format conversion and improve format conversion performance. Figure 6 As shown, in one embodiment, data preprocessing is performed on video data of different coding formats, and format conversion is accelerated during the preprocessing process to obtain video data of a target coding format.

[0089] In one embodiment, video data in an original coding format is format-converted to obtain video data in a target coding format, including: starting a child thread in an Unreal Engine, the child thread is a child thread created in a main thread, the main thread is a thread for acquiring real-time video images and rendering and displaying virtual characters; and using the child thread to format-convert the video data in an original coding format to obtain video data in a target coding format.

[0090] In this embodiment, when it is necessary to start format conversion, a format conversion sub-thread is started. For example, a NEON instruction can be used to start the sub-thread when the format conversion is started. The sub-thread runs independently of the main thread for acquiring real-time video images and rendering and displaying virtual characters. In this way, these main threads will not be blocked by the sub-thread where the format conversion is located. When the sub-thread is completed, the main thread is notified to complete the format conversion of the video data.

[0091] In one embodiment, video data in an original coding format is format-converted to obtain video data in a target coding format, including: calling a graphics drawing interface that matches the current operating system, inputting the video data in the original coding format into a graphics processor, and format-converting the video data in the original coding format in the graphics processor to obtain video data in a target coding format; performing video detection on the video data in the target coding format through a video detection plug-in, including: after reading the video data in the target coding format stored in a video memory into a memory, performing video detection on the video data in the target coding format through the video detection plug-in.

[0092] In this embodiment, the terminal also includes a graphics processor, and the terminal performs format conversion through the graphics processor, i.e., GPU. For terminals with different operating systems, the graphics drawing interface, i.e., graphics API (Application Programming Interface) supported by the corresponding operating system can be used to call drawing functions. For example, the mainstream graphics APIs are: DirectX, OpenGL, and OpenGL ES. DirectX is supported by Windows system, OpenGL is supported by Windows, Linux, and Mac (Unix) system, and OpenGL ES is supported by iOS and Android system. The graphics API is called to input the video data in the original encoding format into the GPU, and the format conversion is performed in the GPU through the shader. Then, by reading the GPU texture (pixel format that can be recognized by the GPU), the converted data in the video memory is transferred back to the memory for subsequent video detection, which can realize large-scale parallel computing and greatly increase the efficiency of video data format conversion.

[0093] In one embodiment, a video detection plug-in is called in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object, including: calling the video detection plug-in in the Unreal Engine to segment the target object area from the real-time video screen, and determining the position of each bone node of the target object in the target object area; according to the position of the same bone node in adjacent real-time video screens, outputting the displacement information and rotation angle information of each bone node.

[0094] In an embodiment of the present application, video detection can be implemented based on a video detection algorithm of a neural network, and the video detection plug-in includes two parts of functions, one part is to detect the action of the target object, and the other part is to detect the facial expression of the target object. Optionally, the video detection plug-in can integrate a motion capture model based on a neural network, detect the action of the target object, determine the position information of each bone node of the target object in the real-time video screen, including displacement information and rotation angle information, and then output the displacement information and rotation angle information of each bone node based on the position of the same bone node in two adjacent frames of video.

[0095] like Figure 7 FIG. 1 is a schematic diagram of a skeleton node in an embodiment. Figure 7 , each bone node is represented by a small circle, and the position and angle changes of each bone node can reflect the movement of the target object.

[0096] In one embodiment, the video detection plug-in is also used to extract image features of the real-time video screen, compare the image features of the current real-time video screen with the image features of the previous frame or the image features of the previous adjacent multi-frame video screen, and determine the difference between the compared image features. If the difference is greater than a preset threshold, it means that the action of the target object may have changed significantly. At this time, it is necessary to calculate and update the displacement information and rotation angle information of each bone node in the current real-time video screen based on the position of the same bone node in the two adjacent frames of video screen; if the difference is less than the preset threshold, it means that the action of the target object may not have changed much. In order to avoid the problem of decreased rendering performance caused by frequent calculation and updating of the position information of the bone nodes, the detection video plug-in can directly use the position information corresponding to the previous or previous adjacent multi-frame video screen as the position information of each bone node in the current real-time video screen.

[0097] In addition, the video detection plug-in can also integrate a facial expression recognition model based on a neural network, detect the face area of ​​the target object from the real-time video screen, identify the facial key points of the target object, and determine the location of the facial key points, including eyebrows, eyes, nose, mouth, facial contours, etc., as real-time facial pinching data to improve the realism of the rendering display and avoid the problem that the pinching data is pre-set for the virtual character model, which brings a very dull and single virtual experience.

[0098] In one embodiment, obtaining a virtual character model includes: obtaining character features corresponding to each preset virtual character model in the Unreal Engine; obtaining object features of a currently logged-in object as object features of a target object; and determining a virtual character model that matches the target object from each preset virtual character model based on a degree of match between the character features and the object features.

[0099] In this embodiment, the virtual character corresponding to the virtual character model has corresponding character characteristics, such as: gender, height, weight, age, body shape, hairstyle, personality, voice, style, clothing, accessories, etc. In one embodiment, the user can use the terminal to register a user ID in the target application. After logging into the target application, the terminal can determine a virtual character model with a high degree of match with the currently logged-in user from a large number of preset or recommended virtual character models as the virtual character model for simulating the target object. In this way, the obtained virtual character model is more matched with the current target object, avoiding the problem of poor virtual experience when each user uses the same, single virtual character model. Of course, the terminal can also display a virtual character setting interface, and the user can select a virtual character model according to his or her preferences in the setting interface.

[0100] In one embodiment, a virtual character model is bound to a skeletal node of a target object, and the movement of the virtual character corresponding to the virtual character model is driven in real time according to the real-time skeletal node data of the skeletal node, including: binding a vertex of the virtual character model to at least one skeletal node of the target object; performing weighted summation of the real-time skeletal node data corresponding to each of the bound skeletal nodes according to the weight of the skeletal node to which the vertex is bound, to obtain real-time updated driving data of the vertex; and driving the movement of the virtual character corresponding to the virtual character model in real time based on the driving data.

[0101] Specifically, the terminal binds the vertices of the virtual character model to the bone nodes through the Unreal Engine, that is, records which bone nodes are affected by each vertex, and the corresponding influence weights. For example, if a vertex V in the virtual character model is affected by bone node 1, bone node 2, and bone node 3, then the vertex V is bound to bone node 1, bone node 2, and bone node 3 according to their respective influence weights. This process is called bone skinning, and based on the real-time bone node data of bone node 1, bone node 2, and bone node 3, weighted summation is performed according to the corresponding influence weights a, b, and c, and the position information of vertex V is updated, thereby driving the movement of the virtual character corresponding to the virtual character model, where the sum of weights a, b, and c should be 1.

[0102] Figure 8 FIG. 1 is a schematic diagram of binding a virtual character model to a skeleton in one embodiment. Figure 8 After obtaining the skeleton nodes of the virtual character model and the target object, perform skeleton skinning, referring to Figure 8For example, vertex V1 is bound to bone node 1 and bone node 2, with weights a and b respectively; vertex V2 is bound to bone node 1 and bone node 3, with weights c and d respectively; vertex V3 is bound to bone nodes 4, 5, and 6, with weights e, f, and g respectively. Similarly, each vertex in the virtual character model is bound to a bone node.

[0103] like Fig. 9 FIG. 1 is a schematic diagram of a virtual character driving in one embodiment. Fig. 9 After binding the virtual character model to the skeleton in Unreal Engine, the driving data is input into the virtual character animation system of Unreal Engine. The driving data includes the skeleton node data and the facial pinching data, which are used to calculate the position information of the vertices of the virtual character model after driving, so as to update the vertex position of the virtual character model, and then render and display the driven virtual character.

[0104] In one embodiment, the above method also includes: when the video detection plug-in is started, the facial pinching data corresponding to the target object is obtained by executing the step of calling the video detection plug-in to perform video detection on the real-time video screen; when the video detection plug-in is not started, the facial pinching data of the target object edited in the facial editing interface is obtained.

[0105] In an embodiment of the present application, when a video detection plug-in is started in the Unreal Engine, the video detection plug-in can be called to perform facial key point recognition on a target object in a real-time video screen to obtain facial pinching data of the target object. When the video detection plug-in is disabled in the Unreal Engine, a method can be provided for users to edit the facial pinching data by themselves to obtain facial pinching data about the target object for rendering and display. This can also reduce the computing pressure of the Unreal Engine and avoid the problem that the facial pinching data is pre-set for the virtual character model, which brings a very dull and single virtual experience.

[0106] In one embodiment, the real-time video picture is the original live picture of the target object; the above method also includes: obtaining the real scene where the target object is located in the original live picture; integrating the rendered virtual character with the real scene to generate a virtual character live picture; uploading the virtual character live picture to the server and then pushing the video.

[0107] In this embodiment, the original live broadcast picture is a collected live broadcast picture of the target object. When you do not want the real person to be off-screen or to increase the fun of the live broadcast, you can use the virtual character rendering method provided in the embodiment of the present application to drive the virtual character based on the action of the target object in the original live broadcast picture, and merge the rendered and displayed virtual character with the real scene where the target object is located to generate a real-time updated virtual character live broadcast picture, which is uploaded to the video live broadcast server for video push. In this way, the target object is replaced by a virtual character for video live broadcast, and its actions and expressions are consistent with the target object. On the one hand, it can reduce the appearance of real people, and on the other hand, it can provide a live broadcast method that combines the virtual and the real, enriching the live broadcast format.

[0108] In one embodiment, the real-time video picture is the original action picture of the target object, and the above method also includes: obtaining a virtual game scene; integrating the rendered virtual character with the virtual game scene to generate an updated virtual game scene picture; uploading the virtual game scene picture to the server and then sending it to the device used by the game participants.

[0109] Specifically, the target object can interact with the game by making gestures, and the screen of the target object when making game gestures is collected to form a real-time video screen. Through the virtual character rendering method provided in the embodiment of the present application, a virtual character consistent with the gesture of the target object is rendered, and the virtual character is merged with the virtual game scene to form a real-time updated virtual game scene screen, which is uploaded to the server and sent to the terminal used by the game participants. The virtual character can be a virtual character provided by the game. In this way, the target object controls the virtual character to participate in the game, providing a game mode that combines virtual and real, and the user has a better immersive game experience. The actions and expressions of the virtual character when participating in the game process are driven in real time based on the target object, without losing personal characteristics, providing a more realistic game experience for users participating in the game.

[0110] In a specific embodiment, the method for rendering and displaying a virtual character may be executed by a terminal, and includes the following steps:

[0111] 1. Call the video acquisition plug-in in Unreal Engine. The video acquisition plug-in integrates the camera call logic of various operating systems;

[0112] 2. Request permission to call the camera through the camera call logic in the video acquisition plug-in that matches the current operating system. After obtaining permission to call the camera, obtain the real-time video image of the target object captured by the camera;

[0113] 3. Determine the original encoding format of the video data corresponding to the real-time video screen; when the original encoding format is the target encoding format processed by the video detection plug-in, perform video detection on the video data corresponding to the real-time video screen through the video detection plug-in;

[0114] 4. When the original encoding format is not the target encoding format processed by the video detection plug-in, the video data in the original encoding format is converted into the video data in the target encoding format;

[0115] 5. Call the video detection plug-in in the Unreal Engine to segment the target object area from the real-time video screen, determine the position of each bone node of the target object in the target object area, output the displacement information and rotation angle information of each bone node according to the position of the same bone node in the adjacent real-time video screen, and call the video detection plug-in in the Unreal Engine to identify the position of the key points of the face of the target object and obtain the facial pinching data corresponding to the target object;

[0116] 6. Obtain the role features corresponding to each preset virtual role model in the Unreal Engine; obtain the object features of the currently logged-in object as the object features of the target object; based on the matching degree between the role features and the object features, determine the virtual role model that matches the target object from each preset virtual role model;

[0117] 7. Bind the vertices of the virtual character model to at least one bone node of the target object; perform weighted summation on the real-time bone node data corresponding to each of the bound bone nodes according to the weights of the bone nodes to which the vertices are bound, and obtain real-time updated driving data of the vertices; based on the driving data, drive the movement of the virtual character corresponding to the virtual character model in real time;

[0118] 8. According to the facial pinching data, the facial expression changes of the virtual character are controlled in real time, and the virtual character is rendered and displayed;

[0119] 9. Obtain the real live scene where the target object is located in the real-time video screen; merge the rendered virtual character with the real live scene to generate a virtual character live screen; upload the virtual character live screen to the server and then push the video.

[0120] In the present embodiment, a video acquisition plug-in is integrated in the Unreal Engine, and the camera is called by the video acquisition plug-in to obtain the real-time video picture including the target object captured by the camera, so that the real-time driving of the virtual character can be realized while obtaining high-quality rendering effect. The video detection plug-in is directly called in the Unreal Engine to perform video detection on the real-time video picture, and the real-time bone node data of the bone node of the target object and the face pinching data corresponding to the target object can be obtained, without the target object wearing an additional motion capture device, and without the need to drive the virtual character by bone animation or offline data, while obtaining high-quality rendering effect, the real-time driving of the virtual character is realized, and the matching degree of the virtual character and the target object of the rendering display is also improved, and the sense of reality of the rendering display is improved, so as to avoid the problem that the pinching face data is set in advance for the virtual character model to bring virtual experience very dull and single. In addition, cross-platform video detection can be realized by format conversion, and each platform can use the video detection plug-in for video detection. The movements and facial expressions of the rendered virtual character are synchronized with the target object in the real-time video, which enhances the virtual experience of the target object. It also opens up a link for obtaining real-time video footage of the target object in the Unreal Engine, outputting real-time skeletal node data to drive the virtual character in real time, and rendering the virtual character with high quality.

[0121] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] Based on the same inventive concept, the embodiment of the present application also provides a virtual character rendering device for implementing the virtual character rendering method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more virtual character rendering device embodiments provided below can refer to the limitations of the virtual character rendering method above, and will not be repeated here.

[0123] In one embodiment, Fig.10 As shown, a virtual character rendering device 1000 is provided, comprising: an acquisition module 1002, a detection module 1004 and a rendering module 1006, wherein:

[0124] An acquisition module 1002 is used to acquire a real-time video image including a target object;

[0125] The detection module 1004 is used to call the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtain the real-time bone node data of the bone nodes of the target object, and the facial pinching data corresponding to the target object;

[0126] The rendering module 1006 is used to obtain the virtual character model, bind the virtual character model to the bone nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time bone node data of the bone nodes, control the changes in the facial expressions of the virtual character in real time according to the facial pinching data, and render and display the virtual character.

[0127] In one embodiment, the acquisition module 1002 is also used to call a video acquisition plug-in in the Unreal Engine, which integrates the camera calling logic of various operating systems; through the camera calling logic in the video acquisition plug-in that matches the current operating system, request permission to call the camera, and after obtaining permission to call the camera, obtain the real-time video image including the target object captured by the camera.

[0128] In one embodiment, the detection module 1004 is also used to determine the original encoding format of the video data corresponding to the real-time video screen; when the original encoding format is the target encoding format processed by the video detection plug-in, the video detection is performed on the video data corresponding to the real-time video screen through the video detection plug-in; when the original encoding format is not the target encoding format processed by the video detection plug-in, the video data in the original encoding format is format-converted to obtain video data in the target encoding format, and video detection is performed on the video data in the target encoding format through the video detection plug-in.

[0129] In one embodiment, the detection module 1004 is further configured to decode the video data in the original coding format to obtain decoded video data; and re-encode the decoded video data in the target coding format to obtain video data in the target coding format.

[0130] In one embodiment, the detection module 1004 is also used to start a child thread in the Unreal Engine, where the child thread is created in the main thread, and the main thread is a thread used to obtain real-time video images and render and display virtual characters; the child thread is used to convert the video data in the original encoding format to obtain video data in the target encoding format.

[0131] In one embodiment, the detection module 1004 is also used to call a graphics drawing interface that matches the current operating system, input the video data in the original encoding format into the graphics processor, and perform format conversion on the video data in the original encoding format in the graphics processor to obtain video data in the target encoding format; after reading the video data in the target encoding format stored in the video memory into the internal memory, video detection is performed on the video data in the target encoding format through a video detection plug-in.

[0132] In one embodiment, the detection module 1004 is also used to call a video detection plug-in in the Unreal Engine to segment a target object area from a real-time video screen, determine the position of each bone node of the target object in the target object area; and output the displacement information and rotation angle information of each bone node based on the position of the same bone node in adjacent real-time video screens.

[0133] In one embodiment, the rendering module 1006 is further used to obtain the character features corresponding to each preset virtual character model in the Unreal Engine; obtain the object features of the currently logged-in object as the object features of the target object; and determine the virtual character model that matches the target object from each preset virtual character model based on the matching degree between the character features and the object features.

[0134] In one embodiment, the rendering module 1006 is also used to bind the vertices of the virtual character model to at least one bone node of the target object; based on the weights of the bone nodes to which the vertices are bound, weighted summation is performed on the real-time bone node data corresponding to each of the bound bone nodes to obtain real-time updated driving data of the vertices; and based on the driving data, the virtual character movement corresponding to the virtual character model is driven in real time.

[0135] In one embodiment, the device 1000 also includes a facial data acquisition module, and the detection module 1004 is used to perform video detection on the real-time video screen by calling the video detection plug-in when the video detection plug-in is started, so as to obtain the facial pinching data corresponding to the target object; when the video detection plug-in is not started, the facial pinching data of the target object edited in the facial editing interface is obtained.

[0136] In one embodiment, the real-time video screen is the original live screen of the target object; the device also includes: a live data transmission module, which is used to obtain the real scene where the target object is located in the original live screen; merge the rendered virtual character with the real scene to generate a virtual character live screen; upload the virtual character live screen to the server and then push the video.

[0137] The virtual character rendering device 1000 can provide high-quality rendering effects through the Unreal Engine. The real-time video screen including the target object is obtained through the Unreal Engine, and the video detection plug-in is directly called in the Unreal Engine to perform video detection on the real-time video screen, so that the real-time bone node data of the bone node of the target object and the facial pinching data corresponding to the target object can be obtained. In this way, after the obtained virtual character model is bound to the bone node of the target object in the Unreal Engine, the virtual character corresponding to the virtual character model can be driven to move in real time according to the real-time bone node data of the bone node, and the facial expression changes of the virtual character can be controlled in real time according to the real-time changing facial pinching data. Therefore, the virtual character rendered and displayed based on the Unreal Engine with high quality not only has high rendering quality, but also does not require the target object to wear additional motion capture equipment, and the rendered virtual character's movements and facial expression changes are synchronized with the target object in the real-time video screen, which improves the virtual experience of the target object, and also opens up the link of obtaining the real-time video screen of the target object in the Unreal Engine, outputting the real-time bone node data to drive the virtual character in real time, and rendering the virtual character with high quality.

[0138] Each module in the above virtual character rendering device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0139] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. This embodiment is described by taking the computer device as a terminal. Its internal structure diagram may be as follows: Fig.11As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, a camera and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a virtual character rendering method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc. The camera of the computer device can be a monocular camera, a binocular camera, etc., and can be a terminal camera, a computer camera, a security camera or an intelligent network camera, etc.

[0140] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0141] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the virtual character rendering method described in any of the above embodiments is implemented.

[0142] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the virtual character rendering method described in any of the above embodiments is implemented.

[0143] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the virtual character rendering method described in any of the above embodiments.

[0144] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A virtual character rendering method, characterized in that: The method comprises: Running a target application based on Unreal Engine, wherein the Unreal Engine integrates a video acquisition plug-in and a video detection plug-in, wherein the video acquisition plug-in integrates camera call logic of various operating systems; The Unreal Engine calls the camera calling logic in the video acquisition plug-in that matches the current operating system to request permission to call the camera, and after obtaining the permission to call the camera, obtains a real-time video picture of the target object collected by the camera, wherein the real-time video picture is a live video picture; Calling the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen to obtain real-time bone node data of the bone nodes of the target object and facial pinching data corresponding to the target object; Acquire a virtual character model, match the character features of the virtual character model with the object features of the currently logged-in object of the target application, bind the virtual character model to the skeleton nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time skeleton node data of the skeleton nodes, control the facial expression changes of the virtual character in real time according to the facial pinching data, and render and display the virtual character; The rendered virtual character is integrated with the real scene where the target object is located in the real-time video screen to generate a real-time updated virtual character live screen, and the virtual character live screen is uploaded to the server for video push.

2. The method according to claim 1, characterized in that: The calling of the video detection plug-in in the Unreal Engine to perform video detection on the real-time video image includes: Determine the original encoding format of the video data corresponding to the real-time video picture; When the original encoding format is the target encoding format processed by the video detection plug-in, the video detection plug-in is used to perform video detection on the video data corresponding to the real-time video picture; When the original encoding format is not the target encoding format processed by the video detection plug-in, the video data in the original encoding format is format-converted to obtain video data in the target encoding format, and video detection is performed on the video data in the target encoding format through the video detection plug-in.

3. The method according to claim 2, characterized in that The step of converting the video data in the original coding format to obtain the video data in the target coding format includes: Decoding the video data in the original encoding format to obtain decoded video data; The decoded video data is re-encoded according to the target coding format to obtain video data in the target coding format.

4. The method according to claim 2, characterized in that: The step of converting the video data in the original coding format to obtain the video data in the target coding format includes: Starting a sub-thread in the Unreal Engine, wherein the sub-thread is a sub-thread created in the main thread, and the main thread is a thread for acquiring real-time video images and rendering and displaying virtual characters; The subthread is used to perform format conversion on the video data in the original coding format to obtain video data in the target coding format.

5. The method according to claim 2, characterized in that: The step of converting the video data in the original coding format to obtain the video data in the target coding format includes: Calling a graphics drawing interface that matches the current operating system, inputting the video data in the original encoding format into a graphics processor, and converting the video data in the original encoding format in the graphics processor to obtain video data in a target encoding format; The performing video detection on the video data in the target coding format by the video detection plug-in includes: After the video data in the target coding format stored in the video memory is read into the internal memory, the video detection plug-in is used to perform video detection on the video data in the target coding format.

6. The method according to claim 1, characterized in that The step of calling the video detection plug-in in the Unreal Engine to perform video detection on the real-time video image to obtain real-time bone node data of the bone nodes of the target object includes: Calling a video detection plug-in in the Unreal Engine to segment a target object area from the real-time video screen, and determining the position of each bone node of the target object in the target object area; According to the position of the same bone node in adjacent real-time video frames, the displacement information and rotation angle information of each bone node are output.

7. The method according to claim 1, characterized in that The step of obtaining the virtual character model comprises: Obtaining the role features corresponding to each preset virtual role model in the Unreal Engine; Obtaining the object characteristics of the current login object as the object characteristics of the target object; Based on the matching degree between the character features and the object features, a virtual character model matching the target object is determined from the preset virtual character models.

8. The method according to claim 1, characterized in that: The step of binding the virtual character model to the skeleton nodes of the target object and driving the movement of the virtual character corresponding to the virtual character model in real time according to the real-time skeleton node data of the skeleton nodes comprises: Binding the vertices of the virtual character model to at least one bone node of the target object; According to the weights of the bone nodes bound to the vertices, weighted summation is performed on the real-time bone node data corresponding to the bound bone nodes to obtain the driving data of the vertices updated in real time; Based on the driving data, the virtual character corresponding to the virtual character model is driven to move in real time.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: When the video detection plug-in is started, the facial pinching data corresponding to the target object is obtained by executing the step of calling the video detection plug-in to perform video detection on the real-time video picture; When the video detection plug-in is not started, the facial pinching data about the target object edited in the facial editing interface is obtained.

10. A virtual character rendering device, characterized in that: The device comprises: An acquisition module is used to run a target application based on the Unreal Engine, wherein the Unreal Engine integrates a video acquisition plug-in and a video detection plug-in, wherein the video acquisition plug-in integrates the camera call logic of various operating systems, and the Unreal Engine calls the camera call logic in the video acquisition plug-in that matches the current operating system to request permission to call the camera, and after obtaining the permission to call the camera, obtains a real-time video picture of the target object collected by the camera, wherein the real-time video picture is a live video picture; A detection module, used to call the video detection plug-in in the Unreal Engine to perform video detection on the real-time video screen, and obtain real-time bone node data of the bone nodes of the target object, and facial pinching data corresponding to the target object; A rendering module is used to obtain a virtual character model, match the character features of the virtual character model with the object features of the current login object of the target application, bind the virtual character model to the skeleton nodes of the target object, drive the movement of the virtual character corresponding to the virtual character model in real time according to the real-time skeleton node data of the skeleton nodes, control the facial expression changes of the virtual character in real time according to the facial pinching data, and render and display the virtual character; The live data transmission module is used to merge the rendered virtual character with the real scene where the target object is located in the real-time video screen, generate a real-time updated virtual character live screen, upload the virtual character live screen to the server and then push the video.

11. The device according to claim 10, characterized in that The detection module is also used to determine the original encoding format of the video data corresponding to the real-time video screen; when the original encoding format is the target encoding format processed by the video detection plug-in, the video detection is performed on the video data corresponding to the real-time video screen through the video detection plug-in; when the original encoding format is not the target encoding format processed by the video detection plug-in, the video data in the original encoding format is format-converted to obtain video data in the target encoding format, and the video detection is performed on the video data in the target encoding format through the video detection plug-in.

12. The device according to claim 11, characterized in that The detection module is further used to decode the video data in the original coding format to obtain decoded video data; and re-encode the decoded video data according to the target coding format to obtain video data in the target coding format.

13. The device according to claim 11, characterized in that The detection module is also used to start a sub-thread in the Unreal Engine, where the sub-thread is a sub-thread created in the main thread, and the main thread is a thread used to obtain real-time video images and render and display virtual characters; the sub-thread is used to convert the video data in the original encoding format to obtain video data in the target encoding format.

14. The device according to claim 11, characterized in that The detection module is further used to call a graphics drawing interface that matches the current operating system, input the video data in the original coding format into a graphics processor, and perform format conversion on the video data in the original coding format in the graphics processor to obtain video data in a target coding format; After the video data in the target coding format stored in the video memory is read into the internal memory, the video detection plug-in is used to perform video detection on the video data in the target coding format.

15. The device according to claim 10, characterized in that The detection module is also used to call a video detection plug-in in the Unreal Engine to segment a target object area from the real-time video screen, determine the position of each bone node of the target object in the target object area; and output the displacement information and rotation angle information of each bone node based on the position of the same bone node in adjacent real-time video screens.

16. The device according to claim 10, characterized in that The rendering module is further used to obtain the character features corresponding to each preset virtual character model in the Unreal Engine; obtain the object features of the currently logged-in object as the object features of the target object; and determine the virtual character model that matches the target object from the preset virtual character models based on the matching degree between the character features and the object features.

17. The device according to claim 10, characterized in that The rendering module is further used to bind the vertices of the virtual character model to at least one bone node of the target object; and perform weighted summation on the real-time bone node data corresponding to each of the bound bone nodes according to the weight of the bone node to which the vertex is bound, so as to obtain the driving data of the vertex updated in real time; Based on the driving data, the virtual character corresponding to the virtual character model is driven to move in real time.

18. The device according to any one of claims 10 to 17, characterized in that The device also includes a facial data acquisition module; The detection module is used for obtaining the facial pinching data corresponding to the target object by executing the step of calling the video detection plug-in to perform video detection on the real-time video screen when the video detection plug-in is started; The facial data acquisition module is used to acquire the facial pinching data of the target object edited in the facial editing interface when the video detection plug-in is not started.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

20. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images