Virtual image live broadcast method and device

Abstract

The embodiment of the invention provides a virtual image live broadcast method and device, and relates to the technical field of live broadcast. The method comprises the following steps: acquiring an emoticon gift given by a target user; determining expression data for driving a virtual image according to the expression package gift; the virtual image is rendered based on the expression data, a virtual image expression picture with a corresponding expression is obtained, and the corresponding expression is an expression matched with the expression package gift; and performing image synthesis processing on the virtual image expression picture and the corresponding live video stream to obtain a target live stream for a user side to decode and play. According to the technical scheme of the embodiment of the invention, real-time interaction between the expression package gift and the virtual image can be realized, the sense of participation of audiences is improved, and live broadcast interaction is more vivid and interesting.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of live broadcast technology, and in particular, to a virtual image live broadcast method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of the online live broadcast industry, using virtual images as anchors for live broadcasts has become a new way of live broadcasting.

[0003] However, during the live broadcast of existing virtual images, if a user interacts with the virtual image, the virtual image generally relies on a fixed expression library or preset animations to interact with the user, and the form of interaction is relatively simple.

[0004] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the invention

[0005] The embodiments of the present application provide a virtual image live broadcast method, apparatus, computer equipment, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0006] One aspect of an embodiment of the present application provides a virtual image live broadcast method for a server, the method comprising: Get the emoji gifts sent by the target user; Determining expression data for driving the virtual image according to the expression package gift; Rendering the virtual image based on the expression data to obtain an expression picture of the virtual image with a corresponding expression, wherein the corresponding expression is an expression that matches the gift in the expression package; and The virtual image expression screen and the corresponding live video stream are subjected to image synthesis processing to obtain a target live stream for decoding and playback by a user terminal.

[0007] Optionally, determining expression data for driving the virtual image according to the expression package gift includes: Performing expression recognition on the expression package gift to obtain expression category information; Detecting whether there is a target expression matching the expression category information in a preset expression library; When the target expression exists in the expression library, expression data associated with the target expression is acquired.

[0008] Optionally, when the target expression exists in the expression library, acquiring expression data associated with the target expression includes: Searching whether there is expression data associated with the target expression in the expression library; In the case where expression data associated with the target expression exists in the expression library, the expression data is acquired from the expression library.

[0009] Optionally, the method further comprises: When the target expression does not exist in the expression library, the expression data is generated based on the target expression.

[0010] Optionally, when the target expression does not exist in the expression library, generating the expression data based on the target expression includes: When the target expression does not exist in the expression library, extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

[0011] Optionally, determining expression data for driving the virtual image according to the expression package gift includes: Extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

[0012] Another aspect of the embodiments of the present application provides a virtual image live broadcast method for a user terminal, the method comprising: In response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server; Obtaining a target live stream from the server, wherein the target live stream is generated based on the virtual image live broadcast method described above; The target live stream is decoded and played to display a virtual image expression screen with corresponding expressions on the live screen, where the corresponding expressions are expressions that match the expression package gift.

[0013] Optionally, in response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server includes: Displaying an emoticon package gift panel, wherein the emoticon package gift panel includes a default emoticon package control and / or a custom emoticon package control; In response to the target user triggering an emoji package gift giving operation based on the default emoji package control and / or the custom emoji package control, the emoji package gift is uploaded to a server.

[0014] Optionally, the method further comprises: In response to the downloading operation of the virtual image expression picture, the virtual image expression picture is saved as a preset expression package; or In response to a sharing operation on the virtual image expression picture, the virtual image expression picture is shared to a target platform.

[0015] Another aspect of an embodiment of the present application provides a virtual image live broadcast device for a server, the device comprising: The acquisition module is used to obtain the emoji gifts sent by the target user; A determination module, used to determine expression data for driving a virtual image according to the expression package gift; A rendering module, used for rendering the virtual image based on the expression data to obtain an expression picture of the virtual image with a corresponding expression, wherein the corresponding expression is an expression matching the gift in the expression package; and The synthesis module is used to perform image synthesis processing on the virtual image expression screen and the corresponding live video stream to obtain a target live stream for decoding and playback by the user end.

[0016] Another aspect of the embodiment of the present application provides a virtual image live broadcast device for a user terminal, the device comprising: An uploading module, used for uploading the emoticon gift to a server in response to an emoticon gift giving operation triggered by a target user; An acquisition module, used for acquiring a target live stream from the server, wherein the target live stream is generated based on the above-mentioned virtual image live broadcast method; The playback module is used to decode and play the target live stream to display a virtual image expression screen with corresponding expressions on the live screen, where the corresponding expressions are expressions that match the expression package gift.

[0017] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0018] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0019] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processor.

[0020] The above technical solution adopted in the embodiment of the present application may include the following advantages: First, after obtaining the emoji gift presented by the user, the expression data for driving the virtual image is generated based on the emoji gift. Afterwards, the virtual image is rendered by the expression data to generate a virtual image expression screen with corresponding expressions, so that the virtual image can display expressions that match the emoji gift. Finally, the target live stream is obtained by performing image synthesis processing on the virtual image expression screen and the corresponding live video stream, so that the user end can decode and play the target live stream, thereby displaying the changes in the expression of the virtual image after the user presents the emoji gift on the user end. As a result, real-time interaction with the virtual image through the emoji gift is achieved, which enhances the audience's sense of participation and makes the live interaction more vivid and interesting. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0021] Figure 1 The operating environment diagram of the virtual image live broadcast method according to the first embodiment of the present application is schematically shown; Figure 2 The flowchart of the virtual image live broadcast method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows Figure 2 Flow chart of sub-steps of step S202; Figure 4 Schematically shows Figure 3 Flow chart of sub-steps of step S304; Figure 5 A detailed flow chart schematically shows the steps of generating the expression data based on the target expression when the target expression does not exist in the expression library; Figure 6 Schematically shows Figure 2 Another sub-step flow chart of step S202; Figure 7 The flowchart of the virtual image live broadcast method according to the second embodiment of the present application is schematically shown; Figure 8 Schematically shows Figure 7 Flow chart of sub-steps of step S700; Fig. 9 A schematic diagram of the emoji gift panel is schematically shown; Fig.10 A diagram schematically shows an application example of a virtual image live broadcast method according to an embodiment of the present application; Fig.11 A block diagram of a virtual image live broadcast device according to Embodiment 3 of the present application is schematically shown; Fig.12 A block diagram of a virtual image live broadcast device according to Embodiment 4 of the present application is schematically shown; and Fig.13 The hardware architecture diagram of the computer device according to the fifth embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0023] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0024] In the description of the present application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as a limitation on the present application.

[0025] First, the following terms are explained: Skeleton-driven technology: A technology widely used in computer graphics, animation production and game development, which realizes the movement of characters or objects by using an internal bone structure to manipulate the character model.

[0026] Mixed deformation technology: also known as "blended shape deformer", can deform the "base mesh" into the "target shape" through vertex interpolation operations.

[0027] Secondly, in order to facilitate those skilled in the art to understand the technical solutions provided in the embodiments of the present application, the relevant technologies are described below: With the development of the online live broadcast industry, using virtual images as anchors for live broadcasts has become a new way of live broadcasting.

[0028] However, during the live broadcast of existing virtual images, if a user interacts with the virtual image, the virtual image generally relies on a fixed expression library or preset animations to interact with the user, and the form of interaction is relatively simple.

[0029] To this end, the embodiment of the present application provides a virtual image live broadcast technical solution. In this technical solution, the following effects can be achieved: (1) through the real-time interaction between the emoticon package gifts and the virtual image expressions, the audience's sense of participation is enhanced, making the live broadcast interaction more vivid and interesting; (2) different virtual image expressions are displayed through different types of emoticon package gifts, which improves the diversity of interaction; (3) users can save and share the generated virtual image expressions, promote the dissemination of live broadcast content, and increase the exposure rate of the live broadcast room; (4) provide users with more personalized interaction options, meet the audience's needs for emotional expression, and enhance the personalized characteristics of live broadcast. See below for details.

[0030] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0031] like Figure 1 As shown, the operating environment diagram includes: server 2, network 4, and client 6, wherein: Server 2 may be comprised of a single or multiple computing devices. The multiple computing devices may include virtualized computing instances. Virtualized computing instances may include virtual machines, such as simulations of computer systems, operating systems, servers, and the like. A computing device may load a virtual machine based on a virtual image and / or other data defining specific software (e.g., operating system, dedicated application, server) for simulation. As the demand for different types of processing services changes, different virtual machines may be loaded and / or terminated on one or more computing devices. A hypervisor may be implemented to manage the use of different virtual machines on the same computing device.

[0032] The server 2 may be configured to communicate with a client 6 or the like via a network 4. The network 4 includes various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, and / or the like. The network 4 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, or wireless links, such as cellular links, satellite links, Wi-Fi links, and the like.

[0033] Server 2 can provide storage, reading, writing, querying, deleting and other services, such as providing material storage, special effects processing, video generation and other services for the client.

[0034] The user terminal 6 may be an electronic device running an operating system such as Windows, Android™ or iOS, such as a smart phone, a tablet device, a laptop computer, a virtual reality device, a gaming device, a set-top box, a vehicle terminal, or a smart TV. Based on the above operating system, various application programs may be run, such as a virtual image live broadcast program.

[0035] The client 6 can provide / configure a user access page for manipulating the server 2 or uploading an object, etc.

[0036] It should be noted that the above devices are exemplary, and the number and type of devices are adjustable in different scenarios or according to different needs.

[0037] The following describes the technical solution of the present application through multiple embodiments, taking the user terminal 6 or the server 2 as the execution subject. It should be noted that these embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments described here.

[0038] Embodiment 1 This method embodiment may be executed in the server 2 .

[0039] Figure 2 The flowchart of the virtual image live broadcast method according to the first embodiment of the present application is schematically shown.

[0040] like Figure 2 As shown, the virtual image live broadcast method may include steps S200 to S206, wherein: Step S200, obtaining the emoticon gift sent by the target user.

[0041] Step S202, determining expression data for driving the virtual image according to the expression package gift.

[0042] Step S204, rendering the virtual image based on the expression data to obtain a virtual image expression screen with a corresponding expression, wherein the corresponding expression is an expression that matches the gift in the expression package.

[0043] Step S206, performing image synthesis processing on the virtual image expression screen and the corresponding live video stream to obtain a target live stream for decoding and playback by the user end.

[0044] The virtual image live broadcast method provided in this embodiment first generates expression data for driving the virtual image based on the expression package gift after obtaining the expression package gift presented by the user. Afterwards, the virtual image is rendered using the expression data to generate a virtual image expression screen with corresponding expressions, so that the virtual image can display expressions that match the expression package gift. Finally, the virtual image expression screen is subjected to image synthesis processing with the corresponding live video stream to obtain a target live stream, so that the user end can decode and play the target live stream, thereby displaying the expression changes of the virtual image after the user presents the expression package gift on the user end. In this way, real-time interaction with the virtual image through the expression package gift is achieved, which enhances the audience's sense of participation and makes the live interaction more vivid and interesting.

[0045] The following combination Figure 2 , each step in steps S200~S206 and other optional steps are described in detail.

[0046] Step S200 , get the emoticon gifts given by the target user.

[0047] The target user can be any user watching the live broadcast of the virtual image in the live broadcast room, or a user who meets the preset conditions among all users watching the live broadcast of the virtual image in the live broadcast room. The preset conditions can be that the user's account level is above level 3, the user is a VIP user, etc.

[0048] The emoticon gift is used as data to subsequently drive the expression of the virtual image. The emoticon gift can be provided directly by the platform or customized by the user. When the emoticon gift is provided directly by the platform, the platform can pre-provide emoticon gifts of multiple expression categories for users to choose from. When the emoticon gift is a user-defined gift, the platform can provide an entry (control) for users to upload pictures, through which users can select the pictures they want to upload as emoticon gifts.

[0049] The emoji gift may include the emoji picture, the value of the emoji gift, the quantity and other attribute information.

[0050] Among them, the emoticon package pictures can be pictures of various expressions, such as smiling emoticon pictures, laughing emoticon pictures, happy emoticon pictures, surprised emoticon pictures, crying emoticon pictures, etc.

[0051] Step S202 , determine the expression data used to drive the virtual image based on the expression package gift.

[0052] The expression data is data used to drive the virtual image to display corresponding expressions, and the expression data may include bone drive data, blend shape data (BlendShape data), animation sequence frame data, etc.

[0053] In an optional embodiment, if Figure 3 As shown, step S202 may include: S300, performing expression recognition on the expression package gift to obtain expression category information.

[0054] S302: Detect whether there is a target expression matching the expression category information in a preset expression library.

[0055] S304: When the target expression exists in the expression library, obtain expression data associated with the target expression.

[0056] The expression category information is information used to determine the category to which the expression package contained in the expression package gift belongs. The expression category information includes smile, cry, surprise, laugh, etc.

[0057] In this embodiment, the expression of the expression package gift can be recognized by a template-based matching method, a neural network-based method, a support vector machine-based method, etc. to obtain the expression category information.

[0058] The expression library stores common avatar expressions, such as smile, cry, surprise, etc. In addition, the expression library also stores expression data corresponding to various avatar expressions.

[0059] It should be noted that, in other implementations, the expression library may also store expression feature data corresponding to various virtual image expressions, so that it is possible to determine which type of expression package gift the expression package gift sent by the user belongs to based on the expression feature data.

[0060] In this embodiment, by first performing expression recognition on the expression package gift to obtain expression category information, and then determining whether there is a target expression matching the expression category information in the expression library based on the expression category information, it is possible to quickly and accurately determine whether the target expression is pre-stored in the expression library.

[0061] In an optional embodiment, if Figure 4 As shown, step S304 may include: Step S400, searching the expression library to see whether there is expression data associated with the target expression.

[0062] Step S402: if expression data associated with the target expression exists in the expression library, obtain the expression data from the expression library.

[0063] In this embodiment, expression data corresponding to multiple expressions are pre-stored in an expression library so that when expression data is needed, it is possible to first query whether the required expression data is stored in the expression library, and when the required expression data is stored in the expression library, the expression data can be obtained in time without generating corresponding expression data based on the target expression.

[0064] In an optional embodiment, the method further includes: generating the expression data based on the target expression when the target expression does not exist in the expression library.

[0065] In this embodiment, when the target expression does not exist in the expression library, the expression data will be generated based on the target expression, so that any expression package gift sent by the user can trigger the matching virtual image expression.

[0066] It should be noted that, if the target expression does not exist in the expression library, the expression package gift may not be processed further, that is, the expression data will not be generated based on the target expression. In other words, only if the expression package gift sent by the user is an expression package gift associated with an expression included in the expression library, the virtual image expression matching it will be triggered later. For expression package gifts that are not associated with an expression included in the expression library, the virtual image expression matching it will not be triggered.

[0067] In an optional embodiment, after the expression data is generated based on the target expression, the expression data can be associated with the corresponding expression package gift and saved in the expression library so that it can be used as a new expression package gift for front-end users.

[0068] In an optional embodiment, if Figure 5 As shown, when the target expression does not exist in the expression library, generating the expression data based on the target expression may include: Step S500, when the target expression does not exist in the expression library, extract the expression feature of the expression package gift to obtain expression feature information.

[0069] Step S502: Process the expression feature information using the skeleton drive technology and the blending deformation technology to obtain the expression data.

[0070] In this embodiment, the expression feature of the expression package gift can be extracted through image processing and deep learning algorithms (such as convolutional neural networks) to obtain expression feature information.

[0071] As an example, if the emoticon gift is a "surprised" emoticon gift, then after extracting the expression features of the emoticon gift through image processing and deep learning algorithms (such as convolutional neural networks), feature data of eyes wide open and feature data of mouth open can be obtained. The feature data of eyes wide open can include the degree of curvature of the eyes, the degree of pupil dilation, etc. The feature data of mouth open can include the degree of mouth opening, the angle change of the mouth, etc.

[0072] Skeleton-driven technology is a technology widely used in computer graphics, animation production and game development. It uses an internal bone structure to manipulate the character model to achieve the movement of the character or object. In 3D facial animation, skeletal-driven technology uses "bones" to drive the model to achieve fine control of facial expressions.

[0073] BlendShape technology, also known as "blended shape deformer", can deform the "base mesh" into the "target shape" through vertex interpolation operations. It is usually used to make facial expressions and assist skeleton animation. BlendShape technology pre-makes a series of basic expressions (micro-expressions), and then generates more composite expressions by linearly weighting these basic expressions at runtime. These basic expressions are actually the vertex position offsets of the model in different expression states.

[0074] In this embodiment, after the expression feature data is obtained, the expression feature data is processed using the skeleton drive technology and the BlendShape technology, so that the expression data for subsequently making the virtual image show the corresponding expression can be obtained.

[0075] As an example, when the expression feature data includes feature data of eyes wide open and feature data of mouth open, the degree of opening of the eyes and mouth will be mapped to the facial model of the avatar based on the BlendShape technology. For example, the BlendShape weight of the mouth is 0.9 based on the degree of opening of the mouth, and the BlendShape weight of the eyes is 0.8 based on the degree of opening of the eyes. In addition, the angle and position of the bones are recalculated based on the feature data of eyes wide open and mouth open, so as to adjust the facial bones of the avatar to ensure that the facial dynamics show the corresponding expressions.

[0076] In an optional embodiment, in order to ensure the smoothness of the switching of the avatar's expression, the expression data may also include a customized transition animation frame, for example, the expression data of surprise includes a transition animation frame of the process from no expression to full expression of surprise. The transition animation frame may be automatically generated according to a preset time and curve.

[0077] This embodiment can generate expression data even when the target expression does not exist in the expression library, thereby increasing the diversity of expression package gifts.

[0078] In an optional embodiment, if Figure 6 As shown, step S202 may also include: Step S600, extracting expression features of the expression package gift to obtain expression feature information.

[0079] Step S602: Process the expression feature information using the skeleton drive technology and the blending deformation technology to obtain the expression data.

[0080] In this embodiment, after obtaining the emoticon package gifts, the expression features are directly extracted, and then the expression feature information is processed using the skeleton drive technology and the mixed deformation technology to obtain the expression data. This makes it possible to quickly generate corresponding expression data without classifying the emoticon package gifts, thereby ensuring the smoothness of the live broadcast process.

[0081] Step S204 , the virtual image is rendered based on the expression data to obtain a virtual image expression screen with a corresponding expression, where the corresponding expression is an expression that matches the gift in the expression package.

[0082] In this embodiment, a GPU-accelerated rendering engine (such as Unity or Unreal Engine) can be used to render the expression changes of the virtual image in real time according to the expression data, thereby ensuring a smooth and high-quality visual effect.

[0083] Step S206 , perform image synthesis processing on the virtual image expression screen and the corresponding live video stream to obtain the target live stream for decoding and playback by the user end.

[0084] In this embodiment, the multi-channel synthesis rendering technology of OpenGL can be used to take the virtual image expression picture as the foreground layer and the picture of the live video stream as the background layer to synthesize and output the picture of the corresponding target live stream.

[0085] Embodiment 2 This method embodiment may be executed in the user terminal 6 . Figure 7 The flowchart of the virtual image live broadcast method according to the second embodiment of the present application is schematically shown. Figure 7 As shown, the virtual image live broadcast method may include steps S700 to S704, wherein: S700, in response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server.

[0086] S702, obtaining a target live broadcast stream from the server, wherein the target live broadcast stream is generated based on the virtual image live broadcast method described above.

[0087] S704, decoding and playing the target live broadcast stream to display a virtual image expression screen with corresponding expressions on the live broadcast screen, wherein the corresponding expressions are expressions that match the expression package gift.

[0088] In this embodiment, the real-time interaction between emoticon package gifts and virtual image expressions enhances the audience's sense of participation and makes the live interaction more vivid and interesting.

[0089] In an optional embodiment, if Figure 8 As shown, step S700 includes: S800, displaying an emoticon package gift panel, wherein the emoticon package gift panel includes a default emoticon package control and / or a custom emoticon package control.

[0090] S802, in response to the target user triggering an emoji gift giving operation based on the default emoji control and / or the custom emoji control, uploading the emoji gift to a server.

[0091] The default emoticon package control may include one or more, and each default emoticon package control is used to trigger the sending of a corresponding emoticon package gift. For example, the happy emoticon package control user triggers the sending of a happy emoticon package gift, and the surprised emoticon package control is used to trigger the sending of a surprised emoticon package gift.

[0092] The custom emoji package control is used to trigger the control of giving a user-defined emoji package gift. The custom emoji package control serves as an entry for users to upload pictures. Through the custom emoji package control, users are allowed to select a suitable picture from the local device as an emoji package gift and upload it. The format of the custom uploaded emoji package gift can be GIF, PNG, etc.

[0093] In this embodiment, the emoticon gift panel can be displayed in any area of ​​the live broadcast interface, for example, at the bottom of the live broadcast interface.

[0094] The emoticon package gift panel can be displayed in the live broadcast interface after the user enters the live broadcast room, or the emoticon package gift panel can be displayed in the live broadcast interface after the user clicks the corresponding control.

[0095] In actual application, the emoji gift panel is as follows Fig. 9 As shown, when the user selects Fig. 9 After selecting the default emoji package in the list, click the Feed button to trigger the gift of the selected default emoji package. Fig. 9After defining the upload control in the click sub-define, the user can select the emoticon package image to be uploaded from the local device as the emoticon package gift. After the user selects the emoticon package image, clicking the feed button can trigger the gift of the uploaded emoticon package.

[0096] This embodiment displays an emoticon gift panel so that users can select emoticon gifts to be given according to their needs, thereby satisfying the user's need for emotional expression and enhancing the personalized features of the live broadcast.

[0097] In an optional implementation, after the user triggers the gift giving operation of the emoji package based on the custom emoji package control, in order to facilitate the subsequent server to process the emoji package gift, the emoji package gift can be pre-processed before uploading it to the server. The pre-processing includes image cropping, scaling, image format verification processing, image content verification processing, etc.

[0098] It should be noted that the image content verification process is used to verify whether the image belongs to an emoticon package image.

[0099] In an optional embodiment, the method further comprises: In response to the downloading operation of the virtual image expression picture, the virtual image expression picture is saved as a preset expression package.

[0100] In actual application, when a virtual image expression picture with corresponding expressions is displayed on a live screen, a download control can be provided for the user to download the virtual image expression picture. When the user triggers the download operation through the download control, the virtual image expression picture can be saved as a preset expression package.

[0101] Among them, the emoticon package supports multiple formats, including GIF, MP4, WebP, etc.

[0102] In some implementations, in order to reduce the size of the emoticon package file while maintaining high quality, tools such as FFmpeg may be used to compress and optimize the saved emoticon package.

[0103] In some implementations, in order to facilitate the tracing of the emoticon package, it is possible to support adding a watermark to the saved emoticon package, wherein the watermark may be the anchor nickname, platform logo, etc.

[0104] In an optional embodiment, the method further comprises: In response to a sharing operation on the virtual image expression picture, the virtual image expression picture is shared to a target platform.

[0105] In actual applications, when a virtual image expression screen with corresponding expressions is displayed on a live screen, a sharing control can also be provided for users to share the virtual image expression screen. When the user triggers the sharing operation through the sharing control, the virtual image expression screen can be shared to a target platform. The target platform can be WeChat, Weibo, etc.

[0106] In this embodiment, the generated virtual image expression screen is uploaded to the platform for other users to browse, download or forward, thereby increasing the exposure of the live broadcast content.

[0107] In order to make this application easier to understand, the following Fig.10 An exemplary application is provided. S10, the user presents an emoticon gift (“surprised” emoticon gift) to the virtual image (anchor).

[0108] S11, the user uploads the “surprise” emoticon gift to the server.

[0109] S12, the server obtains the "surprise" emoticon gift.

[0110] S13, the server determines expression data for driving the virtual image according to the "surprise" expression package gift.

[0111] S14, the server renders the virtual image based on the expression data to obtain an expression picture of the virtual image with a "surprised" expression.

[0112] S15, the server performs image synthesis processing on the virtual image expression picture and the corresponding live video stream to obtain a target live stream, and pushes the target live stream to the user end.

[0113] S16, the user terminal obtains the target live stream.

[0114] S17, the user end decodes and plays the target live stream to display a virtual image expression picture with a "surprised" expression on the live screen.

[0115] Embodiment 3 Fig.11 The block diagram of the virtual image live broadcast device according to the third embodiment of the present application is schematically shown, which is used for a server. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.11As shown, the virtual image live broadcast device 1100 may include: an acquisition module 1110, a determination module 1120, a rendering module 1130, and a synthesis module 1140, wherein: The acquisition module 1110 is used to acquire the emoticon gift sent by the target user; A determination module 1120, configured to determine expression data for driving a virtual image according to the expression package gift; A rendering module 1130 is used to render the virtual image based on the expression data to obtain an expression screen of the virtual image with a corresponding expression, where the corresponding expression is an expression that matches the gift in the expression package; The synthesis module 1140 is used to perform image synthesis processing on the virtual image expression screen and the corresponding live video stream to obtain a target live stream for decoding and playback by the user end.

[0116] As an optional embodiment, determining expression data for driving a virtual image according to the expression package gift includes: Performing expression recognition on the expression package gift to obtain expression category information; Detecting whether there is a target expression matching the expression category information in a preset expression library; When the target expression exists in the expression library, expression data associated with the target expression is acquired.

[0117] As an optional embodiment, when the target expression exists in the expression library, acquiring expression data associated with the target expression includes: Searching whether there is expression data associated with the target expression in the expression library; In the case where expression data associated with the target expression exists in the expression library, the expression data is acquired from the expression library.

[0118] As an optional embodiment, the virtual image live broadcast device 1100 further includes a generation module, which is used to: When the target expression does not exist in the expression library, the expression data is generated based on the target expression.

[0119] As an optional embodiment, when the target expression does not exist in the expression library, generating the expression data based on the target expression includes: When the target expression does not exist in the expression library, extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

[0120] As an optional embodiment, determining expression data for driving a virtual image according to the expression package gift includes: Extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

[0121] Embodiment 4 Fig.12 The block diagram of a virtual image live broadcast device according to the fourth embodiment of the present application is schematically shown, which is used for a server. The device can be divided into one or more program modules, and one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Fig.12 As shown, the virtual image live broadcast device 1200 may include: an upload module 1210, an acquisition module 1220, and a playback module 1230, wherein: The uploading module 1210 is used to upload the emoticon gift to the server in response to the emoticon gift giving operation triggered by the target user; An acquisition module 1220 is used to acquire a target live stream from the server, where the target live stream is generated based on the virtual image live broadcast method described above; The playback module 1230 is used to decode and play the target live broadcast stream to display a virtual image expression screen with corresponding expressions on the live broadcast screen, and the corresponding expressions are expressions that match the expression package gift.

[0122] As an optional embodiment, in response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server includes: Displaying an emoticon package gift panel, wherein the emoticon package gift panel includes a default emoticon package control and / or a custom emoticon package control; In response to the target user triggering an emoji package gift giving operation based on the default emoji package control and / or the custom emoji package control, the emoji package gift is uploaded to a server.

[0123] As an optional embodiment, the virtual image live broadcast device 1200 further includes a download module, which is used to: In response to the downloading operation of the virtual image expression picture, the virtual image expression picture is saved as a preset expression package.

[0124] As an optional embodiment, the virtual image live broadcast device 1200 further includes a sharing module for: In response to a sharing operation on the virtual image expression picture, the virtual image expression picture is shared to a target platform.

[0125] Embodiment 5 Fig.13 The hardware architecture diagram of a computer device 1300 suitable for implementing a method for live broadcasting a virtual image according to the fifth embodiment of the present application is schematically shown. In some embodiments, the computer device 1300 may be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. In other embodiments, the computer device 1300 may be a rack server, a blade server, a tower server, or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Fig.13 As shown, the computer device 1300 includes but is not limited to: a memory 1310, a processor 1320, and a network interface 1330 which can communicate with each other via a system bus. The memory 1310 includes at least one type of computer-readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 1310 may be an internal storage module of the computer device 1300, such as a hard disk or a memory of the computer device 1300. In other embodiments, the memory 1310 may also be an external storage device of the computer device 1300, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 1300. Of course, the memory 1310 may also include both the internal storage module of the computer device 1300 and its external storage device. In this embodiment, the memory 1310 is generally used to store an operating system and various application software installed on the computer device 1300, such as program codes of a virtual image live broadcast method, etc. In addition, the memory 1310 can also be used to temporarily store various data that have been output or are to be output.

[0126] The processor 1320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips in some embodiments. The processor 1320 is generally used to control the overall operation of the computer device 1300, such as performing control and processing related to data interaction or communication with the computer device 1300. In this embodiment, the processor 1320 is used to run the program code stored in the memory 1310 or process data.

[0127] The network interface 1330 may include a wireless network interface or a wired network interface, and the network interface 1330 is generally used to establish a communication link between the computer device 1300 and other computer devices. For example, the network interface 1330 is used to connect the computer device 1300 to an external terminal through a network, and to establish a data transmission channel and a communication link between the computer device 1300 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, etc.

[0128] It should be pointed out that Fig.13 Only a computer device having components 1310 - 1330 is shown, but it should be understood that implementing all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0129] In this embodiment, the virtual image live broadcast method stored in the memory 1310 can also be divided into one or more program modules and executed by one or more processors (such as processor 1320) to complete the embodiment of the present application.

[0130] Embodiment 6 An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the virtual image live broadcast method in the embodiment are implemented.

[0131] In this embodiment, the computer-readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as a hard disk or memory of the computer device. In other embodiments, the computer-readable storage medium may also be an external storage device of a computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Of course, the computer-readable storage medium may also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the computer-readable storage medium is generally used to store an operating system and various application software installed on the computer device, such as the program code of the virtual image live broadcast method in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that have been output or are to be output.

[0132] Embodiment 7 An embodiment of the present application also provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0133] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented by general-purpose computer devices, they can be concentrated on a single computer device, or distributed on a network composed of multiple computer devices, optionally, they can be implemented by executable program codes of computer devices, so that they can be stored in a storage device and executed by the computer device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0134] It should be noted that the above are only preferred embodiments of the present application, and the patent protection scope of the present application is not limited thereto. Any equivalent structure or equivalent process transformation made using the contents of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for live broadcasting a virtual image, characterized in that: For a server, the method comprises: Get the emoji gifts sent by the target user; Determining expression data for driving the virtual image according to the expression package gift; Rendering the virtual image based on the expression data to obtain an expression picture of the virtual image with a corresponding expression, wherein the corresponding expression is an expression that matches the gift in the expression package; and The virtual image expression screen and the corresponding live video stream are subjected to image synthesis processing to obtain a target live stream for decoding and playback by a user terminal.

2. The method according to claim 1, characterized in that: Determining expression data for driving the virtual image according to the expression package gift includes: Performing expression recognition on the expression package gift to obtain expression category information; Detecting whether there is a target expression matching the expression category information in a preset expression library; When the target expression exists in the expression library, expression data associated with the target expression is acquired.

3. The method according to claim 2, characterized in that When the target expression exists in the expression library, acquiring expression data associated with the target expression includes: Searching whether there is expression data associated with the target expression in the expression library; In the case where expression data associated with the target expression exists in the expression library, the expression data is acquired from the expression library.

4. The method according to claim 2, characterized in that: The method further comprises: When the target expression does not exist in the expression library, the expression data is generated based on the target expression.

5. The method according to claim 4, characterized in that In the case that the target expression does not exist in the expression library, generating the expression data based on the target expression includes: When the target expression does not exist in the expression library, extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

6. The method according to claim 1, characterized in that Determining expression data for driving the virtual image according to the expression package gift includes: Extracting expression features of the expression package gift to obtain expression feature information; The expression feature information is processed by using the skeleton drive technology and the mixed deformation technology to obtain the expression data.

7. A method for live broadcasting a virtual image, characterized in that: For a user terminal, the method includes: In response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server; Obtaining a target live stream from the server, wherein the target live stream is generated based on the virtual image live broadcast method according to any one of claims 1 to 6; The target live stream is decoded and played to display a virtual image expression screen with corresponding expressions on the live screen, where the corresponding expressions are expressions that match the expression package gift.

8. The method according to claim 6, characterized in that In response to an emoticon gift giving operation triggered by a target user, uploading the emoticon gift to a server includes: Displaying an emoticon package gift panel, wherein the emoticon package gift panel includes a default emoticon package control and / or a custom emoticon package control; In response to the target user triggering an emoji package gift giving operation based on the default emoji package control and / or the custom emoji package control, the emoji package gift is uploaded to a server.

9. The method according to claim 7 or 8, characterized in that The method further comprises: In response to the downloading operation of the virtual image expression picture, the virtual image expression picture is saved as a preset expression package; or In response to a sharing operation on the virtual image expression picture, the virtual image expression picture is shared to a target platform.

10. A virtual image live broadcast device, characterized in that: For a server, the device comprises: The acquisition module is used to obtain the emoji gifts sent by the target user; A determination module, used to determine expression data for driving a virtual image according to the expression package gift; A rendering module, used for rendering the virtual image based on the expression data to obtain an expression picture of the virtual image with a corresponding expression, wherein the corresponding expression is an expression matching the gift in the expression package; The synthesis module is used to perform image synthesis processing on the virtual image expression screen and the corresponding live video stream to obtain a target live stream for decoding and playback by the user end.

11. A virtual image live broadcast device, characterized in that: For a user terminal, the device comprises: An uploading module, used for uploading the emoticon gift to a server in response to an emoticon gift giving operation triggered by a target user; An acquisition module, used for acquiring a target live stream from the server, wherein the target live stream is generated based on the virtual image live broadcast method according to any one of claims 1 to 6; The playback module is used to decode and play the target live stream to display a virtual image expression screen with corresponding expressions on the live screen, where the corresponding expressions are expressions that match the expression package gift.

12. A computer device, characterized in that: include: at least one processor; and a memory communicatively connected to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images