Stereoscopic special effect display method and device for gifts in live broadcasting room, equipment and medium

Abstract

The invention discloses a live broadcast room gift three-dimensional special effect display method and device, equipment and a medium. The method comprises the steps of receiving a virtual reality live broadcast stream for video decoding, obtaining a virtual reality live broadcast video frame for image cutting, and obtaining a virtual reality live broadcast video image and a depth level gray level image; outputting the virtual reality live video image to a live room interface for display; responding to the gift three-dimensional animation special effect display instruction, and obtaining a gift three-dimensional animation special effect hierarchical image of the gift three-dimensional animation special effect; and according to the output level gray value of the stereoscopic animation special effect level image, determining a depth level gray area of a corresponding image gray value in the depth level gray image, and displaying the stereoscopic animation special effect level image to a corresponding video image area in the virtual reality live video image. The depth hierarchy of the live broadcast picture is divided, the gift animation special effect image is output to the corresponding video image area in the live broadcast picture to be displayed to form a three-dimensional animation special effect, and the impression of the live broadcast picture is improved.

Description

Technical Field

[0001] The present application relates to the field of online live broadcasting, and in particular to a method for displaying three-dimensional special effects of gifts in a live broadcasting room. In addition, the present application also relates to corresponding devices, equipment and non-volatile storage media of the method. Background Art

[0002] The live broadcast room services of existing Internet live broadcast platforms provide audience users with a variety of live broadcast room types, so that audience users can choose the live broadcast room of their interest from various types of live broadcast rooms to watch. The live broadcast room services of some Internet live broadcast platforms also have virtual reality live broadcast rooms that use virtual reality technology for live broadcast services. The anchor end can broadcast the virtual reality live broadcast picture shot by the virtual reality shooting unit as a live stream to the live broadcast room, so that the audience users at the audience end can watch the virtual reality live broadcast picture of the anchor user through the virtual reality viewing mode, such as 360-degree panoramic virtual reality live broadcast or large-format virtual reality live broadcast, etc. In the virtual reality live broadcast room, the gift animation special effects of the virtual gifts given by the audience users are usually output to the top of the layer of the virtual reality live broadcast picture for display, but the display method of the gift animation special effects is relatively conventional and cannot effectively make use of the three-dimensional characteristics of the virtual reality live broadcast picture. Therefore, how to display the gift animation special effects of the virtual gifts given by the audience users on the virtual reality live broadcast picture with three-dimensional picture to improve the display effect of the gift animation special effects has troubled many developers of Internet live broadcast platforms.

[0003] In view of the problem that existing Internet live broadcast platforms have regarding how to enhance the display perception of gift animation special effects in virtual reality live broadcast images, the applicant has made corresponding explorations in order to solve this problem. Summary of the invention

[0004] The purpose of this application is to meet user needs and provide a method for displaying three-dimensional special effects of gifts in a live broadcast room. In addition, it also involves corresponding devices, equipment, non-volatile storage media and computer program products of this method.

[0005] In order to achieve the purpose of this application, the following technical solutions are adopted:

[0006] A method for displaying three-dimensional special effects of gifts in a live broadcast room proposed for the purpose of this application includes the following steps:

[0007] Receive the virtual reality live stream pushed by the server to perform video decoding, obtain the virtual reality live video frame to perform image cropping, and obtain the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values;

[0008] Output the virtual reality live video image to the live broadcast room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image;

[0009] In response to a gift stereoscopic animation special effect display instruction, determine a gift special effect animation special effect corresponding to the gift stereoscopic animation special effect display instruction, and obtain one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect;

[0010] Determine the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determine the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and output each of the stereoscopic animation special effect level images to each of the target video image areas for display.

[0011] In a further embodiment, before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0012] In response to the event of opening the virtual reality live broadcast room, the virtual reality shooting unit is started;

[0013] Acquire a left-eye video screen image, a right-eye video screen image, and a video screen depth image captured by the virtual reality shooting unit;

[0014] The left-eye video screen image and the right-eye video screen image are used as virtual reality live video images, and depth level processing is performed on the video screen depth image.

[0015] In a further embodiment, before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0016] Calling a human image recognition model that has been pre-trained to convergence, and identifying a host human image region having a host human image in a currently captured virtual reality live video image;

[0017] An image region mapped to the host body image region is determined in the currently captured video screen depth image, and the image region is used as the host image region of the video screen depth image.

[0018] In a preferred embodiment, before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0019] Divide the depth level image areas corresponding to the multiple depth levels in the depth image of the currently captured video screen, and generate a depth level grayscale image with an image resolution smaller than that of the depth image of the video screen;

[0020] From each of the depth level image regions, determine a target depth level image region where the image center of the anchor image region in the video screen depth image is located, project the target depth level image region into the depth level grayscale image as the anchor depth level grayscale region, and set the image grayscale value of the anchor depth level grayscale region to a preset anchor image grayscale value;

[0021] Projecting a depth level image region whose depth level is before the target depth level image region into the depth level grayscale image as a pre-anchor depth level grayscale region, and setting the image grayscale value of the pre-anchor depth level grayscale region to a preset pre-anchor image grayscale value;

[0022] Projecting a depth level image region whose depth level is behind the target depth level image region and is not the farthest depth level into the depth level grayscale image as a host rear depth level grayscale region, and setting the image grayscale value of the host rear depth level grayscale region to a preset host rear image grayscale value;

[0023] A depth level image area whose depth level is the farthest depth level is projected into the depth level grayscale image as a live background depth level grayscale area, and the image grayscale value of the live background depth level grayscale area is set to a preset background image grayscale value.

[0024] In a further embodiment, the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps:

[0025] Arrange and splice the acquired virtual reality live video image and the corresponding depth level grayscale image to generate a virtual reality live video frame having the virtual reality live video image and the depth level grayscale image, and generate image coordinate information representing the coordinate positions of the virtual reality live video image and the depth level grayscale image in the live video frame;

[0026] Performing video encoding on the currently generated one or more virtual reality live video frames to generate a virtual reality live stream including video encoding data of each of the live video frames and corresponding image coordinate information;

[0027] The virtual reality live broadcast stream is broadcast to a virtual reality live broadcast room.

[0028] In a further embodiment, the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps:

[0029] Acquire a virtual reality live stream pushed by a server, perform video decoding on the virtual reality live stream, and acquire a virtual reality live video frame corresponding to the virtual reality live stream;

[0030] Determining a virtual reality live video image region and a depth level grayscale image region from the virtual reality live video frame according to image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream;

[0031] The image in the virtual reality live video image area is cropped as the virtual reality live video image, and the image in the depth level grayscale image area is cropped as the depth level grayscale image.

[0032] In a further embodiment, the step of determining the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determining the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and outputting each of the stereoscopic animation special effect level images to each of the target video image areas for display includes the following steps:

[0033] Obtain multiple stereoscopic animation special effect level images of the stereoscopic animation special effect of the gift that needs to be displayed currently;

[0034] Obtaining the output level grayscale value of each of the stereoscopic animation special effect level images, and determining, from the depth level grayscale image currently displaying the virtual reality live video image, a target depth level grayscale area corresponding to the image grayscale value and the output level grayscale value for each of the stereoscopic animation special effect level images;

[0035] The target video special effects area corresponding to each target depth level grayscale area is determined from the virtual reality live video image, and each of the stereoscopic animation special effects level images is output to the corresponding target video special effects area for display.

[0036] A three-dimensional special effects display device for gifts in a live broadcast room is proposed to meet the purpose of this application, and includes:

[0037] A live stream receiving module is used to receive the virtual reality live stream pushed by the server for video decoding, obtain the virtual reality live video frame for image cropping, and obtain the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values;

[0038] A live screen output module is used to output the virtual reality live video image to the live room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image;

[0039] A stereoscopic image acquisition module, for responding to a gift stereoscopic animation special effect display instruction, determining a gift special effect animation special effect corresponding to the gift stereoscopic animation special effect display instruction, and acquiring one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect;

[0040] The stereoscopic image display module is used to determine the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determine the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and output each of the stereoscopic animation special effect level images to each of the target video image areas for display.

[0041] In a further embodiment, the live stream receiving module includes:

[0042] An image stitching submodule is used to arrange and stitch the acquired virtual reality live video image and the corresponding depth level grayscale image to generate a virtual reality live video frame having the virtual reality live video image and the depth level grayscale image, and generate image coordinate information representing the coordinate positions of the virtual reality live video image and the depth level grayscale image in the live video frame;

[0043] A live stream generation submodule, configured to perform video encoding on one or more currently generated virtual reality live video frames, and generate a virtual reality live stream including video encoding data of each of the live video frames and corresponding image coordinate information;

[0044] The live stream broadcasting submodule is used to broadcast the virtual reality live stream to the virtual reality live broadcast room.

[0045] In a preferred embodiment, the live stream receiving module further includes:

[0046] A live stream decoding submodule is used to obtain a virtual reality live stream pushed by a server, perform video decoding on the virtual reality live stream, and obtain a virtual reality live video frame corresponding to the virtual reality live stream;

[0047] An image region determination submodule, configured to determine a virtual reality live video image region and a depth level grayscale image region from the virtual reality live video frame according to image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream;

[0048] The image area cropping submodule is used to crop the image in the virtual reality live video image area as the virtual reality live video image, and crop the image in the depth level grayscale image area as the depth level grayscale image.

[0049] In a further embodiment, the stereoscopic image display module comprises:

[0050] A stereoscopic special effect hierarchical image acquisition submodule is used to acquire multiple stereoscopic animation special effect hierarchical images of the stereoscopic animation special effect of the gift currently to be displayed;

[0051] The target grayscale region determination submodule is used to obtain the output level grayscale value of each of the stereoscopic animation special effect level images, and determine the target depth level grayscale region corresponding to the image grayscale value and the output level grayscale value thereof for each of the stereoscopic animation special effect level images from the depth level grayscale image currently displaying the virtual reality live video image;

[0052] The stereo special effect level image output submodule is used to determine the target video special effect area corresponding to each target depth level grayscale area from the virtual reality live video image, and output each stereo animation special effect level image to its corresponding target video special effect area for display.

[0053] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the steps of the above-mentioned method for displaying three-dimensional special effects of gifts in the live broadcast room.

[0054] In order to solve the above technical problems, an embodiment of the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the above-mentioned method for displaying three-dimensional special effects of gifts in the live broadcast room.

[0055] In order to solve the above technical problems, an embodiment of the present application also provides a computer program product, including a computer program and computer instructions. When the computer program and computer instructions are executed by a processor, the processor executes the steps of the above-mentioned method for displaying three-dimensional special effects of gifts in the live broadcast room.

[0056] Compared with the prior art, the advantages of this application are as follows:

[0057] The present application generates a depth-level grayscale image for outputting a gift stereoscopic animation effect for a virtual reality live video image shot in a virtual reality manner, so that after the audience user gives a virtual gift to the anchor user in the virtual reality live broadcast room, the gift stereoscopic animation effect of the virtual gift given can be displayed in the live broadcast room interface, and then with the help of the three-dimensional sense characteristics of the virtual reality live broadcast screen, the animation effects of the virtual gift can be presented as three-dimensional animation effects in the virtual reality live video screen, which will not only not block the live content displayed in the virtual reality live video screen, but also be combined with the live content to enhance the animation effect of the virtual gift in the virtual reality live broadcast room. The depth-level grayscale image is based on the depth level of each object displayed in the virtual reality live video image. The depth level grayscale image is generated by the relationship, and based on the image grayscale, the depth level grayscale area with the depth level front-to-back relationship between each object is divided, and then based on the output level grayscale value preset for each gift stereoscopic animation special effect level image in the gift stereoscopic animation special effect, the depth level grayscale area corresponding to each gift stereoscopic animation special effect level image in the depth level grayscale image is determined, so as to determine the target image area corresponding to the depth level grayscale area from the virtual reality live video image, and then display each gift stereoscopic animation special effect level image in the corresponding target image area in the output virtual reality live video image, so as to present the stereoscopic special effect display effect of the gift stereoscopic animation special effect in the live broadcast room interface, thereby improving the live viewing experience of the audience users in the virtual reality live broadcast room.

[0058] Secondly, in the present application, a virtual reality live video frame is generated by arranging and splicing the virtual reality live video image and the corresponding depth level grayscale image, and a live stream with video encoding data of the virtual reality live video frame and its image position information is generated for broadcasting. The user end obtains the virtual reality live video image and the depth level grayscale image from the virtual reality live video frame by decoding the virtual reality live stream, so as to ensure that the virtual reality live video image and the corresponding depth level grayscale image can be pushed to the user end synchronously, so that the user end can quickly obtain the depth level grayscale image corresponding to the virtual reality live video image when outputting the gift stereoscopic animation special effect, determine the image area corresponding to the gift stereoscopic animation special effect in the virtual reality live video image for output display, and ensure the special effect display effect of the output and displayed gift stereoscopic animation special effect in the live broadcast room interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0060] Figure 1 A schematic diagram of a typical network deployment architecture related to implementing the technical solution of this application;

[0061] Figure 2 This is a flowchart of a typical embodiment of the method for displaying three-dimensional special effects of gifts in a live broadcast room of the present application;

[0062] Figure 3 A flowchart of a specific implementation method of a virtual reality live video image shot by a host user terminal and a video screen depth image for generating a depth-level grayscale image in this application;

[0063] Figure 4 A flowchart of a specific implementation method of identifying a host's human image area from a virtual reality live video image to map a corresponding host image area in a video screen depth image in this application;

[0064] Figure 5 A flowchart of a specific implementation method of generating a corresponding depth-level grayscale image according to a video screen depth image projection in this application;

[0065] Figure 6 A flowchart of a specific implementation method of splicing a virtual reality live video image with a depth-level grayscale image to generate a virtual reality live video frame, and then encoding the virtual reality live video frame to generate a virtual reality live stream for broadcast push in this application;

[0066] Figure 7A flowchart of a specific implementation method of decoding a virtual reality live stream on a user end to obtain a virtual reality live video frame, and cropping a virtual reality live video image and a depth-level grayscale image from the virtual reality live video frame in this application;

[0067] Figure 8 A flowchart diagram of a specific implementation method of the present application for outputting the three-dimensional animation special effect level images of the gift three-dimensional animation special effect to the corresponding depth level grayscale area in the depth level grayscale image by the user end;

[0068] Fig. 9 This is a principle block diagram of a typical embodiment of the live studio gift stereo special effects display device of the present application;

[0069] Fig.10 This is a basic structural block diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0070] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0071] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0072] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with those in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as here.

[0073] It will be understood by those skilled in the art that the "user end", "terminal" and "terminal device" used herein include both devices with wireless signal receivers, which are devices with only wireless signal receivers without transmission capabilities, and devices with receiving and transmitting hardware, which are devices with receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have single-line displays or multi-line displays or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service, personal communication system), which can combine voice, data processing, fax and / or data communication capabilities; PDA (Personal Digital Assistant, personal digital assistant), which may include a radio frequency receiver, pager, Internet / intranet access, web browser, notepad, calendar and / or GPS (Global Positioning System, global positioning system) receiver; traditional laptop and / or palmtop computers or other devices, which have and / or include traditional laptop and / or palmtop computers or other devices with and / or including radio frequency receivers. The "user end", "terminal" and "terminal device" used herein may be portable, transportable, installed in a vehicle (air, sea and / or land), or suitable for and / or configured to run locally, and / or in a distributed form, at any other location on the earth and / or in space. The "user end", "terminal" and "terminal device" used herein may also be a communication terminal, an Internet terminal, a music / video playing terminal, for example, a PDA, a MID (Mobile Internet Device) and / or a mobile phone with a music / video playing function, or a smart TV, a set-top box and other devices.

[0074] The hardware referred to by the names such as "server", "user terminal", "work node" etc. in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann principle, such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, and an output device. The computer program is stored in its memory, and the central processing unit calls the program stored in the external memory into the internal memory for execution, executes the instructions in the program, and interacts with the input and output devices to complete specific functions.

[0075] It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent of each other but can be called through interfaces, or integrated into a physical computer or a set of computer clusters. Those skilled in the art should understand this flexibility, and should not use it to restrict the implementation of the network deployment method of this application.

[0076] See also Figure 1 , the hardware foundation required for the implementation of the relevant technical solutions of this application can be deployed according to the architecture shown in the figure. The server 80 referred to in this application is deployed in the cloud. As an online server, it can be responsible for further connecting relevant data servers and other servers that provide relevant support, etc., so as to form a logically related service cluster to provide services for related terminal devices such as the smartphone 81 and personal computer 82 shown in the figure or a third-party server (not shown). The smartphone and personal computer can access the Internet through a well-known network access method and establish a data communication link with the server 80 in the cloud to run terminal applications related to the services provided by the server.

[0077] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0078] The application described herein refers to an application running on a server or terminal device. This application implements the relevant technical solutions of the present application in a programming manner. Its program code can be stored in a non-volatile storage medium recognizable by a computer in the form of computer executable instructions, and can be loaded into the memory by the central processing unit for execution. The relevant device of the present application is constructed by the operation of the application on the computer.

[0079] For the server, the application is usually constructed as a service process, opening the corresponding program interface for remote calls by applications running on various terminal devices. The relevant technical solutions suitable for running on the server in this application can be implemented in the server in this way.

[0080] Those skilled in the art should be aware that, although the various methods of the present application are described based on the same concept and thus present commonality to each other, unless otherwise specified, these methods can be independently executed. Similarly, for each embodiment disclosed in the present application, they are all proposed based on the same inventive concept, therefore, concepts with the same expression, and concepts that are appropriately changed for convenience despite different expressions, should be understood as equivalent.

[0081] See also Figure 2 , a method for displaying three-dimensional special effects of gifts in a live broadcast room of the present application, in its typical embodiment, comprises the following steps:

[0082] Step S11, receiving the virtual reality live stream pushed by the server to perform video decoding, obtaining the virtual reality live video frame to perform image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values:

[0083] On the Internet live broadcast platform, the host user can open a live broadcast room that uses virtual reality technology through the virtual reality live broadcast room service provided by the platform. This type of live broadcast room is generally called a virtual reality live broadcast room. The virtual reality live broadcast room broadcasts the virtual reality live stream to the audience user end so that the audience users at the audience user end can watch the live broadcast through the virtual reality viewing mode.

[0084] The virtual reality live broadcast stream includes a virtual reality live broadcast video frame for displaying the virtual reality live broadcast screen pushed by the anchor user terminal in the virtual reality live broadcast room. The virtual reality live broadcast video frame includes a virtual reality live broadcast video image for generating live broadcast content by video recording in a virtual reality shooting manner, and a depth-level grayscale image for displaying a gift stereoscopic animation special effect. The virtual reality live broadcast video image and the depth-level grayscale image in the virtual reality live broadcast video frame are arranged and spliced ​​in a non-overlapping manner, and the virtual reality live broadcast video frame has image coordinate information representing the image coordinate positions of the virtual reality live broadcast video image and the depth-level grayscale image, so that the user terminal can crop the virtual reality live broadcast video image and the depth-level grayscale image from the virtual reality live broadcast video frame according to the image coordinate information of the virtual reality live broadcast video frame, merge the images, and generate a virtual reality live broadcast video image for the gift stereoscopic animation special effect, which is output to the live broadcast room interface for display.

[0085] If the anchor user at the anchor user end wants to open a virtual reality live broadcast room with a virtual reality live broadcast function, it is necessary to push a virtual reality live broadcast start instruction to the server responsible for the virtual reality live broadcast service in the Internet live broadcast platform, so that the server can respond to the virtual reality live broadcast start instruction and open the virtual reality live broadcast room for the live broadcast user at the anchor user end. When the virtual reality live broadcast room is opened, the anchor user end will trigger a response to the virtual reality live broadcast room start event to start the virtual reality shooting unit of the anchor user end to shoot the virtual reality live broadcast video. The virtual reality shooting unit of the anchor user end generally has left and right parallel fisheye cameras to shoot virtual reality live broadcast video images with a 180-degree wide frame, wherein the fisheye on the left The video screen image taken by the left-eye camera is generally called the left-eye video screen image, and the video screen image taken by the fisheye camera on the right is generally called the right-eye video screen image. The image composed of the left-eye video screen image and the right-eye video screen image is called the virtual reality live video image. In addition, the virtual reality shooting unit generally also has a depth shooting camera. The image taken by the depth shooting camera is a video screen depth image used to map and generate a depth-level grayscale image. Of course, the virtual reality shooting unit can also generate a corresponding video screen depth image by performing image depth processing on the virtual reality live video image taken by the fisheye cameras distributed in parallel on the left and right without the depth shooting camera.

[0086] The video screen depth image refers to a depth map in the field of computer vision, which can represent the distance between the object displayed in the virtual reality live video image and the camera of the virtual reality shooting unit. If the video screen depth image is shot by the depth shooting camera, it represents the distance between each object and the depth shooting camera. If the video screen depth image is generated by performing image depth processing on the virtual reality live video image, it represents the distance between each object and the fisheye camera. The video screen depth image generally represents the distance between each object and the camera in a color style formed based on thermal imaging, or represents the distance between each object and the camera in a grayscale style. It can be understood that the virtual reality live video image and the video screen depth image display the same shooting content, but the video screen depth image represents the distance between the object displayed in the virtual reality live video image and the virtual reality shooting unit in the form of depth.

[0087] After obtaining the video screen depth image corresponding to the virtual reality live video image, the video screen depth image will be depth-level processed to generate a corresponding depth-level grayscale image. First, a human image recognition model that has been pre-trained to convergence will be used to identify an image area with a person image from the virtual reality live video image as the anchor image area, and through the image mapping relationship between the virtual reality live video image and the video screen depth image, an image area mapped to the anchor human image area is determined from the video screen depth image, and the image area is used as the anchor image area of ​​the video screen depth image.

[0088] When the anchor image area in the video screen depth image is determined, a depth level grayscale image with the same image size as the video screen depth image and a smaller image resolution than the video screen depth image is generated. The depth level grayscale image just generated is in an initialized state, and the image grayscale values ​​of all image pixels in the image are generally 0. The depth level grayscale image is divided into depth level grayscale areas corresponding to preset depth levels by grayscale values. As mentioned above, the distance between each object displayed in the image and the camera can be represented in the form of depth in the video screen depth image. However, if the depth distance between each object and the camera is not classified, the depth distance between each object and the camera cannot be effectively used for the output display of the gift stereoscopic animation image. Therefore, by presetting various depth levels, the anchor image area in the video screen depth image can be used as a reference to classify the depth distance between the object and the camera. Objects at different depth distances between the anchor user and the camera are classified into depth levels such as objects in front of the anchor user, objects behind the anchor user, or live broadcast background. Specifically, multiple depth levels are pre-set first. The depth levels pre-set for the video screen depth image can be respectively the anchor front depth level representing the depth level of objects in front of the anchor user in the image, the anchor depth level representing the depth level of the anchor user in the image, the anchor back depth level representing the depth level of objects behind the anchor user in the image, and the live broadcast background depth scene representing the background of the live broadcast scene in the image. The hierarchical front-back relationship between the depth levels is determined based on the depth distance of the objects represented by each depth level from the camera. For example, the level of the anchor front depth level is before the anchor depth level, the level of the anchor back depth level is behind the anchor depth level, and the level of the live broadcast scene depth level is behind the level of the anchor back depth level.

[0089] Regarding dividing the depth level image areas corresponding to each preset depth level from the video screen depth image, generally, the depth level image areas corresponding to each depth level in the video screen depth image are divided by calling a depth level division model that has been pre-trained to convergence or a depth level division algorithm constructed based on the K-Means algorithm. Specifically, the depth level division model that has been pre-trained to convergence is called, and the video screen depth image is used as a model input to obtain a video screen depth image output by the depth level division model that divides the depth level image areas corresponding to each depth level, or the depth level division algorithm is used to cluster and divide each image pixel in the video screen depth image, and each image pixel is clustered into a corresponding depth level based on the depth distance between each image pixel and the camera represented in the video screen depth image, and then the image pixels clustered at each depth level are grouped into groups. The image area composed of pixels is divided into depth level image areas corresponding to each depth level in the depth image of the video screen. It should be noted that the depth level of each depth level image area divided from the depth image of the video screen does not represent the specific object displayed in the depth level image area. The depth level of each depth level image area is only used to represent the depth distance of the object displayed in the depth level image area from the camera. For example, when the number of depth level image areas divided from the depth image of the video screen is 4, 4 depth levels with preset depth distances are represented, and the object representing the depth distance from the camera can be the closest, relatively close, not close and far away. It can be understood that the depth level image area divided from the depth image of the video screen only represents the depth distance between the object displayed in its image area and the camera, and does not represent what type of attribute the displayed object has.

[0090] After dividing the depth level image areas corresponding to each depth level from the video screen depth image, the target depth level image area where the image center of the anchor image area in the video screen depth image is located will be determined, and the target depth level image area will be determined as the image area corresponding to the anchor depth level, so as to map the target depth level image area to the depth level grayscale image as the anchor depth level grayscale area, and set the image grayscale value of the anchor depth level grayscale area to a preset anchor image grayscale value, and the anchor image grayscale value can generally be set to a value of 100.

[0091] After determining the target depth level image area corresponding to the anchor's depth level in the video screen depth image, a depth level image area whose depth level is before the target depth level image area is determined from the video screen depth image, and the depth level image area is used as the depth level image area corresponding to the anchor's front depth level, and then the depth level image area is projected into the depth level grayscale image as the anchor's front depth level grayscale area, and the image grayscale value of the anchor's front depth level grayscale area is set to a preset anchor's front image grayscale value. The anchor's front image grayscale value can be set within a numerical range of 0 to 99. For example, the anchor's front image grayscale value can be set to 50.

[0092] In addition, after determining the target depth level image area corresponding to the anchor depth level in the video screen depth image, it will also be determined from the video screen depth image that the depth level is after the target depth level image area, and the depth distance represented by the depth level is not the last depth level image area, so as to use the depth level image area as the depth level image area corresponding to the anchor's rear depth level, and then project the depth level image area into the depth level grayscale image as the anchor's rear depth level grayscale area, and set the image grayscale value of the anchor's rear depth level grayscale area to a preset anchor's rear image grayscale value. The anchor's rear image grayscale value can be set within a numerical range of 101 to 244. For example, the anchor's rear image grayscale value can be set to 150.

[0093] After determining the depth level image area corresponding to the depth level behind the anchor in the video screen depth image, the depth level image area corresponding to the depth level representing the farthest depth distance between the object displayed in the image and the camera is determined from the video screen depth image, and the depth level image area is used as the depth level image area of ​​the live background depth level, and then the depth level image area is projected into the depth level grayscale image as the live background depth level grayscale area, and the image grayscale value of the live background depth level grayscale area is set to a preset background image grayscale value, and the background image grayscale value is generally set to a value of 255.

[0094] After dividing the depth level image areas of each depth level with the depth distance front-to-back relationship in the depth image of the video screen, taking the anchor depth level image area where the anchor image area showing the anchor character object in the depth image of the video screen is located as the benchmark, according to the depth distance front-to-back relationship between the depth levels of the remaining depth level image areas and the anchor depth level image area, the remaining depth level image areas are respectively determined as the anchor front depth level image area where the object object is located in the depth distance before the anchor character object, and the anchor rear depth level image area where the object object is located in the depth distance behind the anchor character object. And it displays the live broadcast background depth level image area where the object at the last of the anchor character object is located in the depth distance, and projects the anchor depth level image area, the anchor front depth level image area, the anchor back depth level image area and the live broadcast background depth level image area in the video screen depth image into the corresponding depth level grayscale image as the corresponding depth level grayscale area, so as to generate the anchor depth level grayscale area, the anchor front depth level grayscale area, the anchor back depth level grayscale area and the live broadcast background depth level grayscale area in the depth level grayscale image, and each of the depth level grayscale areas has its corresponding image grayscale value for area division.

[0095] After obtaining the virtual reality live video image currently shot by the anchor user end, and obtaining the depth-level grayscale image generated by dividing the depth-level grayscale image according to the video screen depth image corresponding to the virtual reality anchor video image, the virtual reality live video image and the depth-level grayscale image are arranged and spliced ​​without overlapping images to generate a virtual reality live video frame having the virtual reality live video image and the depth-level grayscale image, and image coordinate information representing the coordinate positions of the virtual reality live video image and the depth-level grayscale image in the live video frame is generated, and then the virtual reality live video frame is video-encoded to generate a virtual reality live stream including the video encoding data of each live video frame and the image coordinate information, and the virtual reality live stream is broadcast to the virtual reality live room where the anchor user end is located.

[0096] After receiving the virtual reality live stream, the user end needs to decode the virtual reality live video stream to obtain the virtual reality live video frames contained in the virtual reality live video stream, and determine the virtual reality live video image area and the depth level grayscale image area from the virtual reality live video frame according to the image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream, so as to crop the image in the virtual reality live video image area as the virtual reality live video image, and crop the image in the depth level grayscale image area as the depth level grayscale image.

[0097] Step S12, outputting the virtual reality live video image to the live studio interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image;

[0098] Output the virtual reality live video image to the live broadcast room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image;

[0099] The user end decodes and obtains a virtual reality live video frame from the currently received virtual reality live stream, and after cropping the virtual reality live video image and the depth-level grayscale image contained in the virtual reality live video frame, the user end outputs the virtual reality live video image to the live broadcast room interface for display.

[0100] The depth-level grayscale image in the same virtual reality live video frame as the virtual reality live video image currently output to the live studio interface will be used to determine the gift stereoscopic animation special effect to be output to the virtual reality live video image, and the display level displayed in the virtual reality live video image, determine the depth-level grayscale area corresponding to the stereoscopic animation special effect level image of the gift stereoscopic animation special effect in the depth-level grayscale image, and determine the video image area corresponding to the depth-level grayscale area in the virtual reality live video image, and then output the stereoscopic animation special effect level image to the video image area for display, and there may be part of the image in the stereoscopic animation special effect level image that does not need to be output and displayed, and this part of the image is the image that is blocked by the video image area that is not output and displayed by the stereoscopic animation special effect level image in the virtual reality live video image. For details, please refer to the subsequent steps.

[0101] Step S13, responding to the gift stereoscopic animation special effect display instruction, determining the gift special effect animation special effect corresponding to the gift stereoscopic animation special effect display instruction, and obtaining one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect:

[0102] When an audience user in a virtual reality live broadcast room presents a virtual gift to the anchor user, and the virtual gift has a corresponding gift stereoscopic animation special effect that needs to be output to the live broadcast room interface for display, the user terminal in the virtual reality live broadcast room will trigger a gift stereoscopic animation special effect display instruction that responds to the gift stereoscopic animation special effect.

[0103] After the user end responds to the gift stereoscopic animation special effects display instruction, it will determine the gift stereoscopic animation special effects that need to be output to the live broadcast room interface and that correspond to the gift stereoscopic animation special effects display instruction, and obtain one or more gift stereoscopic animation special effects level images corresponding to the gift stereoscopic animation special effects, so as to output each of the gift stereoscopic animation image level images to the depth level grayscale area corresponding to the depth level grayscale image in the virtual reality live video image currently displayed in the live broadcast room interface, so as to display the gift stereoscopic animation special effects with stereoscopic animation special effects in the live broadcast room interface.

[0104] The gift stereoscopic animation special effects level image corresponding to the gift stereoscopic animation special effects has a corresponding output level grayscale value to specify the depth level grayscale area where the gift stereoscopic animation special effects level image needs to be output and displayed in the depth level grayscale image. The numerical range of the output level grayscale value of each gift stereoscopic animation special effects level image is generally designed according to the image grayscale value of the depth level grayscale area that needs to be output and displayed. For example, for the gift stereoscopic animation special effects level image in the depth level grayscale area in front of the anchor that needs to be output to the depth level grayscale image, the value range of the image grayscale value of the depth level grayscale area in front of the anchor is generally in the numerical range of 0 to 99, and the output level grayscale value of the gift stereoscopic animation special effects level image also needs to be set in the numerical range of 0 to 99.

[0105] Step S14, determining the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determining the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and outputting each of the stereoscopic animation special effect level images to each of the target video image areas for display:

[0106] After obtaining each stereoscopic animation special effect level image of the gift stereoscopic animation special effect that needs to be output to the live broadcast room interface for display, it is necessary to determine the target depth level grayscale area corresponding to the image grayscale value corresponding to the output level grayscale value of each stereoscopic animation special effect level image from the depth level grayscale image of the virtual reality live video image currently displayed in the live broadcast room interface.

[0107] The user end obtains the output level grayscale value of each stereoscopic animation special effect level image of the gift stereoscopic animation special effect that needs to be displayed currently, and determines the target depth level grayscale area corresponding to the image grayscale value and the output level grayscale value for each of the stereoscopic animation special effect level images from the depth level grayscale image currently displayed in the live broadcast room interface, so as to output each of the stereoscopic animation special effect level images to the target depth level grayscale area and display it in the corresponding mapped video image area in the currently displayed virtual display live video screen, thereby displaying the gift stereoscopic animation special effect composed of each of the stereoscopic animation image level images in the live broadcast room interface.

[0108] A target depth level grayscale area of ​​a stereoscopic animation special effect level image to be displayed in a depth level grayscale image corresponding to a currently displayed virtual reality live video image is determined, and based on a correspondence between the depth level grayscale image and an image in the virtual reality live video image, a video image area corresponding to the target depth level grayscale area is projected from the virtual reality live video image to output the stereoscopic animation special effect level image to the video image area for display, while special effect images in the stereoscopic animation special effect level image that are obscured by video image areas corresponding to other depth level grayscale areas in the virtual reality live video image will not be displayed in the virtual reality live video image to prevent such special effect images from obscuring images in video image areas corresponding to other depth level grayscale areas.

[0109] In order to facilitate developers to set the output level grayscale values ​​of each stereoscopic animation special effect level image of the gift stereoscopic animation special effect, there is no need to correspond one-to-one with the image grayscale values ​​of each depth level grayscale area in the depth level grayscale image, so as to improve the development efficiency of the gift stereoscopic animation special effect, the output level grayscale value of the stereoscopic animation special effect level image can be set to a corresponding numerical range. For example, for the stereoscopic animation special effect level image that needs to be output to the depth level grayscale area in front of the anchor, the image grayscale value of the depth level grayscale area in front of the anchor is generally set in the numerical range of 0 to 99, then the output level grayscale value of the stereoscopic animation special effect level image can be set to a numerical range of 0 to 99, when the stereoscopic animation special effect level image needs to be displayed, the depth level grayscale area in front of the anchor whose image grayscale value is in the numerical range of 0 to 99 can be determined from the depth level grayscale image currently displayed in the live broadcast room interface.

[0110] The above typical embodiments and their variations fully reveal the implementation scheme of the method for displaying three-dimensional special effects of gifts in the live broadcast room of the present application. However, various variations of the method can still be derived by changing and amplifying some technical means. Other embodiments are briefly described as follows:

[0111] In one embodiment, please refer to Figure 3, before the step of decoding the virtual reality live stream pushed by the receiving server, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0112] Step S08, responding to the virtual reality live broadcast room opening event, starting the virtual reality shooting unit:

[0113] When the anchor user terminal opens a virtual reality live broadcast room with a virtual reality live broadcast service, the virtual reality shooting unit of the main package user terminal will be started to shoot virtual reality live video images applied to the virtual reality live broadcast service, as well as video picture depth images used to project and generate corresponding depth level grayscale images.

[0114] Step S09, obtaining the left-eye video screen image, the right-eye video screen image and the video screen depth image captured by the virtual reality shooting unit:

[0115] The left-eye video screen image and the right-eye video screen image are generally captured by left and right parallel fisheye cameras of a virtual reality shooting unit, respectively, and the video screen depth image can generally be captured by a depth shooting camera of the virtual reality shooting unit, or generated by performing image depth processing on a virtual reality live video image that is a combination of the left-eye video screen image and the right-eye video screen image.

[0116] Step S10: using the left-eye video screen image and the right-eye video screen image as virtual reality live video images, and performing depth level processing on the video screen depth image:

[0117] After obtaining the left-eye video screen image and the right-eye video screen image, the left-eye video screen image and the right-eye video screen image can be spliced ​​to generate a corresponding virtual reality live video image, and the video screen depth image is used for depth level processing to project and generate a corresponding depth level grayscale image.

[0118] In this embodiment, a virtual reality shooting unit is used to obtain a left-eye video screen image and a right-eye video screen image and stitch them together to generate a virtual reality live video image, and a video screen depth image is obtained for depth level processing and projection to generate a corresponding depth level grayscale image for outputting and displaying gift stereo animation effects, thereby providing a virtual reality live viewing mode for audience users.

[0119] In one embodiment, please refer to Figure 4, before the step of decoding the virtual reality live stream pushed by the receiving server, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0120] Step S08', calling the human image recognition model that has been pre-trained to convergence, and identifying the host human image area having the host human image in the currently captured virtual reality live video image:

[0121] The human image recognition model is an image recognition model constructed based on a neural network model. After the human image recognition model is trained to a convergence state, the human image recognition model will be called to identify the host human image area with human image pixels in the currently shot virtual reality live video image.

[0122] Step S09', determining an image area mapped to the host human image area in the currently captured video screen depth image, and using the image area as the host image area of ​​the video screen depth image:

[0123] Because the video screen depth image is the same as the object captured in the virtual reality live video image, after the host human image area of ​​the virtual reality live video image is identified, the host image area in the video screen depth image that also has human body image pixels can be mapped out through image mapping.

[0124] In this embodiment, by calling the human image recognition model, a virtual reality live video image with high image pixel accuracy is identified as having a host human image area, and then the host image area corresponding to the host human image area is mapped from the video screen depth image taken at the same time, so as to improve the recognition accuracy of the host image area identified in the video screen depth image.

[0125] In one embodiment, please refer to Figure 5 , before the step of decoding the virtual reality live stream pushed by the receiving server, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the following steps are included:

[0126] Step S06, divide the depth level image areas corresponding to the multiple depth levels in the depth image of the currently captured video screen, and generate a depth level grayscale image with an image resolution smaller than the video screen depth image:

[0127] The depth levels have a front-to-back relationship in the depth image of the video screen. The distances between the objects in the depth level image areas corresponding to different depth levels and the camera are different. The distance between the objects in the depth level image areas with a front depth level and the camera is closer, and so on, the distance between the objects in the depth level image areas with a back depth level and the camera is farther.

[0128] The depth level grayscale image generated for the video screen depth image generally has an image resolution smaller than the video screen depth image, but generally has the same image size as the video screen depth image, so as to facilitate projecting the corresponding depth level grayscale area from the video screen depth image.

[0129] Step S07", from each of the depth level image areas, determine the target depth level image area where the image center of the anchor image area in the video screen depth image is located, project the target depth level image area into the depth level grayscale image as the anchor depth level grayscale area, and set the image grayscale value of the anchor depth level grayscale area to the preset anchor image grayscale value:

[0130] A target depth level image area having image pixels of a host image area is determined from a video screen depth image. Generally, a rectangular image having an image center is connected to the host image area, and then a depth level image area matching the image center of the rectangular image is matched from each depth level image area in the video screen depth image, so that the depth level image area is used as the target depth level image area of ​​the host image area.

[0131] Step S08", projecting a depth level image area whose depth level is before the target depth level image area into the depth level grayscale image as a front-host depth level grayscale area, and setting the image grayscale value of the front-host depth level grayscale area to a preset front-host image grayscale value:

[0132] After determining the target depth level image area having image pixels of the anchor image area in the video screen depth image, one or more depth level image areas whose depth level in the video screen depth image is before the target depth level image area will be determined, and these depth level image areas will all be regarded as the anchor's front depth level image area, and the anchor's front depth level image area will be projected into the depth level grayscale image as the anchor's front depth level grayscale area.

[0133] Step S09", projecting a depth level image area whose depth level is behind the target depth level image area and is not the farthest depth level into the depth level grayscale image as a host rear depth level grayscale area, and setting the image grayscale value of the host rear depth level grayscale area to a preset host rear image grayscale value:

[0134] After determining the target depth level image area having image pixels of the anchor image area in the video screen depth image, one or more depth level image areas whose depth level in the video screen depth image is before the target depth level image area will be determined, and the depth level image area whose depth level is not the rearmost depth level image area in the video screen depth image will be taken as the anchor rear depth level image area, and the anchor rear depth level image area will be projected into the depth level grayscale image as the anchor rear depth level grayscale area.

[0135] Step S10", projecting the depth level image area whose depth level is the farthest depth level into the depth level grayscale image as the live background depth level grayscale area, and setting the image grayscale value of the live background depth level grayscale area to the preset background image grayscale value:

[0136] After determining the target depth level image area with image pixels of the anchor image area in the video screen depth image, the depth level image area in front of the anchor and the depth level image area behind the anchor, the depth level image area with the farthest depth level in the video screen depth image is used as the live background depth level image area, and the live background depth level image area is projected into the depth level grayscale image as the live background depth level grayscale area.

[0137] In this embodiment, by dividing the depth level image areas corresponding to the depth levels with the front and back relationship of the depth levels in the depth image of the video screen, the anchor depth level image area with the image pixels of the anchor character object is used as a benchmark, and the remaining depth level image areas are respectively determined as the depth level image areas in front of the anchor, behind the anchor and the live broadcast background, and are projected onto the depth level grayscale image accordingly to generate the depth level grayscale areas of the depth level grayscale image, so as to determine the depth level grayscale area corresponding to the gift stereoscopic animation special effect, and then determine the depth level grayscale area in the virtual reality live video screen The corresponding video area displays the gift stereoscopic animation special effect, forming a gift animation special effect with a stereoscopic effect, thereby enhancing the user's live viewing experience.

[0138] In one embodiment, please refer to Figure 6, the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps:

[0139] Step S111, arranging and splicing the acquired virtual reality live video image and the corresponding depth level grayscale image to generate a virtual reality live video frame having the virtual reality live video image and the depth level grayscale image, and generating image coordinate information representing the coordinate positions of the virtual reality live video image and the depth level grayscale image in the live video frame:

[0140] After obtaining the virtual reality live video image currently shot by the anchor user terminal and the depth-level grayscale image generated by dividing the depth-level grayscale image according to the video screen depth image corresponding to the virtual reality anchor video image, the virtual reality live video image and the depth-level grayscale image are arranged and spliced ​​without overlapping images to generate a virtual reality live video frame having the virtual reality live video image and the depth-level grayscale image, and image coordinate information representing the coordinate positions of the virtual reality live video image and the depth-level grayscale image in the live video frame is generated.

[0141] Step S112, performing video encoding on the currently generated one or more virtual reality live video frames to generate a virtual reality live stream including the video encoding data of each live video frame and the corresponding image coordinate information:

[0142] Video encoding processing is performed on one or more virtual reality live video frames currently being generated to generate video encoding data corresponding to each of the virtual reality live video frames to form a corresponding virtual reality live stream, and the image coordinate information of each video encoding data in the virtual reality live stream and the virtual reality live video frame to which it belongs is used as mapping relationship data.

[0143] Step S113, broadcasting the virtual reality live stream to the virtual reality live broadcast room:

[0144] If the virtual reality live stream is generated by the server of the Internet live broadcast platform, the server will broadcast the virtual reality live stream to the corresponding virtual reality live broadcast room. If the virtual reality live stream is generated for the anchor user terminal, the virtual reality live stream will be pushed to the server, and the server will be responsible for forwarding and broadcasting the virtual reality live stream to the virtual reality live broadcast room where the anchor user terminal is located.

[0145] In this embodiment, a virtual reality live video frame is generated by arranging and splicing the virtual reality live video image and the corresponding depth level grayscale image, and a live stream with video encoding data of the virtual reality live video frame and its image position information is generated for broadcasting, so as to ensure that the virtual reality live video image and the corresponding depth level grayscale image can be pushed to the user end synchronously, and ensure the special effect display effect of the output gift stereo animation special effect in the live broadcast room interface.

[0146] In one embodiment, please refer to Figure 7 , the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps:

[0147] Step S111', obtaining a virtual reality live stream pushed by a server, performing video decoding on the virtual reality live stream, and obtaining a virtual reality live video frame corresponding to the virtual reality live stream:

[0148] After receiving the virtual reality live broadcast stream, the user end needs to perform video decoding on the virtual reality live broadcast video stream to obtain the virtual reality live broadcast video frame contained in the virtual reality live broadcast video stream.

[0149] Step S112', according to the image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream, determine the virtual reality live video image area and the depth level grayscale image area from the virtual reality live video frame:

[0150] After decoding the virtual reality live video stream to obtain the virtual reality live video frames it contains, the user end will divide the virtual reality live video image area where the virtual reality live video image is located and the depth level grayscale image area where the depth level grayscale image is located from the virtual reality live video frame according to the image coordinate information corresponding to the virtual reality live video frame.

[0151] Step S113', cropping the image in the virtual reality live video image area as the virtual reality live video image, and cropping the image in the depth level grayscale image area as the depth level grayscale image:

[0152] After determining the virtual reality live video image area and the depth level grayscale image area in the virtual reality live video frame currently decoded and obtained, the image cropped from the virtual reality live video image area is used as the virtual reality live video image, and the image cropped from the depth level grayscale image area is used as the depth level grayscale image.

[0153] In this embodiment, the user terminal decodes the virtual reality live broadcast stream to obtain the virtual reality live broadcast video frame, and crops the virtual reality live broadcast video image and the depth level grayscale image from the virtual reality live broadcast video frame, so that the virtual reality live broadcast video image and the depth level grayscale image shot at the same time can be recorded at the same time, ensuring the stereoscopic display effect of the output gift stereoscopic animation special effect in the live broadcast room interface.

[0154] In one embodiment, please refer to Figure 8 The step of determining the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determining the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and outputting each of the stereoscopic animation special effect level images to each of the target video image areas for display comprises the following steps:

[0155] Step S141, obtaining multiple stereoscopic animation special effect level images of the stereoscopic animation special effect of the gift currently to be displayed:

[0156] Determine the gift stereoscopic animation effects that need to be displayed for the virtual gift currently given by the audience user to the anchor user in the virtual reality live broadcast room, and obtain multiple stereoscopic animation effects level images of the gift stereoscopic animation effects, each of the stereoscopic animation effects level images having an output level grayscale value displayed in a depth level grayscale area corresponding to its depth level grayscale image.

[0157] Step S142, obtaining the output level grayscale value of each of the stereoscopic animation special effect level images, and determining, from the depth level grayscale image currently displaying the virtual reality live video image, a target depth level grayscale area corresponding to the image grayscale value and the output level grayscale value for each of the stereoscopic animation special effect level images:

[0158] The gift stereoscopic animation special effect level image corresponding to the gift stereoscopic animation special effect has a corresponding output level grayscale value to specify the depth level grayscale area where the gift stereoscopic animation special effect level image needs to be output and displayed in the depth level grayscale image. Therefore, it is necessary to determine the target depth level grayscale area corresponding to the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images from the depth level grayscale image of the virtual reality live video image currently displayed in the live broadcast room interface according to the output level grayscale value of each of the stereoscopic animation special effect level images, so as to determine the corresponding video special effects area in each of the target depth level grayscale areas in the virtual reality live video image for outputting and displaying the gift stereoscopic animation special effect level image.

[0159] Step S143, determining the target video special effect area corresponding to each of the target depth level grayscale areas from the virtual reality live video image, and outputting each of the stereoscopic animation special effect level images to the corresponding target video special effect area for display:

[0160] After determining the target depth level grayscale area corresponding to each of the stereoscopic animation special effect level images in the depth level grayscale image, the target video special effect area corresponding to each target depth level grayscale area will be determined from the virtual reality live video image, so as to output each of the stereoscopic animation special effect level images to its corresponding target video special effect area for display, and then display the gift stereoscopic animation special effect composed of each of the stereoscopic animation image level images in the live broadcast room interface.

[0161] In this embodiment, based on the stereoscopic animation special effects level images of each stereoscopic animation special effects level image of the gift stereoscopic animation image, the target depth level grayscale area corresponding to each stereoscopic animation special effects level image in the depth level grayscale image is determined, so as to determine the video image area corresponding to each target depth level grayscale area from the virtual reality live video image and output each stereoscopic animation special effects level image, and then display the gift stereoscopic animation image with a stereoscopic display effect in the live broadcast room interface, thereby enhancing the virtual reality live viewing experience of the audience users.

[0162] Furthermore, by functionalizing each step in the method disclosed in the above embodiments, a three-dimensional special effect display device for gifts in a live broadcast room of the present application can be constructed. According to this idea, please refer to Fig. 9In a typical embodiment, the device includes: a live stream receiving module 11, used to receive the virtual reality live stream pushed by the server for video decoding, obtain the virtual reality live video frame for image cropping, and obtain the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by multiple image grayscale values; a live screen output module 12, used to output the virtual reality live video image to the live room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effects of the virtual reality live video image; stereo image acquisition The acquisition module 13 is used to respond to the gift stereoscopic animation special effect display instruction, determine the gift special effect animation effect corresponding to the gift stereoscopic animation special effect display instruction, and obtain one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect; the stereoscopic image display module 14 is used to determine the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determine the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and output each of the stereoscopic animation special effect level images to each of the target video image areas for display.

[0163] In one embodiment, the live stream receiving module 11 includes: an image stitching submodule, which is used to arrange and stitch the acquired virtual reality live video image and the corresponding depth level grayscale image to generate a virtual reality live video frame having the virtual reality live video image and the depth level grayscale image, and generate image coordinate information representing the coordinate positions of the virtual reality live video image and the depth level grayscale image in the live video frame; a live stream generating submodule, which is used to perform video encoding on one or more currently generated virtual reality live video frames to generate a virtual reality live stream containing video encoding data of each of the live video frames and corresponding image coordinate information; a live stream broadcasting submodule, which is used to broadcast the virtual reality live stream to a virtual reality live broadcast room.

[0164] In another embodiment, the live stream receiving module 11 further includes: a live stream decoding submodule, used to obtain a virtual reality live stream pushed by a server, perform video decoding on the virtual reality live stream, and obtain a virtual reality live video frame corresponding to the virtual reality live stream; an image area determination submodule, used to determine a virtual reality live video image area and a depth level grayscale image area from the virtual reality live video frame according to image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream; and an image area cropping submodule, used to crop the image in the virtual reality live video image area as a virtual reality live video image, and crop the image in the depth level grayscale image area as a depth level grayscale image.

[0165] In one embodiment, the stereoscopic image display module 14 includes: a stereoscopic special effect level image acquisition submodule, which acquires multiple stereoscopic animation special effect level images of the stereoscopic animation special effect of the gift currently to be displayed; a target grayscale area determination submodule, which is used to obtain the output level grayscale value of each of the stereoscopic animation special effect level images, and determine the target depth level grayscale area corresponding to the image grayscale value and the output level grayscale value for each of the stereoscopic animation special effect level images from the depth level grayscale image of the currently displayed virtual reality live video image; a stereoscopic special effect level image output submodule, which is used to determine the target video special effect area corresponding to each of the target depth level grayscale areas from the virtual reality live video image, and output each of the stereoscopic animation special effect level images to the corresponding target video special effect area for display.

[0166] In order to solve the above technical problems, the embodiment of the present application also provides a computer device for running a computer program implemented according to the method for displaying three-dimensional special effects of gifts in the live broadcast room. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.

[0167] like Fig.10As shown, a schematic diagram of the internal structure of a computer device. The computer device includes a processor, a non-volatile storage medium, a memory, and a network interface connected via a system bus. Among them, the non-volatile storage medium of the computer device stores an operating system, a database, and computer-readable instructions. The database may store a control information sequence. When the computer-readable instructions are executed by the processor, the processor can implement a method for displaying three-dimensional special effects of gifts in a live broadcast room. The processor of the computer device is used to provide computing and control capabilities to support the operation of the entire computer device. The memory of the computer device may store computer-readable instructions. When the computer-readable instructions are executed by the processor, the processor can execute a method for displaying three-dimensional special effects of gifts in a live broadcast room. The network interface of the computer device is used to connect and communicate with a terminal. Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0168] In this embodiment, the processor is used to execute the specific functions of each module / submodule in the live studio gift stereo special effects display device of this application, and the memory stores the program code and various data required to execute the above modules. The network interface is used to transmit data between user terminals or servers. The memory in this embodiment stores the program code and data required to execute all modules / submodules in the live studio gift stereo special effects display device, and the server can call the server's program code and data to execute the functions of all submodules.

[0169] The present application also provides a non-volatile storage medium, and the method for displaying three-dimensional special effects of gifts in a live broadcast room is written into a computer program and stored in the storage medium in the form of computer-readable instructions. When the computer-readable instructions are executed by one or more processors, it means that the program is running in the computer, thereby enabling one or more processors to execute the steps of the method for displaying three-dimensional special effects of gifts in a live broadcast room in any of the above-mentioned embodiments.

[0170] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0171] To summarize, the present application divides the depth levels in the virtual live broadcast screen, outputs the gift animation special effect image to the corresponding video image area in the live broadcast screen to form a three-dimensional animation special effect, thereby improving the viewing experience of the live broadcast screen.

[0172] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0173] Those skilled in the art will appreciate that the various operations, methods, steps, measures, and schemes in the processes discussed in this application may be alternated, altered, combined, or deleted. Further, other steps, measures, and schemes in the various operations, methods, and processes discussed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted. Further, the steps, measures, and schemes in the prior art that are similar to those disclosed in this application may also be alternated, altered, rearranged, decomposed, combined, or deleted.

[0174] The above is only a partial implementation method of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for displaying three-dimensional special effects of gifts in a live broadcast room, characterized in that: The steps include: Receive the virtual reality live stream pushed by the server to perform video decoding, obtain the virtual reality live video frame to perform image cropping, and obtain the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values; Output the virtual reality live video image to the live broadcast room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image; In response to a gift stereoscopic animation special effect display instruction, determine a gift special effect animation special effect corresponding to the gift stereoscopic animation special effect display instruction, and obtain one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect; Determine the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determine the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and output each of the stereoscopic animation special effect level images to each of the target video image areas for display.

2. The method according to claim 1, characterized in that Before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the method comprises the following steps: In response to the event of opening the virtual reality live broadcast room, the virtual reality shooting unit is started; Acquire a left-eye video screen image, a right-eye video screen image, and a video screen depth image captured by the virtual reality shooting unit; The left-eye video screen image and the right-eye video screen image are used as virtual reality live video images, and depth level processing is performed on the video screen depth image.

3. The method according to claim 1, characterized in that Before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the method comprises the following steps: Calling a human image recognition model that has been pre-trained to convergence, and identifying a host human image region having a host human image in a currently captured virtual reality live video image; An image region mapped to the host body image region is determined in the currently captured video screen depth image, and the image region is used as the host image region of the video screen depth image.

4. The method according to claim 3, characterized in that Before the step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, the method comprises the following steps: Divide the depth level image areas corresponding to the multiple depth levels in the depth image of the currently captured video screen, and generate a depth level grayscale image with an image resolution smaller than that of the depth image of the video screen; From each of the depth level image regions, determine a target depth level image region where the image center of the anchor image region in the video screen depth image is located, project the target depth level image region into the depth level grayscale image as the anchor depth level grayscale region, and set the image grayscale value of the anchor depth level grayscale region to a preset anchor image grayscale value; Projecting a depth level image region whose depth level is before the target depth level image region into the depth level grayscale image as a pre-anchor depth level grayscale region, and setting the image grayscale value of the pre-anchor depth level grayscale region to a preset pre-anchor image grayscale value; Projecting a depth level image region whose depth level is behind the target depth level image region and is not the farthest depth level into the depth level grayscale image as a host rear depth level grayscale region, and setting the image grayscale value of the host rear depth level grayscale region to a preset host rear image grayscale value; A depth level image area whose depth level is the farthest depth level is projected into the depth level grayscale image as a live background depth level grayscale area, and the image grayscale value of the live background depth level grayscale area is set to a preset background image grayscale value.

5. The method according to claim 1, characterized in that The step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps: Arrange and splice the acquired virtual reality live video image and the corresponding depth level grayscale image to generate a virtual reality live video frame having the virtual reality live video image and the depth level grayscale image, and generate image coordinate information representing the coordinate positions of the virtual reality live video image and the depth level grayscale image in the live video frame; Performing video encoding on the currently generated one or more virtual reality live video frames to generate a virtual reality live stream including video encoding data of each of the live video frames and corresponding image coordinate information; The virtual reality live broadcast stream is broadcast to a virtual reality live broadcast room.

6. The method according to claim 1, characterized in that The step of receiving the virtual reality live stream pushed by the server for video decoding, obtaining the virtual reality live video frame for image cropping, and obtaining the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values, comprises the following steps: Acquire a virtual reality live stream pushed by a server, perform video decoding on the virtual reality live stream, and acquire a virtual reality live video frame corresponding to the virtual reality live stream; Determining a virtual reality live video image region and a depth level grayscale image region from the virtual reality live video frame according to image coordinate information corresponding to the virtual reality live video frame in the virtual reality live stream; The image in the virtual reality live video image area is cropped as the virtual reality live video image, and the image in the depth level grayscale image area is cropped as the depth level grayscale image.

7. The method according to claim 1, characterized in that The step of determining the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determining the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and outputting each of the stereoscopic animation special effect level images to each of the target video image areas for display comprises the following steps: Obtain multiple stereoscopic animation special effect level images of the stereoscopic animation special effect of the gift that needs to be displayed currently; Obtaining the output level grayscale value of each of the stereoscopic animation special effect level images, and determining, from the depth level grayscale image currently displaying the virtual reality live video image, a target depth level grayscale area corresponding to the image grayscale value and the output level grayscale value for each of the stereoscopic animation special effect level images; The target video special effects area corresponding to each target depth level grayscale area is determined from the virtual reality live video image, and each of the stereoscopic animation special effects level images is output to the corresponding target video special effects area for display.

8. A three-dimensional special effects display device for gifts in a live broadcast room, characterized in that: include: A live stream receiving module is used to receive the virtual reality live stream pushed by the server for video decoding, obtain the virtual reality live video frame for image cropping, and obtain the virtual reality live video image and the depth level grayscale image contained in the virtual reality live video frame, wherein the depth level grayscale image has a depth level grayscale area divided by a plurality of image grayscale values; A live screen output module is used to output the virtual reality live video image to the live room interface for display, and the depth level grayscale image is used to display the gift stereo animation special effect of the virtual reality live video image; A stereoscopic image acquisition module, for responding to a gift stereoscopic animation special effect display instruction, determining a gift special effect animation special effect corresponding to the gift stereoscopic animation special effect display instruction, and acquiring one or more gift stereoscopic animation special effect level images of the gift stereoscopic animation special effect; The stereoscopic image display module is used to determine the depth level grayscale area of ​​the image grayscale value corresponding to the output level grayscale value of each of the stereoscopic animation special effect level images in the depth level grayscale image, determine the target video image area corresponding to each of the depth level grayscale areas in the virtual reality live video image, and output each of the stereoscopic animation special effect level images to each of the target video image areas for display.

9. An electronic device, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run a computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

Images