Special effect rendering method, device, equipment, computer readable storage medium and product

Abstract

Embodiments of the present disclosure provide a special effect rendering method, device, equipment, computer readable storage medium and product, the method comprises: obtaining the identification information of a target special effect and a to-be-rendered image frame corresponding to a virtual reality live broadcast; determining rendering data corresponding to the target special effect and a graphics processor combination for rendering the to-be-rendered image frame according to the identification information of the target special effect; performing a rendering operation on the to-be-rendered image frame according to the rendering data through the graphics processor combination to obtain a rendered target image frame; and sending the target image frame to a display device for playing. Thus, the display device does not need to have a high level of hardware equipment. In addition, the content quality of the rendered target image frame can be effectively improved by using the graphics processor combination to perform a special effect rendering operation on the to-be-rendered image frame. Further, the user experience of a virtual reality live broadcast viewer can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a special effects rendering method, apparatus, device, computer-readable storage medium, and product. Background Technology

[0002] With the development of technology, Virtual Reality (VR) technology is gradually entering users' lives. Users can use VR technology to conduct 3D VR live broadcasts. During VR live broadcasts, there is usually a need for special effects rendering.

[0003] In VR live streaming scenarios, local effects are typically rendered on the streaming client, pushing the raw live stream to the viewer, where network effects are then rendered. However, VR live streaming generally uses binocular cameras for real-time shooting and employs high-resolution ultra-high-definition video frames. Therefore, rendering effects on the viewer often places high demands on its performance, significantly impacting resource consumption and energy usage, and limiting the live stream frame rate, thus affecting the viewing experience for users. Summary of the Invention

[0004] This disclosure provides a special effects rendering method, apparatus, device, computer-readable storage medium, and product to address the technical problem that existing special effects rendering methods place high demands on the hardware of the viewing terminal and put significant pressure on the resource consumption and energy consumption of the viewing terminal.

[0005] In a first aspect, embodiments of this disclosure provide a special effects rendering method, including:

[0006] Obtain the image frames to be rendered and the identification information of the target effects corresponding to the virtual reality live broadcast;

[0007] The rendering data corresponding to the target effect and the graphics processor combination used to render the image frame to be rendered are determined based on the identification information of the target effect.

[0008] The graphics processor combination performs a rendering operation on the image frame to be rendered according to the rendering data to obtain the rendered target image frame.

[0009] The target image frame is sent to a display device for playback.

[0010] Secondly, embodiments of this disclosure provide a special effects rendering apparatus, including:

[0011] The acquisition module is used to acquire the image frames to be rendered and the identification information of the target effects corresponding to the virtual reality live broadcast;

[0012] The determining module is used to determine the rendering data corresponding to the target effect and the graphics processor combination used to render the image frame to be rendered based on the identification information of the target effect;

[0013] The rendering module is used to perform rendering operations on the image frame to be rendered according to the rendering data through the graphics processor combination to obtain the rendered target image frame.

[0014] The sending module is used to send the target image frame to the display device for playback.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0016] The memory stores computer-executed instructions;

[0017] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the special effects rendering method as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0020] The special effects rendering method, apparatus, device, computer-readable storage medium, and product provided in this embodiment, after obtaining the image frame to be rendered and the identification information of the target special effects corresponding to the virtual reality live broadcast, use a graphics processing unit (GPU) combination to perform special effects rendering operations on the image frame to be rendered in a cloud server. This eliminates the need for the viewer to have high-level hardware. Furthermore, using a GPU combination to perform special effects rendering operations on the image frame to be rendered effectively improves the content quality of the rendered target image frame. This, in turn, enhances the user experience for viewers of virtual reality live broadcasts. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a system architecture diagram on which this disclosure is based;

[0023] Figure 2 A flowchart illustrating the special effects rendering method provided in this embodiment of the disclosure;

[0024] Figure 3 A flowchart illustrating a special effects rendering method provided in yet another embodiment of this disclosure;

[0025] Figure 4 A flowchart illustrating a special effects rendering method provided in yet another embodiment of this disclosure;

[0026] Figure 5 This is a schematic diagram of special effects rendering provided for an embodiment of the present disclosure;

[0027] Figure 6 A flowchart illustrating a special effects rendering method provided in yet another embodiment of this disclosure;

[0028] Figure 7 This is a schematic diagram of the structure of the special effects rendering device provided in the embodiments of this disclosure;

[0029] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0031] To address the technical problems of existing special effects rendering methods that place high demands on the hardware of the viewing end and put significant pressure on the resource consumption and energy consumption of the viewing end, this disclosure provides a special effects rendering method, apparatus, device, computer-readable storage medium, and product.

[0032] It should be noted that the special effects rendering methods, apparatus, devices, computer-readable storage media and products provided in this disclosure can be applied in any special effects rendering application scenario.

[0033] Current special effects rendering methods typically involve sending the image frames to be rendered for the virtual reality live stream to the viewing client, where the rendering is then performed. This often places high demands on the viewing client's hardware and can lead to excessive data processing pressure, impacting its performance.

[0034] In the process of solving the above-mentioned technical problems, the inventors discovered through research that in order to avoid the adverse effects of special effects rendering operations on the viewing end and improve the live streaming effect of virtual reality live streaming, special effects rendering operations can be performed on the image frames to be rendered corresponding to the virtual reality live streaming through the cloud server, and the rendered target image frames can be sent to the viewing end for playback.

[0035] Furthermore, to ensure the quality of virtual reality live streaming, cloud servers can be equipped with multiple high-performance graphics processors (GPUs). Therefore, during special effects rendering, a graphics processing combination consisting of at least one GPU can be selected to perform the rendering operation. This allows for the complete presentation of various special effects without being limited by device performance constraints.

[0036] Figure 1 The system architecture diagram on which this disclosure is based is as follows: Figure 1 As shown, the system architecture upon which this disclosure is based includes at least a server 11, a binocular image acquisition device 12, a display device 13, and a special effects server 14. The server 11 is equipped with a special effects rendering device, which can be written in languages ​​such as C / C++, Java, Shell, or Python. The display device 13 can be a VR headset, mobile phone, tablet, or other display device capable of displaying live content.

[0037] Based on the above system architecture, server 11 can acquire the image frames to be rendered corresponding to the virtual reality live broadcast acquired by binocular image acquisition device 12, as well as the identification information of the target special effects sent by special effects server 14. Then, based on the image frames to be rendered and the identification information of the target special effects, a graphics processor combination can be used to perform special effects rendering operations to obtain the target image frame. The target image frame is then sent to display device 13 for playback.

[0038] Figure 2 This is a flowchart illustrating the special effects rendering method provided in the embodiments of this disclosure, as shown below. Figure 2 As shown, the method includes:

[0039] Step 201: Obtain the image frames to be rendered and the identification information of the target effects corresponding to the virtual reality live broadcast.

[0040] In this embodiment, the execution entity is a special effects rendering device, which can be coupled to a cloud server. This cloud server can communicate with both a binocular image acquisition device and a display device. Therefore, it can perform special effects rendering operations based on the image frames to be rendered corresponding to the virtual reality live stream captured by the binocular image acquisition device.

[0041] In this embodiment, in order to avoid putting computational pressure on the display device of the viewing terminal and to ensure the effect of special effects rendering, the special effects rendering operation can be performed on the cloud server.

[0042] Optionally, the special effects rendering device can acquire the image frame to be rendered corresponding to the virtual reality live stream and the identification information of the target special effects. The image frame to be rendered corresponding to the virtual reality live stream can be acquired by a binocular image acquisition device, and the identification information of the target special effects can be sent by a preset server based on preset trigger operations by the live stream audience or broadcaster.

[0043] It should be noted that this server can specifically be a data server used to store the special effects data corresponding to the special effects.

[0044] Step 202: Determine the rendering data corresponding to the target effect and the graphics processor combination used to render the image frame to be rendered based on the identification information of the target effect.

[0045] In this embodiment, since virtual reality live streaming generally uses high-resolution video frames, multiple high-performance graphics processors can be pre-configured to ensure the effectiveness of special effects processing. During special effects processing, these multiple graphics processors can be flexibly configured according to the actual application requirements to achieve high-quality special effects rendering.

[0046] After obtaining the identification information of the target effect, the rendering data corresponding to the target effect can be obtained based on the identification information. Furthermore, the graphics processor combination required to render the image frame to be rendered can be determined based on the rendering data, wherein the graphics processor combination may include at least two graphics processors.

[0047] Step 203: The graphics processor combination performs a rendering operation on the image frame to be rendered according to the rendering data to obtain the rendered target image frame.

[0048] In this embodiment, after determining the graphics processor combination required to render the image frame to be rendered, the graphics processor combination can perform rendering operations on the image frame to be rendered based on the rendering data to obtain the rendered target image frame. By using a graphics processor combination for special effects rendering operations, the effects of various required special effects can be fully presented without being limited by device performance. This effectively improves the quality of virtual reality live streaming while avoiding computational pressure on the viewing end.

[0049] Step 204: Send the target image frame to the display device for playback.

[0050] In this embodiment, after the rendering operation of the image frame to be rendered is completed in the cloud server, the target image frame can be sent to a display device for playback. This display device includes, but is not limited to, virtual reality devices, mobile phones, tablets, televisions, etc., that have a display interface.

[0051] This allows the display device to watch high-quality live content simply by playing the target image frames without needing to perform special effects rendering.

[0052] One feasible approach is to use a virtual reality (VR) device. To conserve the VR device's computing power while ensuring high-quality live streaming, the VR device can connect to a cloud server. The cloud server then selects a graphics processor combination to render the live content corresponding to the VR broadcast. The rendered target video stream is then sent to the VR device for playback.

[0053] Furthermore, based on any of the above embodiments, step 201 includes:

[0054] The system acquires the image frames to be rendered corresponding to the virtual reality live stream captured by the binocular image acquisition device, and acquires the target special effects sent by the server, wherein the target special effects are determined when the user triggers a virtual resource transfer operation on the server.

[0055] In this embodiment, when a user watches live content on the streaming client, they can trigger virtual resource transfer operations according to their actual needs. For example, a user can send a small gift to the live stream through a preset trigger operation. Different virtual resources may correspond to different special effects.

[0056] The cloud server can also communicate with pre-defined servers. When a user triggers a virtual resource transfer operation on the server, the server can send the identifier information of the target effect to the cloud server. Accordingly, the cloud server can obtain the image frames to be rendered corresponding to the virtual reality live stream captured by the binocular image acquisition device, and also obtain the target effect sent by the server.

[0057] The special effects rendering method provided in this embodiment, after obtaining the image frame to be rendered and the identification information of the target special effects corresponding to the virtual reality live broadcast, uses a graphics processing unit (GPU) combination to perform special effects rendering operations on the image frame to be rendered in a cloud server. This eliminates the need for the viewer to have high-level hardware. Furthermore, using a GPU combination to perform special effects rendering operations on the image frame to be rendered can effectively improve the content quality of the rendered target image frame. This, in turn, enhances the user experience for viewers of virtual reality live broadcasts.

[0058] Furthermore, based on any of the above embodiments, step 202 includes:

[0059] The rendering data corresponding to the target effect is obtained based on the identification information of the target effect. The rendering data includes the effect type, scene type, and algorithm calling information corresponding to the target effect.

[0060] The graphics processor combination used for rendering special effects is determined based on the rendering data.

[0061] In this embodiment, after obtaining the identification information of the target effect, the rendering data corresponding to the target effect can be obtained based on the identification information of the target effect. The rendering data includes the effect type, scene type, and algorithm calling information corresponding to the target effect.

[0062] Based on the above rendering data, the graphics processor combination corresponding to the target effect can be determined according to the preset mapping relationship between rendering data and graphics processors.

[0063] Optionally, users can also customize the selection of multiple graphics processors according to their actual needs, and this disclosure does not impose any restrictions on this. For example, users can select at least two graphics processors as the graphics processor combination.

[0064] The special effects rendering method provided in this embodiment determines the graphics processor combination used for special effects rendering based on the rendering data corresponding to the target special effects. This enables the special effects rendering operation of the image frame to be rendered to be achieved through the high-performance graphics processor combination, thereby improving the rendering effect of special effects rendering, optimizing the live broadcast effect of virtual reality live broadcast, and enhancing the user experience.

[0065] Furthermore, based on any of the above embodiments, the graphics processor combination includes at least two graphics processors. Step 203 includes:

[0066] The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination.

[0067] The preprocessed image frame to be rendered is processed by transcoding algorithm and encoded by at least one other graphics processor in the graphics processor combination.

[0068] In this embodiment, when at least two graphics processors are used for special effects processing, data transfer operations between the graphics processors are inevitable. Since the resolution of the image frames to be rendered in virtual reality live streaming is high, excessive data transfer operations will reduce the efficiency of special effects processing.

[0069] Therefore, in order to improve the efficiency of special effects processing while ensuring the quality of the effects, decoding and special effects can use the same graphics processor, while other operations can use different graphics processors.

[0070] Optionally, the image frame to be rendered can be preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination. The preprocessed image frame can then be transcoded and encoded by at least one other graphics processor in the graphics processor combination.

[0071] For example, in a practical application, when a graphics processing unit (GPU) combination includes two GPUs, one GPU can perform decoding, effects algorithms, and effects rendering, while the other GPU performs encoding. When a GPU combination includes three GPUs, one GPU can perform decoding, effects algorithms, and effects rendering, another GPU can perform transcoding algorithms, and a third GPU can perform encoding.

[0072] The special effects rendering method provided in this embodiment uses the same graphics processor for decoding and special effects processing, and uses other graphics processors for other operations. This can improve the special effects rendering effect while minimizing cross-card data transfer and improving the efficiency of special effects rendering.

[0073] Figure 3 This is a flowchart illustrating a special effects rendering method according to another embodiment of the present disclosure. Based on any of the above embodiments, the number of target special effects is at least one. For example... Figure 3 As shown, step 203 includes:

[0074] Step 301: For each target effect, determine the rendering time range corresponding to the target effect based on the rendering data corresponding to the target effect.

[0075] Step 302: Render the target effect according to the rendering time range corresponding to each target effect.

[0076] In this embodiment, during the actual live broadcast, the number of target effects corresponding to the virtual reality live broadcast can be at least one. Different target effects correspond to different rendering time ranges.

[0077] Therefore, in order to achieve accurate rendering of the target effects, for each target effect, a rendering time range corresponding to the target effect is determined based on the rendering data corresponding to the target effect. Within this rendering time range, the rendering operation is performed on the image frame to be rendered based on the rendering data corresponding to the target effect.

[0078] The special effects rendering method provided in this embodiment determines the rendering time range of each target special effect and performs rendering operations on the target special effect within the corresponding rendering time range, thereby accurately realizing the special effects rendering operation and further improving the rendering effect of the special effects rendering operation.

[0079] Figure 4 This is a flowchart illustrating a special effects rendering method according to another embodiment of this disclosure. Based on any of the above embodiments, the image frame to be rendered is an encoded image frame; as shown... Figure 4 As shown, step 203 includes:

[0080] Step 401: The graphics processor combination decodes the image frame to be rendered to obtain the decoded image frame to be rendered.

[0081] Step 402: The graphics processor unit identifies the rendering information corresponding to the decoded image frame to be rendered, wherein the rendering information includes the target rendering area and depth information.

[0082] Step 403: The graphics processor performs a rendering operation on the image frame to be rendered based on the rendering information and the rendering data to obtain the rendered target image frame.

[0083] In this embodiment, the image frame to be rendered can specifically be an encoded image frame. Therefore, after obtaining the image frame to be rendered, a decoding operation can be performed on the image frame to be rendered to obtain the decoded image frame to be rendered.

[0084] To achieve precise rendering of target effects, after obtaining the decoded image frame to be rendered, the graphics processing unit (GPU) can further identify the rendering information corresponding to the decoded image frame. This rendering information includes the target rendering area and depth information. Based on this rendering information, the location and depth of the effect to be rendered can be determined. Therefore, after identifying the rendering information, the GPU can perform rendering operations on the image frame to be rendered based on the rendering information and rendering data to obtain the rendered target image frame.

[0085] Optionally, the rendering information corresponding to the decoded image frame to be rendered can be identified through a preset depth recognition model and a preset region segmentation model.

[0086] Alternatively, any method of region recognition and depth recognition can be used to identify the rendered information, and this disclosure does not impose any restrictions on this.

[0087] Furthermore, based on any of the above embodiments, step 402 includes:

[0088] Perform fisheye image processing on the decoded image frame to be rendered to obtain an image frame with a fisheye effect.

[0089] Identify the rendering information corresponding to the image frame to be rendered that has the fisheye effect.

[0090] In this embodiment, to improve the accuracy of rendering information recognition, a fisheye image processing operation is first performed on the decoded image frame to be rendered to obtain an image frame with a fisheye effect. The graphics processor then identifies the rendering information corresponding to the fisheye effect image frame.

[0091] Furthermore, based on any of the above embodiments, after step 402, the method further includes:

[0092] Generate a depth texture map corresponding to the image frame to be rendered based on the rendering information corresponding to the image frame to be rendered.

[0093] The step of rendering the image frame to be rendered by combining the graphics processor with the rendering information and the rendering data to obtain the rendered target image frame includes:

[0094] The depth texture map is rendered based on the rendering data to obtain the rendered image frame.

[0095] In this embodiment, after identifying the rendering information corresponding to the decoded image frame to be rendered, a depth texture map corresponding to the image frame to be rendered can be generated based on the rendering information. This depth texture map can be a visual texture map, based on which the brightness variations, target rendering areas, and other content in the image frame to be rendered can be determined. The rendering operation is then performed on this depth texture map according to the rendering data to obtain the rendered image frame.

[0096] Figure 5 This is a schematic diagram of special effects rendering provided for embodiments of this disclosure, such as... Figure 5 As shown, after obtaining the image frame 51 to be rendered, a decoding operation can be performed on the image frame 51 to obtain the decoded image frame 52 to be rendered. A fisheye image processing operation is then performed on the decoded image frame 52 to obtain the fisheye-effect image frame 53 to be rendered. For each image frame 54 in the fisheye-effect image frame 53 to be rendered, the rendering information corresponding to that image frame is identified, and a depth texture map 54 corresponding to that image frame is generated based on the rendering information. A rendering operation is then performed on the depth texture map 54 based on the rendering data to obtain the rendered image frame 55.

[0097] The special effects rendering method provided in this embodiment identifies the depth information and target rendering area in the image frame after acquiring it, and constructs a depth texture map based on the depth information and target rendering area. This enables accurate special effects rendering operations based on the depth texture map, thereby improving the effect of special effects rendering.

[0098] Figure 6 This is a flowchart illustrating a special effects rendering method according to another embodiment of the present disclosure. Based on any of the above embodiments, the number of display devices used to play the target image frame is multiple. For example... Figure 6 As shown, step 203 includes:

[0099] Step 601: Determine the resolution corresponding to each display device.

[0100] Step 602: Decode the image frame to be rendered by any one of the graphics processors in the graphics processor combination, perform algorithm recognition and special effects rendering operations, and obtain the preprocessed image frame to be rendered.

[0101] Step 603: Use a graphics processor corresponding to the resolution of the display device to perform transcoding algorithm processing and encoding operation on the preprocessed image frame to be rendered.

[0102] In this embodiment, there are multiple display devices used for playing target image frames in virtual reality live streaming. These different display devices have different resolutions. For example, if the display device is a mobile phone, the resolution is generally 1080P, while if the display device is a virtual reality device, it is generally 8K resolution. To ensure that the target image frames after special effects rendering can adapt to different display devices, different graphics processors can be used for transcoding algorithms and encoding operations for different virtual reality devices.

[0103] Optionally, the resolution corresponding to each display device can be determined. The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination. For different display devices, a graphics processor corresponding to the resolution of the display device is used to perform transcoding algorithm processing and encoding operations on the preprocessed image frame to be rendered.

[0104] The special effects rendering method provided in this embodiment uses different graphics processors to perform special effects rendering operations for different display device resolutions, thereby enabling multi-channel output of live content at different resolutions and increasing the applicable scenarios for virtual reality live streaming.

[0105] Figure 7This is a schematic diagram of the structure of the special effects rendering device provided in the embodiments of this disclosure, as shown below. Figure 7 As shown, the device includes: an acquisition module 71, a determination module 72, a rendering module 73, and a sending module 74. The acquisition module 71 is used to acquire the image frame to be rendered and the identification information of the target effect corresponding to the virtual reality live stream. The determination module 72 is used to determine the rendering data corresponding to the target effect and the graphics processor combination for rendering the image frame to be rendered based on the identification information of the target effect. The rendering module 73 is used to perform a rendering operation on the image frame to be rendered according to the rendering data using the graphics processor combination to obtain the rendered target image frame. The sending module 74 is used to send the target image frame to a display device for playback.

[0106] Furthermore, based on any of the above embodiments, the acquisition module is used to: acquire the image frame to be rendered corresponding to the virtual reality live broadcast acquired by the binocular image acquisition device, and acquire the target effect sent by the server, wherein the target effect is determined by the user when triggering a virtual resource transfer operation on the server.

[0107] Furthermore, based on any of the above embodiments, the determining module is configured to: obtain rendering data corresponding to the target effect based on the identification information of the target effect, wherein the rendering data includes the effect type, scene type, and algorithm calling information corresponding to the target effect; and determine the graphics processor combination used for effect rendering based on the rendering data.

[0108] Furthermore, based on any of the above embodiments, the number of target effects is at least one. The rendering module is configured to: for each target effect, determine the rendering time range corresponding to the target effect based on the rendering data corresponding to the target effect; and perform rendering operations on the target effect according to the rendering time range corresponding to each target effect.

[0109] Further, based on any of the above embodiments, the image frame to be rendered is an encoded image frame. The rendering module is configured to: decode the image frame to be rendered using the graphics processing unit (GPU) to obtain a decoded image frame to be rendered; identify rendering information corresponding to the decoded image frame to be rendered using the GPU, wherein the rendering information includes a target rendering area and depth information; and perform rendering operations on the image frame to be rendered using the GPU based on the rendering information and the rendering data to obtain a rendered target image frame.

[0110] Furthermore, based on any of the above embodiments, the rendering module is configured to: perform fisheye image processing on the decoded image frame to be rendered to obtain an image frame to be rendered with a fisheye effect; and identify the rendering information corresponding to the image frame to be rendered with the fisheye effect.

[0111] Furthermore, based on any of the above embodiments, the rendering module is configured to: generate a depth texture map corresponding to the image frame to be rendered based on the rendering information corresponding to the image frame to be rendered; and perform a rendering operation on the depth texture map based on the rendering data to obtain the rendered image frame.

[0112] Furthermore, based on any of the above embodiments, the rendering module is used to: identify the rendering information corresponding to the decoded image frame to be rendered through a preset depth recognition model and a preset region segmentation model.

[0113] Furthermore, based on any of the above embodiments, the graphics processor combination includes at least two graphics processors. The rendering module is used to: perform decoding, algorithm recognition, and special effects rendering operations on the image frame to be rendered using any one of the graphics processors in the graphics processor combination to obtain a preprocessed image frame to be rendered. Then, it performs transcoding algorithm processing and encoding operations on the preprocessed image frame to be rendered using at least one other graphics processor in the graphics processor combination.

[0114] Furthermore, based on any of the above embodiments, the number of display devices used to play the target image frame is multiple. The rendering module is used to: determine the resolution corresponding to each display device; perform decoding, algorithm recognition, and special effects rendering operations on the image frame to be rendered using any one of the graphics processors in the graphics processor combination to obtain a preprocessed image frame to be rendered; and perform transcoding algorithm processing and encoding operations on the preprocessed image frame to be rendered using a graphics processor corresponding to the resolution of the display device.

[0115] Furthermore, based on any of the above embodiments, the display device includes at least a virtual reality device.

[0116] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0117] To implement the above embodiments, this disclosure also provides an electronic device, including a processor and a memory.

[0118] The memory stores computer-executed instructions.

[0119] The processor executes computer execution instructions stored in the memory, causing the processor to perform the special effects rendering method as described in any of the above embodiments.

[0120] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. The electronic device 800 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0121] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0122] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0123] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0124] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0125] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the special effects rendering method as described in any of the above embodiments.

[0126] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the special effects rendering method as described in any of the above embodiments.

[0127] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0128] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0129] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0132] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0133] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0134] In a first aspect, according to one or more embodiments of this disclosure, a special effects rendering method is provided, comprising:

[0135] Obtain the image frames to be rendered and the identification information of the target effects corresponding to the virtual reality live broadcast;

[0136] The rendering data corresponding to the target effect and the graphics processor combination used to render the image frame to be rendered are determined based on the identification information of the target effect.

[0137] The graphics processor combination performs a rendering operation on the image frame to be rendered according to the rendering data to obtain the rendered target image frame.

[0138] The target image frame is sent to a display device for playback.

[0139] According to one or more embodiments of this disclosure, obtaining the image frame to be rendered and the identification information of the target special effect corresponding to the virtual reality live broadcast includes:

[0140] The system acquires the image frames to be rendered corresponding to the virtual reality live stream captured by the binocular image acquisition device, and acquires the target special effects sent by the server, wherein the target special effects are determined when the user triggers a virtual resource transfer operation on the server.

[0141] According to one or more embodiments of this disclosure, determining the rendering data corresponding to the target effect and the graphics processor combination for rendering the effect based on the identification information of the target effect includes:

[0142] The rendering data corresponding to the target effect is obtained based on the identification information of the target effect. The rendering data includes the effect type, scene type, and algorithm calling information corresponding to the target effect.

[0143] The graphics processor combination used for rendering special effects is determined based on the rendering data.

[0144] According to one or more embodiments of this disclosure, the number of target effects is at least one; the step of using the graphics processor to perform rendering operations on the image frame to be rendered based on the rendering data to obtain the rendered target image frame includes:

[0145] For each target effect, the rendering time range corresponding to the target effect is determined based on the rendering data corresponding to the target effect;

[0146] The target effect is rendered according to the rendering time range corresponding to each target effect.

[0147] According to one or more embodiments of this disclosure, the image frame to be rendered is an encoded image frame; the step of performing a rendering operation on the image frame to be rendered based on the rendering data using the graphics processor to obtain a rendered target image frame includes:

[0148] The graphics processor combination decodes the image frame to be rendered to obtain the decoded image frame to be rendered.

[0149] The graphics processor unit identifies the rendering information corresponding to the decoded image frame to be rendered, wherein the rendering information includes the target rendering area and depth information;

[0150] The graphics processor combines the graphics processing unit to perform rendering operations on the image frame to be rendered based on the rendering information and the rendering data, thereby obtaining the rendered target image frame.

[0151] According to one or more embodiments of this disclosure, the step of identifying rendering information corresponding to the decoded image frame to be rendered by the graphics processor combination includes:

[0152] Perform fisheye image processing on the decoded image frame to be rendered to obtain an image frame to be rendered with a fisheye effect;

[0153] Identify the rendering information corresponding to the image frame to be rendered that has the fisheye effect.

[0154] According to one or more embodiments of this disclosure, after the graphics processor identifies the rendering information corresponding to the decoded image frame to be rendered, the method further includes:

[0155] Generate a depth texture map corresponding to the image frame to be rendered based on the rendering information corresponding to the image frame to be rendered;

[0156] The step of rendering the image frame to be rendered by combining the graphics processor with the rendering information and the rendering data to obtain the rendered target image frame includes:

[0157] The depth texture map is rendered based on the rendering data to obtain a rendered image frame.

[0158] According to one or more embodiments of this disclosure, the step of identifying rendering information corresponding to the decoded image frame to be rendered by the graphics processor combination includes:

[0159] The rendering information corresponding to the decoded image frame to be rendered is identified by a preset depth recognition model and a preset region segmentation model.

[0160] According to one or more embodiments of this disclosure, the graphics processor combination includes at least two graphics processors;

[0161] The step of rendering the image frame to be rendered based on the rendering data using the graphics processor to obtain the rendered target image frame includes:

[0162] The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination.

[0163] The preprocessed image frame to be rendered is processed by transcoding algorithm and encoded by at least one other graphics processor in the graphics processor combination.

[0164] According to one or more embodiments of this disclosure, the number of display devices for playing target image frames is multiple;

[0165] The step of rendering the image frame to be rendered based on the rendering data using the graphics processor to obtain the rendered target image frame includes:

[0166] Determine the resolution corresponding to each display device;

[0167] The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination.

[0168] The preprocessed image frame to be rendered is processed by transcoding algorithm and encoded using a graphics processor corresponding to the resolution of the display device.

[0169] Furthermore, based on any of the above embodiments, the display device includes at least a virtual reality device.

[0170] Secondly, according to one or more embodiments of this disclosure, a special effects rendering apparatus is provided, comprising:

[0171] The acquisition module is used to acquire the image frames to be rendered and the identification information of the target effects corresponding to the virtual reality live broadcast;

[0172] The determining module is used to determine the rendering data corresponding to the target effect and the graphics processor combination used to render the image frame to be rendered based on the identification information of the target effect;

[0173] The rendering module is used to perform rendering operations on the image frame to be rendered according to the rendering data through the graphics processor combination to obtain the rendered target image frame.

[0174] The sending module is used to send the target image frame to the display device for playback.

[0175] According to one or more embodiments of this disclosure, the acquisition module is used to:

[0176] The system acquires the image frames to be rendered corresponding to the virtual reality live stream captured by the binocular image acquisition device, and acquires the target special effects sent by the server, wherein the target special effects are determined when the user triggers a virtual resource transfer operation on the server.

[0177] According to one or more embodiments of this disclosure, the determining module is used to:

[0178] The rendering data corresponding to the target effect is obtained based on the identification information of the target effect. The rendering data includes the effect type, scene type, and algorithm calling information corresponding to the target effect.

[0179] The graphics processor combination used for rendering special effects is determined based on the rendering data.

[0180] According to one or more embodiments of this disclosure, the number of the target special effects is at least one; the rendering module is used for:

[0181] For each target effect, the rendering time range corresponding to the target effect is determined based on the rendering data corresponding to the target effect;

[0182] The target effect is rendered according to the rendering time range corresponding to each target effect.

[0183] According to one or more embodiments of this disclosure, the image frame to be rendered is an encoded image frame; the rendering module is used for:

[0184] The graphics processor combination decodes the image frame to be rendered to obtain the decoded image frame to be rendered.

[0185] The graphics processor unit identifies the rendering information corresponding to the decoded image frame to be rendered, wherein the rendering information includes the target rendering area and depth information;

[0186] The graphics processor combines the graphics processing unit to perform rendering operations on the image frame to be rendered based on the rendering information and the rendering data, thereby obtaining the rendered target image frame.

[0187] According to one or more embodiments of this disclosure, the rendering module is used for:

[0188] Perform fisheye image processing on the decoded image frame to be rendered to obtain an image frame to be rendered with a fisheye effect;

[0189] Identify the rendering information corresponding to the image frame to be rendered that has the fisheye effect.

[0190] According to one or more embodiments of this disclosure, the rendering module is used for:

[0191] Generate a depth texture map corresponding to the image frame to be rendered based on the rendering information corresponding to the image frame to be rendered;

[0192] The depth texture map is rendered based on the rendering data to obtain the rendered image frame.

[0193] According to one or more embodiments of this disclosure, the rendering module is used for:

[0194] The rendering information corresponding to the decoded image frame to be rendered is identified by a preset depth recognition model and a preset region segmentation model.

[0195] According to one or more embodiments of this disclosure, the graphics processor combination includes at least two graphics processors;

[0196] The rendering module is used for:

[0197] The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination.

[0198] The preprocessed image frame to be rendered is processed by transcoding algorithm and encoded by at least one other graphics processor in the graphics processor combination.

[0199] According to one or more embodiments of this disclosure, the number of display devices for playing target image frames is multiple;

[0200] The rendering module is used for:

[0201] Determine the resolution corresponding to each display device;

[0202] The image frame to be rendered is preprocessed by performing decoding, algorithm recognition, and special effects rendering operations on any of the graphics processors in the graphics processor combination.

[0203] The preprocessed image frame to be rendered is processed by transcoding algorithm and encoded using a graphics processor corresponding to the resolution of the display device.

[0204] Furthermore, based on any of the above embodiments, the display device includes at least a virtual reality device.

[0205] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0206] The memory stores computer-executed instructions;

[0207] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the special effects rendering method as described in the first aspect and various possible designs of the first aspect.

[0208] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the special effects rendering method described in the first aspect and various possible designs of the first aspect is implemented.

[0209] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0210] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0211] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0212] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method of special effect rendering, characterized by, The method comprises the steps of: acquiring an image frame to be rendered corresponding to a virtual reality live broadcast and identification information of a target special effect; acquiring rendering data corresponding to the target special effect according to the identification information of the target special effect, and determining a combination of graphic processors for special effect rendering according to the rendering data; performing a rendering operation on the image frame to be rendered according to the rendering data by using the combination of graphic processors, to obtain a target image frame after rendering; sending the target image frame to a display device for playing; if the combination of graphic processors comprises at least two graphic processors, the step of performing a rendering operation on the image frame to be rendered according to the rendering data by using the combination of graphic processors, to obtain a target image frame after rendering, comprises the steps of: performing a decoding operation, an algorithm identification operation and a special effect rendering operation on the image frame to be rendered by using any graphic processor in the combination of graphic processors, to obtain a preprocessed image frame to be rendered; performing transcoding algorithm processing and an encoding operation on the preprocessed image frame to be rendered by using at least one other graphic processor in the combination of graphic processors.

2. The method of claim 1, wherein, The step of acquiring an image frame to be rendered corresponding to a virtual reality live broadcast and identification information of a target special effect comprises the steps of: acquiring an image frame to be rendered corresponding to a virtual reality live broadcast collected by a binocular image acquisition device, and acquiring a target special effect sent by a server, wherein the target special effect is determined when a user triggers a virtual resource transfer operation on the server.

3. The method of claim 1, wherein, The rendering data comprises a special effect type, a scene type and calling algorithm information corresponding to the target special effect.

4. The method of claim 1, wherein, The number of target special effects is at least one, and the step of performing a rendering operation on the image frame to be rendered according to the rendering data by using the combination of graphic processors, to obtain a target image frame after rendering, comprises the steps of: for each target special effect, determining a rendering time range corresponding to the target special effect according to the rendering data corresponding to the target special effect; performing a rendering operation on each target special effect according to the rendering time range corresponding to the target special effect.

5. The method according to any one of claims 1 to 4, characterized in that, The image frame to be rendered is an encoded image frame. The step of performing a rendering operation on the image frame to be rendered according to the rendering data by using the combination of graphic processors, to obtain a target image frame after rendering, comprises the steps of: performing a decoding operation on the image frame to be rendered by using the combination of graphic processors, to obtain a decoded image frame to be rendered; identifying rendering information corresponding to the decoded image frame to be rendered by using the combination of graphic processors, wherein the rendering information comprises a target rendering area and depth information; performing a rendering operation on the image frame to be rendered according to the rendering information and the rendering data by using the combination of graphic processors, to obtain a target image frame after rendering.

6. The method of claim 5, wherein, The step of identifying rendering information corresponding to the decoded image frame to be rendered by using the combination of graphic processors comprises the steps of: performing a fisheye image processing operation on the decoded image frame to be rendered, to obtain a fisheye effect image frame to be rendered; identifying rendering information corresponding to the fisheye effect image frame to be rendered.

7. The method of claim 5, wherein, After the step of identifying rendering information corresponding to the decoded image frame to be rendered by using the combination of graphic processors, the method further comprises the steps of: generate a depth texture map corresponding to the to-be-rendered image frame according to rendering information corresponding to the to-be-rendered image frame; the rendering operation on the to-be-rendered image frame according to the rendering data by the combination of the plurality of graphic processors to obtain a rendered target image frame, comprises: rendering operation on the depth texture map according to the rendering data to obtain a rendered image frame.

8. The method of claim 5, wherein, the combination of the plurality of graphic processors to identify the rendering information corresponding to the decoded to-be-rendered image frame, comprises: identifying the rendering information corresponding to the decoded to-be-rendered image frame by a preset depth identification model and a preset region segmentation model.

9. The method according to any one of claims 1 to 4, characterized in that, The number of display devices for playing the target image frame is multiple. the rendering operation on the to-be-rendered image frame according to the rendering data by the combination of the plurality of graphic processors to obtain a rendered target image frame, comprises: determining the resolution corresponding to each display device; the combination of the plurality of graphic processors to identify the rendering information corresponding to the decoded to-be-rendered image frame, comprises: respectively adopting a graphic processor corresponding to the resolution corresponding to the display device to perform transcoding algorithm processing and encoding operation on the preprocessed to-be-rendered image frame.

10. The method according to any one of claims 1 to 4, characterized in that, The display device at least includes a virtual reality device.

11. A special effect rendering apparatus characterized by comprising: comprises: an acquisition module, configured to acquire to-be-rendered image frames corresponding to a virtual reality live broadcast and identification information of a target special effect; a determination module, configured to acquire rendering data corresponding to the target special effect according to the identification information of the target special effect, and determine a combination of graphic processors for special effect rendering according to the rendering data; a rendering module, configured to perform rendering operation on the to-be-rendered image frames according to the rendering data by the combination of the graphic processors to obtain a rendered target image frame; a sending module, configured to send the target image frame to a display device for playing; if the combination of the graphic processors comprises at least two graphic processors, the rendering module is specifically configured to perform decoding operation, algorithm identification operation and special effect rendering operation on the to-be-rendered image frames by any one of the graphic processors in the combination of the graphic processors to obtain preprocessed to-be-rendered image frames; perform transcoding algorithm processing and encoding operation on the preprocessed to-be-rendered image frames by other at least one graphic processor in the combination of the graphic processors.

12. An electronic device, comprising: comprises: a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, so that the processor executes the special effect rendering method in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the special effect rendering method in any one of claims 1 to 10 is realized.

14. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to realize the special effect rendering method in any one of claims 1 to 10.

Images