Livestreaming special effect rendering method and apparatus, device, readable storage medium, and product

Abstract

Provided are a livestreaming special effect rendering method and apparatus, a device, a readable storage medium, and a product. The method comprises: acquiring a livestreaming image frame corresponding to virtual reality livestreaming content and a preset target special effect (201); determining key point information corresponding to at least some of target objects in the livestreaming image frame (202); performing a special effect rendering operation on the livestreaming image frame according to the target special effect and the key point information to obtain a target image frame (203); and displaying the target image frame (204). Therefore, a special effect processing region can be concentrated at a position associated with the key point information, thereby effectively narrowing the region requiring special effect processing, and improving the special effect processing efficiency.

Description

Live broadcast special effects rendering method, device, equipment, readable storage medium and product

[0001] This application claims priority to the Chinese invention patent application entitled “Live broadcast special effects rendering method, device, equipment, readable storage medium and product” filed on December 15, 2022, with application number 202211612984X. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to a live broadcast special effects rendering method, apparatus, device, readable storage medium, and product. Background Art

[0003] VR panoramic live broadcasts typically use real-time capture with a binocular camera. VR live broadcasts typically use 8K video frames (7680*4320) or higher. Compared to traditional 2K (2048*unspecified value) and 720P (1280*720), these are ultra-high-definition video frames. Due to latency requirements during live broadcasts, the actual time available for special effects rendering is relatively short. Therefore, the 8K image algorithm and special effects rendering must be completed within a limited timeframe to ensure a good VR live broadcast experience. Ensuring fast special effects rendering during VR live broadcasts has become a pressing technical challenge.

[0004] Summary of the Invention

[0005] The embodiments of the present disclosure provide a live broadcast special effects rendering method, apparatus, device, readable storage medium, and product, which are used to solve the technical problem that in VR live broadcast scenarios, the special effects rendering speed of collected ultra-high-definition video frames is slow and the live broadcast effect cannot be guaranteed.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for rendering special effects for a live broadcast, comprising:

[0007] Obtain the live image frames corresponding to the virtual reality live content and the preset target special effects;

[0008] Determining key point information corresponding to at least part of the target object in the live image frame;

[0009] Performing a special effects rendering operation on the live image frame according to the target special effects and the key point information to obtain a target image frame;

[0010] The target image frame is displayed.

[0011] In a second aspect, an embodiment of the present disclosure provides a live broadcast special effects rendering device, comprising:

[0012] An acquisition module is used to obtain live image frames corresponding to virtual reality live content and preset target special effects;

[0013] A determination module, configured to determine key point information corresponding to at least part of the target object in the live image frame;

[0014] A rendering module, configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame;

[0015] A display module is configured to display the target image frame.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0017] The memory stores computer-executable instructions;

[0018] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the live broadcast special effects rendering method described in the first aspect and various possible designs of the first aspect is implemented.

[0020] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the live broadcast special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0021] The live special effects rendering method, apparatus, device, readable storage medium, and product provided in this embodiment determine key point information in a live image frame after obtaining a live image frame corresponding to the virtual reality live content, and perform special effects rendering operations on positions in the live image frame associated with the key point information based on the key point information. This allows the area for special effects processing to be concentrated at the position associated with the key point information, effectively reducing the area requiring special effects processing and thereby improving the efficiency of special effects processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] FIG1 is a schematic diagram of the system architecture on which the present disclosure is based;

[0024] FIG2 is a flow chart of a method for rendering special effects for live broadcasting provided by an embodiment of the present disclosure;

[0025] FIG3 is a schematic diagram of an external expansion according to an embodiment of the present disclosure;

[0026] FIG4 is a flow chart of a live broadcast special effects rendering method provided by another embodiment of the present disclosure;

[0027] FIG5 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0028] FIG6 is a flowchart of a live broadcast special effects rendering method provided by another embodiment of the present disclosure;

[0029] FIG7 is a flow chart of a live broadcast special effects rendering method provided by another embodiment of the present disclosure;

[0030] FIG8 is a schematic diagram of an interface interaction provided by an embodiment of the present disclosure;

[0031] FIG9 is another schematic diagram of an interface interaction provided by an embodiment of the present disclosure;

[0032] FIG10 is another schematic diagram of an interface interaction provided by an embodiment of the present disclosure;

[0033] FIG11 is a schematic structural diagram of a live broadcast special effects rendering device provided by an embodiment of the present disclosure;

[0034] FIG12 is a schematic structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0036] In order to solve the technical problem that the special effects rendering speed of the collected ultra-high-definition video frames in VR live broadcast scenarios is slow and the live broadcast effect cannot be guaranteed, the present disclosure provides a live broadcast special effects rendering method, device, equipment, readable storage medium and product.

[0037] It should be noted that the live broadcast special effects rendering method, device, equipment, readable storage medium and product provided by the present disclosure can be applied to image rendering scenarios in any VR scene.

[0038] Existing VR panoramic live broadcasts use binocular cameras for real-time capture, typically using 8K video frames (7680*4320) or higher. Compared to traditional 2K (2048*unspecified value) and 720P (1280*720), these are ultra-high-definition video frames. A single 8K frame weighs 7680*4320*4Byte = 126MB. Live broadcast frame rates typically require 30-60fps, with a single-frame latency capped at 16ms-33ms. The actual window for special effects rendering can be even shorter, requiring algorithms and special effects rendering for 8K images to be completed within a limited timeframe to ensure a positive VR live broadcast experience.

[0039] In the process of solving the above technical problems, the inventors found through research that in order to improve the speed of special effects rendering and ensure a good VR live broadcast experience, a graphics processor can be used for special effects rendering operations, and a central processing unit can be used for identification and detection operations. In order to further improve the speed of special effects rendering, the scope of special effects rendering can be concentrated on or around the host, thereby reducing the actual pixel area that needs to be processed. In addition, due to the large amount of data in the live image frame, it is necessary to avoid time-consuming data transmission between the CPU and the GPU. Therefore, before the live image frame obtained by the graphics processor is sent to the central processing unit, the live image frame can be compressed and / or cropped to reduce the amount of transmitted data and increase the transmission speed.

[0040] Figure 1 is a schematic diagram of the system architecture on which the present disclosure is based. As shown in Figure 1, the system architecture on which the present disclosure is based includes at least: a binocular image acquisition device 11 and a server 12, wherein the server 12 is provided with a graphics processor and a central processing unit, and the graphics processor and the central processing unit are provided with a live broadcast special effects rendering device, and the live broadcast special effects rendering device can be written in languages ​​such as C / C++, Java, Shell or Python.

[0041] FIG2 is a flow chart of a live broadcast special effects rendering method provided by an embodiment of the present disclosure. As shown in FIG2 , the method includes:

[0042] Step 201: Acquire live image frames corresponding to virtual reality live content and preset target special effects.

[0043] The execution subject of this embodiment is a live broadcast special effects rendering device, which can be coupled to a server, and the server is respectively provided with a graphics processor and a central processing unit.

[0044] In this embodiment, when users are broadcasting virtual reality (VR) live, they can select special effects, beautification, filters, and other content according to their actual needs to enhance the live broadcast effect. When the target special effect selected by the user is obtained, special effects rendering operations need to be performed on the live broadcast content according to the target special effect to achieve a decorative effect.

[0045] In VR live broadcasting, in order to ensure the live broadcast effect, a binocular image acquisition device is used to capture the live broadcast content. However, the live broadcast image frames captured by the binocular image acquisition device are usually 8k image frames (7680*4320) or above, which are large in size and the special effects rendering process takes a long time.

[0046] Accordingly, to implement special effects rendering for live content, the live special effects rendering device can acquire live image frames corresponding to the live VR content. The live image frames can be acquired at preset time intervals or at a preset frequency, which is not limited in this disclosure. The live image frames can be acquired by a binocular image acquisition device or other image acquisition device capable of capturing live VR content, which is not limited in this disclosure.

[0047] Accordingly, in order to implement special effects rendering operations on live broadcast image frames, a preset target special effect may also be obtained, and the target special effect may be selected by the user according to actual needs during the live broadcast process.

[0048] Step 202: Determine key point information corresponding to at least part of the target object in the live image frame.

[0049] In this embodiment, in order to increase the speed of special effects rendering and avoid lag in the live broadcast process, the special effects rendering operation can be concentrated around at least part of the target object in the live image frame, where the target object can be a person, animal, specific object, etc. in the live image frame.

[0050] Therefore, after acquiring a live image frame, key point information corresponding to at least a portion of the target object in the live image frame can be determined. Optionally, key point information corresponding to at least a portion of the target object in the live image frame can be determined based on a preset detection algorithm, wherein the key point information can specifically be coordinate information of key locations in the target object.

[0051] Step 203: Perform a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame.

[0052] In this embodiment, after obtaining the key point information, the target special effect can be used to perform special effect rendering on the live image frame based on the key point information to obtain the target image frame. This eliminates the need to perform special effect rendering on all positions of the live image frame, thereby improving the efficiency of special effect rendering while optimizing the display effect of the target object.

[0053] Step 204: Display the target image frame.

[0054] In this embodiment, after the special effects rendering operation on the live image frame is completed and the target image frame is obtained, the target image frame can be displayed.

[0055] Optionally, the target image frame is distributed to a preset terminal device for display. The preset terminal device may be at least part of a virtual reality device for viewing VR live broadcast, so that the user can view the virtual reality live broadcast through the virtual display device.

[0056] Alternatively, if the live special effects rendering device is coupled to a terminal device, it can directly control a preset display interface of the terminal device to display the target image frame.

[0057] It should be noted that since the server is equipped with a graphics processor and a central processing unit, the graphics processor can be used for special effects rendering operations, and the central processing unit can be used for key point identification. Alternatively, the central processing unit can be used for special effects rendering operations, and the graphics processor can be used for key point identification. This disclosure does not impose any restrictions on this.

[0058] The live special effects rendering method provided in this embodiment determines key point information in the live image frame after obtaining the live image frame corresponding to the virtual reality live content, and performs special effects rendering operations on the positions associated with the key point information in the live image frame based on the key point information. This can concentrate the area for special effects processing at the positions associated with the key point information, effectively reducing the area that needs to be processed by special effects, and thereby improving the efficiency of special effects processing.

[0059] In practice, different target effects may correspond to different display effects and, accordingly, different rendering locations. For example, effects such as beautification, facial stickers, and head decorations can be rendered on the face or head. However, effects such as filters and global displays can be rendered on the entire live image frame. Therefore, different rendering methods can be used for different target effects.

[0060] Optionally, based on any of the above embodiments, step 203 includes:

[0061] A target area where at least part of the target object is located is determined in the live image frame according to the key point information corresponding to at least part of the target object.

[0062] If the target special effect is a special effect applied locally, then for at least a portion of the target area, a local rendering operation is performed on the target area according to the target special effect to obtain a rendering result of the target area.

[0063] For at least a portion of the target area, the rendering result of the target area is overlaid onto the live image frame to obtain the target image frame.

[0064] In this embodiment, the target special effect can specifically be a special effect applied locally. For example, it can be a beauty special effect, which is a special effect applied only to the face. Alternatively, it can be a headwear special effect, which is a special effect applied only to the head. Therefore, for such a localized special effect, the special effect rendering operation can be performed only on the target area, while the special effect rendering operation is not performed on other locations in the live image frame. This allows the special effect processing area to be concentrated at the location associated with the key point information, effectively reducing the scope of the special effect rendering and improving the speed of the special effect rendering.

[0065] Specifically, when the target special effect is a special effect applied locally, a local rendering operation can be performed on each target area according to the target special effect to obtain a target area rendering result. After the special effect rendering is completed, the target area rendering result can be overlaid on the live image frame to obtain a target image frame.

[0066] Furthermore, based on any of the above embodiments, the performing a local rendering operation on the target area according to the target special effect includes:

[0067] If it is detected that the target special effect meets the preset expansion condition, an expansion operation is performed on the target area according to a preset area expansion algorithm to obtain an area to be rendered.

[0068] Perform a local rendering operation on the area to be rendered according to the target special effect.

[0069] In this embodiment, to ensure the rendering effect at the edge of the target area, an expansion operation can be performed on the target area. Specifically, if it is detected that the target special effect meets the preset expansion conditions, the target area is expanded according to the preset regional expansion algorithm to obtain the area to be rendered. The preset expansion condition may be that the expansion operation can be performed when the target special effect is a special effect acting on the face or other preset locations. The expanded area to be rendered is then locally rendered according to the target special effect.

[0070] FIG3 is a schematic diagram of expansion provided by an embodiment of the present disclosure. As shown in FIG3 , in order to obtain a better rendering effect, after acquiring a target area 31 , an expansion operation may be performed on the target area 31 to obtain an area 32 to be rendered.

[0071] The live broadcast special effects rendering method provided in this embodiment can ensure the effect of special effects rendering and improve the live broadcast quality by performing an expansion operation on the target area when the target special effects meet the preset expansion conditions.

[0072] Optionally, based on any of the above embodiments, step 203 includes:

[0073] If the target special effect is a special effect applied globally, a special effect rendering operation is performed on the live image frame according to the target special effect to obtain the target image frame.

[0074] In this embodiment, the target special effect may also include a special effect applied globally, such as a filter, raindrops displayed globally, etc. Therefore, when the target special effect is a special effect applied globally, the live image frame may be subjected to a special effect rendering operation according to the target special effect to obtain the target image frame.

[0075] The live broadcast special effects rendering method provided in this embodiment performs special effects rendering operations on the positions associated with the key point information in the live broadcast image frame according to the key point information, thereby concentrating the area for special effects processing at the positions associated with the key point information, effectively reducing the area that needs to be processed by special effects, and thereby improving the efficiency of special effects processing.

[0076] It should be noted that since the server is equipped with a graphics processor and a central processing unit respectively, the central processing unit can be used to perform key point recognition operations according to the processing characteristics of different processors, and the graphics processor can be used to perform special effects rendering processing based on the key point information.

[0077] Further, based on any of the above embodiments, step 202 includes:

[0078] Determining key point information corresponding to at least part of the target object in the live image frame by a preset central processing unit;

[0079] Step 203 includes:

[0080] A special effects rendering operation is performed on the live image frame according to the target special effects and the key point information by a preset graphics processor to obtain a target image frame.

[0081] In this embodiment, the graphics processor is respectively connected to the central processing unit and the binocular image acquisition device, so as to obtain the live image frames captured by the binocular image acquisition device, perform detection operations through the central processing unit, and perform special effects rendering operations through the graphics processor.

[0082] Therefore, after acquiring a live video frame, the graphics processor can send it to the central processing unit. Accordingly, after acquiring the live video frame, the central processing unit determines key point information corresponding to at least a portion of the target object in the live video frame based on a preset detection algorithm. Specifically, this key point information can be the coordinate information of key locations within the target object. This key point information is then fed back to the graphics processor.

[0083] After obtaining the key point information, the graphics processor can render the live image frame using a rendering method corresponding to the target special effect based on the key point information.

[0084] The live special effects rendering method provided in this embodiment, after acquiring a live image frame corresponding to the virtual reality live content, uses a central processing unit to calculate key point information in the live image frame. Based on this key point information, a graphics processing unit performs special effects rendering operations on the locations in the live image frame associated with the key point information. This allows the special effects processing area to be concentrated at the locations associated with the key point information, effectively reducing the area requiring special effects processing and thereby improving the efficiency of special effects processing. Furthermore, by using a graphics processing unit for special effects processing, the large-scale transmission of live image frames can be effectively avoided, reducing data transmission time and further improving the efficiency of special effects processing.

[0085] Further, based on any of the above embodiments, step 202 includes:

[0086] A size adjustment operation is performed on the live image frame to obtain an adjusted live image frame.

[0087] The central processing unit determines key point information corresponding to at least part of the target object in the adjusted live image frame.

[0088] In this embodiment, since the live image frame is generally an 8K (7680*4320) or larger image frame, the size is relatively large. Therefore, the key point recognition operation based on the live image frame takes a long time. In order to ensure the live broadcast effect, before the key point information of the live image frame is recognized, the live image frame can be resized to obtain an adjusted live image frame. The resizing operation can be a resizing operation on the live image frame, scaling the live image frame to a 1K image frame, and then the key point recognition efficiency based on the adjusted live image frame is higher.

[0089] Furthermore, since the server is equipped with a graphics processor and a central processing unit, the central processing unit can be used to identify key point information. In addition, the graphics processor can be used to obtain live image frames, or the central processing unit can be used to obtain live image frames. Alternatively, the user can set it according to actual needs. In this embodiment, there is no restriction on the execution entity of live image frame acquisition.

[0090] Therefore, after the size adjustment of the live image frame is completed and the adjusted live image frame is obtained, the central processing unit can determine the key point information corresponding to at least part of the target object in the adjusted live image frame.

[0091] The live special effects rendering method provided in this embodiment resizes the live image frames corresponding to the VR live content after acquiring them, and uses a central processing unit to calculate key point information within the live image frames. This effectively reduces the computational effort required for key point identification and improves the efficiency of live image frame rendering. This ensures smooth VR live streaming without lags, enhancing the user experience.

[0092] FIG4 is a flow chart of a live broadcast special effects rendering method provided by another embodiment of the present disclosure. Based on any of the above embodiments, as shown in FIG4 , step 202 includes:

[0093] Step 401: Perform a first scaling operation on the live image frame by the graphics processor to obtain a live image frame with a first preset resolution, and send the live image frame with the first preset resolution to the central processor.

[0094] Step 402: The central processor detects a predicted area corresponding to at least part of the target object in the live image frame of the first preset resolution according to a preset first detection algorithm, and sends the predicted area corresponding to at least part of the target object to the graphics processor.

[0095] Step 403: The graphics processor performs a cropping operation on at least a portion of the target object in the live image frame according to the predicted area to obtain an original pixel map corresponding to at least a portion of the predicted area, and sends the original pixel map corresponding to at least a portion of the predicted area to the central processor.

[0096] Step 404: Determine, by the central processing unit according to a preset second detection algorithm, key points corresponding to the target object in at least part of the predicted area.

[0097] In this embodiment, since the pixel value of the live image frame is generally (7680*4320) or above, the data volume of a single frame is 7680*4320*4Byte=126MB, so the transmission time of the live image frame is also relatively long.

[0098] Optionally, the graphics processor can perform the acquisition of live image frames and special effects rendering operations, and the central server can perform key point information recognition operations. Therefore, after the graphics processor acquires the live image frame, it is necessary to send the live image frame to the central processing unit for key point detection. In order to increase the speed of special effects rendering, the amount of transmitted data can be reduced during the data transmission process. Specifically, the graphics processor can perform a first scaling operation on the live image frame to obtain a live image frame of a first preset resolution, and send the live image frame of the first preset resolution to the central processing unit. In actual applications, the corresponding first preset resolution can be set according to actual needs, and the present disclosure does not limit this. For example, an 8K live image frame can be scaled to a 1k live image frame.

[0099] After the central processing unit obtains the live video frame of the first preset resolution, because the content clarity of the live video frame of the first preset resolution is lower than that of the original live video frame, the central processing unit may perform a coarse-grained prediction of the region where the target object is located in the live video frame of the first preset resolution, obtain a predicted region corresponding to at least a portion of the target object in the live video frame of the first preset resolution, and send the predicted region corresponding to at least a portion of the target object in the live video frame of the first preset resolution to the graphics processor.

[0100] After obtaining the predicted area corresponding to at least part of the target object, the graphics processor can directly perform special effects rendering operations on the predicted area to obtain the target image frame. Optionally, in order to further improve the accuracy of special effects rendering, the graphics processor can also perform a cropping operation on the target object based on the predicted area corresponding to at least part of the target object to obtain the original pixel map corresponding to at least part of the predicted area. The pixels of the original pixel map are the same as those of the live image frame. Since the size of the original pixel map is much smaller than the live image frame, the transmission speed is faster when it is transmitted to the central processing unit.

[0101] Accordingly, after acquiring at least a portion of the original pixel image, the CPU can perform an identification operation on key point information of the target object in the original pixel image, obtain at least a portion of the key point information of the target object, and feed it back to the GPU. If the target object is a person, the key point information can include coordinate information of key locations such as the person's head and facial features.

[0102] FIG5 is a schematic diagram of an application scenario provided by an embodiment of the present disclosure. As shown in FIG5 , a graphics processor 51 can perform a first scaling operation on a live image frame 52 to obtain a live image frame 53 of a first preset resolution, and transmit the live image frame 53 of the first preset resolution to a central processing unit 54. The central processing unit 54 can detect a predicted region on the live image frame 53 of the first preset resolution to obtain a predicted region 55 corresponding to at least a portion of a target object in the live image frame. The predicted region 55 corresponding to at least a portion of the target object in the live image frame is transmitted to the graphics processor 51, so that the graphics processor 51 can perform a cropping operation on the predicted region 55 corresponding to at least a portion of the target object in the live image frame and transmit a raw pixel image 56 corresponding to the cropped at least portion of the predicted region to the central processing unit 54. The central processing unit 54 can detect key points in the raw pixel image 56 corresponding to at least a portion of the predicted region and feed the key points back to the graphics processor 51, so that the graphics processor 51 can perform special effects rendering on the live image frame based on the key point information to obtain the target image frame.

[0103] Further, based on any of the above embodiments, step 404 includes:

[0104] A second scaling operation is performed on the original pixel image corresponding to at least a portion of the predicted area by the graphics processor to obtain an original pixel image of a second preset resolution corresponding to at least a portion of the predicted area.

[0105] The original pixel image of the second preset resolution corresponding to at least part of the predicted area is sent to the central processing unit.

[0106] In this embodiment, to further improve the speed of image transmission, a second scaling operation may be performed on the original pixel image before transmission to obtain an original pixel image of a second preset resolution corresponding to at least a portion of the predicted area. The scaling scale of the second scaling operation is smaller than the scaling scale of the first scaling operation, that is, the second preset resolution is larger than the first preset resolution. The original pixel image of the second preset resolution corresponding to at least a portion of the predicted area is sent to the central processing unit.

[0107] The live broadcast special effects rendering method provided in this embodiment can effectively reduce the amount of data transmitted, improve the speed of data transmission between the graphics processor and the central processing unit, and further improve the speed of special effects rendering in VR live broadcast scenarios by sending the scaled live broadcast image frame of the first preset resolution to the central processing unit during data transmission, and sending the original pixel image corresponding to at least a portion of the cropped prediction area to the central processing unit.

[0108] Further, based on any of the above embodiments, step 402 includes:

[0109] Performing a detection operation on the target object in the live image frame of the first preset resolution using a preset first detection algorithm to determine a first area where at least part of the target object is located;

[0110] For at least two first regions that meet a preset merging condition, determining whether a size of a merged region after the at least two first regions are merged is larger than a size of the at least two first regions before the merging;

[0111] If yes, determining the first area as the prediction area;

[0112] If not, the merged area is determined as the prediction area.

[0113] In this embodiment, during the generation of the predicted region, to reduce the computational complexity of subsequent special effects rendering, regions that meet preset merging conditions may be merged. Specifically, a preset first detection algorithm may be used to detect the target object in the live image frame of the first preset resolution to determine the first region where at least part of the target object is located.

[0114] Determine whether at least part of the first areas meet a preset merging condition, wherein the preset merging condition includes but is not limited to: the distance between at least two first areas is less than a preset distance threshold; at least two first areas have an intersection; any first area has a larger coverage area while the surrounding first areas have a smaller coverage area; etc.

[0115] For at least two first regions that meet a preset merging condition, determine whether the size of the merged region after the at least two first regions are merged is larger than the size of the at least two first regions before the merging. If so, determine the first region as a predicted region. If not, determine the merged region as a predicted region.

[0116] The live special effects rendering method provided in this embodiment uses a graphics processor to perform special effects rendering operations and a central processing unit for identification and detection operations. To further improve the special effects rendering speed, the special effects rendering range can be concentrated on or around the host, thereby reducing the actual pixel area that needs to be processed, thereby improving the efficiency of special effects processing.

[0117] FIG6 is a flow chart of a live special effects rendering method provided by another embodiment of the present disclosure. Based on any of the above embodiments, the key point information includes coordinate information of multiple key points corresponding to the target object. As shown in FIG6 , step 203 includes:

[0118] Step 601: Determine, by the graphics processor, a target area where at least a portion of the target object is located in the live image frame according to key point information corresponding to at least a portion of the target object.

[0119] Step 602: For the target area, a special effect rendering operation is performed on the target area or the live image frame using a rendering method that matches the target special effect to obtain the target image frame.

[0120] In this embodiment, in order to increase the speed of special effect rendering of live image frames and ensure the live broadcast effect, the area for special effect processing can be concentrated at the position associated with key point information.

[0121] Therefore, after obtaining key point information corresponding to at least a portion of the target object, the graphics processor can determine the target region in the live image frame where at least a portion of the target object is located based on the key point information corresponding to at least a portion of the target object. For each target region, a special effects rendering operation can be performed on the target region or the live image frame according to a rendering method corresponding to a target special effect preset by the user to obtain the target image frame.

[0122] The live special effects rendering method provided in this embodiment can improve the efficiency of special effects rendering by concentrating the rendering area around the target area.

[0123] Furthermore, based on any of the above embodiments, before step 201, the method further includes:

[0124] The graphics processor obtains the original image frames corresponding to the virtual reality live broadcast content captured by the binocular image capture device, and performs hardware decoding operations and format conversion operations on the original image frames to obtain the live broadcast image frames.

[0125] In this embodiment, in order to further improve the speed of special effect rendering and avoid excessive information interaction between the graphics processor and the central processing unit, the pre-processing of the original image frame can be performed by the graphics processor.

[0126] Accordingly, the live special effects rendering device can obtain the original image frame corresponding to the virtual reality live broadcast content captured by the binocular image acquisition device, perform hardware decoding and format conversion operations on the original image frame, and obtain the live broadcast image frame.

[0127] The live special effects rendering method provided in this embodiment can effectively avoid the delay caused by excessive transmission of live image frames by placing the pre-processing of the original image frames on the graphics processor, thereby improving the rendering speed of the live image frames.

[0128] FIG7 is a flowchart of a live broadcast special effects rendering method provided by another embodiment of the present disclosure. Based on any of the above embodiments, as shown in FIG7 , before step 201, the method further includes:

[0129] Step 701: In response to a test instruction triggered by a user, a test image frame corresponding to the virtual reality live broadcast content is obtained.

[0130] Step 702: According to the test type corresponding to the test instruction, a test operation is performed on the test image frame using a test method corresponding to the test type.

[0131] Step 201 includes:

[0132] Step 703: When the test image frame meets the preset live broadcast condition, the live broadcast image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained.

[0133] In this embodiment, in order to ensure the live broadcast effect of the virtual reality live broadcast, before performing special effects rendering on the live broadcast image frames corresponding to the virtual reality live broadcast, it is first necessary to test the current live broadcast effect.

[0134] Optionally, a preset test control can be displayed on the live broadcast display interface, and the user can trigger the test control according to actual needs. In response to the test instruction triggered by the user triggering the test control, a test image frame corresponding to the virtual reality live broadcast content can be obtained. The test image frame can be an image frame captured by a preset binocular image acquisition device, and the test image frame can be composed of an image frame captured by the left image acquisition device of the binocular image acquisition device and an image frame captured by the right image acquisition device of the binocular image acquisition device.

[0135] Furthermore, to ensure the live broadcast quality, different test types can be performed on the test image frames. For example, the test types may include a first test type and a second test type. The first test type may be a brightness test, which ensures the viewing quality of the live broadcast image frames by performing a brightness test on the test image frames. The second test type may be a focus test to ensure the clarity of the live broadcast image frames.

[0136] Different test methods can be pre-set for different test types. After determining the current test type, the test method corresponding to the test type can be used to perform a test operation on the test image frame.

[0137] After completing the test operation on the test image frame, the test result can be obtained. After obtaining the test result, if it is detected that the test result meets the preset live broadcast conditions, the live broadcast image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained.

[0138] FIG8 is a schematic diagram of an interface interaction provided by an embodiment of the present disclosure. As shown in FIG8 , a preset test control 82 may be displayed on a live display interface 81. In response to a user triggering operation on the test control 82, a test image frame 83 and a test result 84 may be displayed in a preset display area of ​​the live display interface 81.

[0139] The live special effects rendering method provided in this embodiment can effectively ensure the display effect of virtual reality live broadcast by performing a test operation on the test image frame before performing special effects rendering on the live image frame, thereby improving the user experience.

[0140] Optionally, based on any of the above embodiments, the test type includes a first test type, and step 702 includes:

[0141] Performing brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0142] In this embodiment, the test type includes a first test type, which may be a brightness test. By performing a brightness test on the test image frame, the viewing effect of the live image frame can be guaranteed.

[0143] Optionally, a preset brightness test control may be displayed on the live display interface, and the user may trigger a test operation of the first test type by triggering the brightness test control. When it is determined that the type of the current test operation is the first test type, a brightness detection algorithm corresponding to the first test type may be used to perform brightness detection on the test image frame to obtain a brightness detection result.

[0144] Among them, any algorithm that can realize brightness detection can be used to realize brightness detection of the test image frame, and the present disclosure does not impose any limitation on this.

[0145] FIG9 is another interface interaction diagram provided by an embodiment of the present disclosure. As shown in FIG9 , a preset brightness test control 92 can be displayed on a live display interface 91. In response to a user triggering operation of the brightness test control 92, a test image frame 93 and a brightness test result 94 can be displayed in a preset display area of ​​the live display interface 91.

[0146] Further, based on any of the above embodiments, after performing brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type and obtaining a brightness detection result, the method further includes:

[0147] If it is detected that the brightness detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are obtained;

[0148] If it is detected that the brightness detection result does not meet the preset live broadcast condition, a preset first prompt message is displayed, and the first prompt message is used to prompt the user to adjust the brightness of the current position to a preset brightness threshold.

[0149] In this embodiment, after obtaining the brightness detection result, if it is detected that the brightness detection result meets the preset live broadcast conditions, the live image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained to perform subsequent special effects rendering operations on the live image frame. Optionally, if it is detected that the brightness detection result meets the preset live broadcast conditions, a prompt message indicating that the test is successful can also be displayed to prompt the user to perform subsequent live broadcast operations. Conversely, if it is detected that the brightness detection result does not meet the preset live broadcast conditions, a preset first prompt message is displayed, and the first prompt message is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold. For example, the first prompt message can specifically be: The current brightness is insufficient, please adjust the lights in the room to 600 lumens.

[0150] The live broadcast special effects rendering method provided in this embodiment detects the brightness of the test image frame before performing special effects rendering on the live broadcast image frame, and performs subsequent live broadcast image frame rendering operations when the brightness meets the preset live broadcast conditions, thereby ensuring the live broadcast effect of virtual reality live broadcast.

[0151] Optionally, based on any of the above embodiments, the test type includes a second test type, and step 702 includes:

[0152] The test image frame is tested using a focus test algorithm corresponding to the second test type to obtain a focus test result.

[0153] In this embodiment, the test type includes a second test type, which can be a focus test to ensure the clarity of the live image frame. When the current test type is determined to be the second test type, the test image frame can be tested using a focus test algorithm corresponding to the second test type to obtain a focus test result. The test operation on the test image frame can be implemented using any algorithm capable of implementing focus detection, and this disclosure does not impose any restrictions on this.

[0154] FIG10 is another schematic diagram of an interface interaction provided by an embodiment of the present disclosure. As shown in FIG10 , a preset focus test control 1002 may be displayed on a live display interface 1001. In response to a user triggering operation on the focus test control 1002, a test image frame 1003 and a focus test result 1004 may be displayed in a preset display area of ​​the live display interface 1001.

[0155] Further, based on any of the above embodiments, after detecting the test image frame using the focus test algorithm corresponding to the second test type and obtaining the focus detection result, the method further includes:

[0156] If it is detected that the focus detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are obtained;

[0157] If it is detected that the focus detection result does not meet the preset live broadcast condition, a preset second prompt information is displayed, and the second prompt information is used to prompt the user to perform a focus operation again.

[0158] In this embodiment, if the focus detection result is detected to meet the preset live broadcast conditions, the live image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained to perform subsequent special effects rendering operations on the live image frame. Optionally, if the focus detection result is detected to meet the preset live broadcast conditions, a prompt message indicating a successful test may also be displayed to prompt the user to perform subsequent live broadcast operations. Conversely, if the focus detection result is detected to not meet the preset live broadcast conditions, a preset second prompt message is displayed, which is used to prompt the user to refocus the operation.

[0159] Optionally, an adjustment method may be determined based on the focus detection result, and the adjustment method may be displayed in the second prompt information so that the user can adjust the focus according to the adjustment method. For example, the second prompt information may be: focus failed, left camera adjusted backward.

[0160] The live special effects rendering method provided in this embodiment detects whether a test image frame is in focus before rendering special effects on the live image frame. If the focus detection result meets the preset live broadcast conditions, the live image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained to perform subsequent special effects rendering operations on the live image frame. This ensures the clarity of the live image frame and further improves the user experience.

[0161] Figure 11 is a structural diagram of a live broadcast special effects rendering device provided by an embodiment of the present disclosure, which is applied to a graphics processor. As shown in Figure 11, the device includes: an acquisition module 1101, a determination module 1102, a rendering module 1103, and a display module 1104. Among them, the acquisition module 1101 is used to obtain a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect. The determination module 1102 is used to determine the key point information corresponding to at least part of the target object in the live broadcast image frame. The rendering module 1103 is used to perform a special effects rendering operation on the live broadcast image frame according to the target special effect and the key point information to obtain a target image frame. The display module 1104 is used to display the target image frame.

[0162] Further, based on any of the above embodiments, the rendering module is configured to: determine a target area within the live image frame where at least a portion of the target object is located based on key point information corresponding to at least a portion of the target object. If the target special effect is a localized special effect, a local rendering operation is performed on at least a portion of the target area based on the target special effect to obtain a rendering result for the target area. For at least a portion of the target area, the rendering result of the target area is overlaid onto the live image frame to obtain the target image frame.

[0163] Furthermore, based on any of the above embodiments, the rendering module is configured to: if it is detected that the target special effect meets a preset expansion condition, perform an expansion operation on the target area according to a preset area expansion algorithm to obtain a to-be-rendered area, and perform a partial rendering operation on the to-be-rendered area according to the target special effect.

[0164] Further, based on any of the above embodiments, the rendering module is used to: if the target special effect is a special effect applied globally, perform a special effect rendering operation on the live image frame according to the target special effect to obtain the target image frame.

[0165] Furthermore, based on any of the above embodiments, the determination module is configured to determine key point information corresponding to at least a portion of the target object in the live image frame using a preset central processing unit. The rendering module is configured to perform a special effects rendering operation on the live image frame based on the target special effect and the key point information using a preset graphics processor to obtain a target image frame.

[0166] Furthermore, based on any of the above embodiments, the determining module is configured to: resize the live video frame to obtain an adjusted live video frame, and determine key point information corresponding to at least part of the target object in the adjusted live video frame by the central processing unit.

[0167] Further, based on any of the above embodiments, the determination module is used to: perform a first scaling operation on the live image frame by the graphics processor to obtain a live image frame of a first preset resolution, and send the live image frame of the first preset resolution to the central processing unit. Detect the predicted area corresponding to at least part of the target object in the live image frame of the first preset resolution by the central processing unit according to a preset first detection algorithm, and send the predicted area corresponding to at least part of the target object to the graphics processor. Perform a cropping operation on at least part of the target object in the live image frame according to the predicted area by the graphics processor to obtain an original pixel map corresponding to at least part of the predicted area, and send the original pixel map corresponding to at least part of the predicted area to the central processing unit. Determine the key points corresponding to the target object in at least part of the predicted area by the central processing unit according to a preset second detection algorithm.

[0168] Furthermore, based on any of the above embodiments, the determining module is configured to: perform a second scaling operation on the original pixel image corresponding to the at least portion of the prediction area via the graphics processor to obtain an original pixel image having a second preset resolution corresponding to the at least portion of the prediction area, and send the original pixel image having the second preset resolution corresponding to the at least portion of the prediction area to the central processing unit.

[0169] Furthermore, based on any of the above embodiments, the key point information includes coordinate information of multiple key points corresponding to the target object. The rendering module is configured to: determine, via the graphics processor, a target area in the live image frame where at least a portion of the target object is located based on the key point information corresponding to at least a portion of the target object. With respect to the target area, perform a special effects rendering operation on the target area or the live image frame using a rendering method that matches the target special effect to obtain the target image frame.

[0170] Furthermore, based on any of the above embodiments, the device also includes: a preprocessing module, which is used to: obtain the original image frames corresponding to the virtual reality live broadcast content captured by the binocular image acquisition device through the graphics processor, and perform hardware decoding operations and format conversion operations on the original image frames to obtain the live broadcast image frames.

[0171] Furthermore, based on any of the above embodiments, the rendering module is configured to: detect the target object in the live image frame of the first preset resolution using a preset first detection algorithm to determine a first region where at least part of the target object is located. For at least two first regions that meet a preset merging condition, determine whether the size of the merged region after merging the at least two first regions is greater than the size of the at least two first regions before merging. If so, determine the first region as the predicted region. If not, determine the merged region as the predicted region.

[0172] Furthermore, based on any of the above embodiments, the apparatus further includes: an acquisition module, further configured to acquire a test image frame corresponding to the virtual reality live broadcast content in response to a test instruction triggered by a user;

[0173] The testing module is used to perform a test operation on the test image frame according to the test type corresponding to the test instruction using a test method corresponding to the test type; the acquisition module is also used to: when the test image frame meets the preset live broadcast conditions, obtain the live image frame corresponding to the virtual reality live broadcast content and the preset target special effects.

[0174] Further, based on any of the above embodiments, the test type includes a first test type, and the test module is used to: perform brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0175] Furthermore, based on any of the above embodiments, the device also includes: a processing module, which is used to obtain a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the brightness detection result meets the preset live broadcast condition; the processing module is also used to display a preset first prompt information if it is detected that the brightness detection result does not meet the preset live broadcast condition, and the first prompt information is used to prompt the user to adjust the brightness of the current position to a preset brightness threshold.

[0176] Further, based on any of the above embodiments, the test type includes a second test type, and the test module is used to: detect the test image frame through a focus test algorithm corresponding to the second test type to obtain a focus detection result.

[0177] Furthermore, based on any of the above embodiments, the device also includes: a processing module, which is used to obtain a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the focus detection result meets the preset live broadcast condition; a processing module, which is used to display a preset second prompt information if it is detected that the focus detection result does not meet the preset live broadcast condition, and the second prompt information is used to prompt the user to re-focus the operation.

[0178] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0179] In order to implement the above embodiment, an embodiment of the present disclosure further provides an electronic device, including: a processor and a memory.

[0180] The memory stores computer-executable instructions.

[0181] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the live special effects rendering method as described in any of the above embodiments.

[0182] FIG12 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. As shown in FIG12 , the electronic device 1200 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG12 is merely an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0183] As shown in FIG12 , the electronic device 1200 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage device 1208 into a random access memory (RAM) 1203. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0184] Typically, the following devices may be connected to the I / O interface 1205: an input device 1206 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1208 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1209. The communication device 1209 may allow the electronic device 1200 to communicate with other devices wirelessly or by wire to exchange data. Although FIG12 shows an electronic device 1200 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0185] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 1209, or installed from the storage device 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0186] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0187] An embodiment of the present disclosure further provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the live broadcast special effects rendering method as described in any of the above embodiments is implemented.

[0188] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the method for rendering live special effects as described in any of the above embodiments.

[0189] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0190] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0191] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0194] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0195] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0196] In a first aspect, according to one or more embodiments of the present disclosure, a method for rendering special effects for a live broadcast is provided, comprising:

[0197] Obtain the live image frames corresponding to the virtual reality live content and the preset target special effects;

[0198] Determining key point information corresponding to at least part of the target object in the live image frame;

[0199] Performing a special effects rendering operation on the live image frame according to the target special effects and the key point information to obtain a target image frame;

[0200] The target image frame is displayed.

[0201] According to one or more embodiments of the present disclosure, performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain the target image frame includes:

[0202] Determining a target area where at least part of the target object is located in the live image frame according to key point information corresponding to at least part of the target object;

[0203] If the target special effect is a special effect applied locally, then for at least part of the target area, a local rendering operation is performed on the target area according to the target special effect to obtain a rendering result of the target area;

[0204] For at least a portion of the target area, the rendering result of the target area is overlaid onto the live image frame to obtain the target image frame.

[0205] According to one or more embodiments of the present disclosure, performing a local rendering operation on the target area according to the target special effect includes:

[0206] If it is detected that the target special effect meets the preset expansion condition, the target area is expanded according to the preset area expansion algorithm to obtain the area to be rendered;

[0207] Perform a local rendering operation on the area to be rendered according to the target special effect.

[0208] According to one or more embodiments of the present disclosure, performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain the target image frame includes:

[0209] If the target special effect is a special effect applied globally, a special effect rendering operation is performed on the live image frame according to the target special effect to obtain the target image frame.

[0210] According to one or more embodiments of the present disclosure, determining key point information corresponding to at least part of the target object in the live image frame includes:

[0211] Determining key point information corresponding to at least part of the target object in the live image frame by a preset central processing unit;

[0212] The performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain the target image frame includes:

[0213] A special effects rendering operation is performed on the live image frame according to the target special effects and the key point information by a preset graphics processor to obtain a target image frame.

[0214] According to one or more embodiments of the present disclosure, determining key point information corresponding to at least part of the target object in the live image frame includes:

[0215] Performing a size adjustment operation on the live image frame to obtain an adjusted live image frame;

[0216] The central processing unit determines key point information corresponding to at least part of the target object in the adjusted live image frame.

[0217] According to one or more embodiments of the present disclosure, determining key point information corresponding to at least part of the target object in the live image frame includes:

[0218] performing a first scaling operation on the live image frame by the graphics processor to obtain a live image frame with a first preset resolution, and sending the live image frame with the first preset resolution to the central processor;

[0219] Detecting, by the central processor, a predicted area corresponding to at least a portion of the target object in the live image frame of the first preset resolution according to a preset first detection algorithm, and sending the predicted area corresponding to at least a portion of the target object to the graphics processor;

[0220] performing a cropping operation on at least a portion of the target object in the live image frame according to the predicted area by the graphics processor to obtain an original pixel map corresponding to at least a portion of the predicted area, and sending the original pixel map corresponding to at least a portion of the predicted area to the central processor;

[0221] The central processing unit determines key points corresponding to the target object in at least part of the predicted area according to a preset second detection algorithm.

[0222] According to one or more embodiments of the present disclosure, the sending of the original pixel map corresponding to at least part of the predicted area to the central processor includes:

[0223] performing a second scaling operation on the original pixel image corresponding to at least a portion of the prediction area by the graphics processor to obtain an original pixel image of a second preset resolution corresponding to at least a portion of the prediction area;

[0224] The original pixel image of the second preset resolution corresponding to at least part of the predicted area is sent to the central processing unit.

[0225] According to one or more embodiments of the present disclosure, the key point information includes coordinate information of multiple key points corresponding to the target object; performing a special effects rendering operation on the live image frame according to the target special effect and the key point information to obtain the target image frame includes:

[0226] Determining, by the graphics processor, a target area where at least part of the target object is located in the live image frame according to key point information corresponding to at least part of the target object;

[0227] For the target area, a special effects rendering operation is performed on the target area or the live image frame using a rendering method that matches the target special effect to obtain the target image frame.

[0228] According to one or more embodiments of the present disclosure, before obtaining the live image frames corresponding to the virtual reality live content and the preset target special effects, the process further includes:

[0229] The graphics processor obtains the original image frames corresponding to the virtual reality live broadcast content captured by the binocular image capture device, and performs hardware decoding operations and format conversion operations on the original image frames to obtain the live broadcast image frames.

[0230] According to one or more embodiments of the present disclosure, detecting, by the central processor according to a preset first detection algorithm, a predicted area corresponding to at least part of the target object in the live image frame of the first preset resolution includes:

[0231] Performing a detection operation on the target object in the live image frame of the first preset resolution using a preset first detection algorithm to determine a first area where at least part of the target object is located;

[0232] For at least two first regions that meet a preset merging condition, determining whether a size of a merged region after the at least two first regions are merged is larger than a size of the at least two first regions before the merging;

[0233] If yes, determining the first area as the prediction area;

[0234] If not, the merged area is determined as the prediction area.

[0235] According to one or more embodiments of the present disclosure, before obtaining the live image frames corresponding to the virtual reality live content and the preset target special effects, the process further includes:

[0236] In response to a test instruction triggered by a user, obtaining a test image frame corresponding to the virtual reality live broadcast content;

[0237] According to the test type corresponding to the test instruction, a test operation is performed on the test image frame using a test method corresponding to the test type;

[0238] The obtaining of live image frames corresponding to the virtual reality live content and preset target special effects includes:

[0239] When the test image frame meets the preset live broadcast condition, the live broadcast image frame corresponding to the virtual reality live broadcast content and the preset target special effects are obtained.

[0240] According to one or more embodiments of the present disclosure, the test type includes a first test type, and performing a test operation on the test image frame according to the test type corresponding to the test instruction and using a test method corresponding to the test type includes:

[0241] Performing brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0242] According to one or more embodiments of the present disclosure, after performing brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type and obtaining a brightness detection result, the method further includes:

[0243] If it is detected that the brightness detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are obtained;

[0244] If it is detected that the brightness detection result does not meet the preset live broadcast condition, a preset first prompt message is displayed, and the first prompt message is used to prompt the user to adjust the brightness of the current position to a preset brightness threshold.

[0245] According to one or more embodiments of the present disclosure, the test type includes a second test type, and performing a test operation on the test image frame according to the test type corresponding to the test instruction and using a test method corresponding to the test type includes:

[0246] The test image frame is tested using a focus test algorithm corresponding to the second test type to obtain a focus test result.

[0247] According to one or more embodiments of the present disclosure, after detecting the test image frame using the focus test algorithm corresponding to the second test type and obtaining a focus detection result, the method further includes:

[0248] If it is detected that the focus detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are obtained;

[0249] If it is detected that the focus detection result does not meet the preset live broadcast condition, a preset second prompt information is displayed, and the second prompt information is used to prompt the user to perform a focus operation again.

[0250] In a second aspect, according to one or more embodiments of the present disclosure, a live broadcast special effects rendering device is provided, comprising:

[0251] An acquisition module is used to obtain live image frames corresponding to virtual reality live content and preset target special effects;

[0252] A determination module, configured to determine key point information corresponding to at least part of the target object in the live image frame;

[0253] A rendering module, configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame;

[0254] A display module is configured to display the target image frame.

[0255] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0256] Determining a target area where at least part of the target object is located in the live image frame according to key point information corresponding to at least part of the target object;

[0257] If the target special effect is a special effect applied locally, then for at least part of the target area, a local rendering operation is performed on the target area according to the target special effect to obtain a rendering result of the target area;

[0258] For at least a portion of the target area, the rendering result of the target area is overlaid onto the live image frame to obtain the target image frame.

[0259] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0260] If it is detected that the target special effect meets the preset expansion condition, the target area is expanded according to the preset area expansion algorithm to obtain the area to be rendered;

[0261] Perform a local rendering operation on the area to be rendered according to the target special effect.

[0262] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0263] If the target special effect is a special effect applied globally, a special effect rendering operation is performed on the live image frame according to the target special effect to obtain the target image frame.

[0264] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0265] Determining key point information corresponding to at least part of the target object in the live image frame by a preset central processing unit;

[0266] The rendering module is used to:

[0267] A special effects rendering operation is performed on the live image frame according to the target special effects and the key point information by a preset graphics processor to obtain a target image frame.

[0268] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0269] Performing a size adjustment operation on the live image frame to obtain an adjusted live image frame;

[0270] The central processing unit determines key point information corresponding to at least part of the target object in the adjusted live image frame.

[0271] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0272] performing a first scaling operation on the live image frame by the graphics processor to obtain a live image frame with a first preset resolution, and sending the live image frame with the first preset resolution to the central processor;

[0273] Detecting, by the central processor, a predicted area corresponding to at least a portion of the target object in the live image frame of the first preset resolution according to a preset first detection algorithm, and sending the predicted area corresponding to at least a portion of the target object to the graphics processor;

[0274] performing a cropping operation on at least a portion of the target object in the live image frame according to the predicted area by the graphics processor to obtain an original pixel map corresponding to at least a portion of the predicted area, and sending the original pixel map corresponding to at least a portion of the predicted area to the central processor;

[0275] The central processing unit determines key points corresponding to the target object in at least part of the predicted area according to a preset second detection algorithm.

[0276] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0277] performing a second scaling operation on the original pixel image corresponding to at least a portion of the prediction area by the graphics processor to obtain an original pixel image of a second preset resolution corresponding to at least a portion of the prediction area;

[0278] The original pixel image of the second preset resolution corresponding to at least part of the predicted area is sent to the central processing unit.

[0279] According to one or more embodiments of the present disclosure, the key point information includes coordinate information of multiple key points corresponding to the target object; and the rendering module is configured to:

[0280] Determining, by the graphics processor, a target area where at least part of the target object is located in the live image frame according to key point information corresponding to at least part of the target object;

[0281] For the target area, a special effects rendering operation is performed on the target area or the live image frame using a rendering method that matches the target special effect to obtain the target image frame.

[0282] According to one or more embodiments of the present disclosure, the apparatus further includes a pre-processing module configured to:

[0283] The graphics processor obtains the original image frames corresponding to the virtual reality live broadcast content captured by the binocular image capture device, and performs hardware decoding operations and format conversion operations on the original image frames to obtain the live broadcast image frames.

[0284] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0285] Performing a detection operation on the target object in the live image frame of the first preset resolution using a preset first detection algorithm to determine a first area where at least part of the target object is located;

[0286] For at least two first regions that meet a preset merging condition, determining whether a size of a merged region after the at least two first regions are merged is larger than a size of the at least two first regions before the merging;

[0287] If yes, determining the first area as the prediction area;

[0288] If not, the merged area is determined as the prediction area.

[0289] According to one or more embodiments of the present disclosure, the device further includes: an acquisition module, which is further used to obtain a test image frame corresponding to the virtual reality live broadcast content in response to a test instruction triggered by a user; a testing module, which is used to perform a test operation on the test image frame according to the test type corresponding to the test instruction using a test method corresponding to the test type; the acquisition module is further used to: when the test image frame meets a preset live broadcast condition, obtain a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect.

[0290] According to one or more embodiments of the present disclosure, the test type includes a first test type, and the test module is configured to perform brightness detection on the test image frame using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0291] According to one or more embodiments of the present disclosure, the device further includes: a processing module, configured to obtain a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the brightness detection result meets the preset live broadcast condition; and a processing module, configured to display a preset first prompt message if it is detected that the brightness detection result does not meet the preset live broadcast condition, wherein the first prompt message is used to prompt the user to adjust the brightness of the current position to a preset brightness threshold.

[0292] According to one or more embodiments of the present disclosure, the test type includes a second test type, and the test module is used to: detect the test image frame through a focus test algorithm corresponding to the second test type to obtain a focus detection result.

[0293] According to one or more embodiments of the present disclosure, the device further includes: a processing module for obtaining a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the focus detection result meets the preset live broadcast condition; a processing module for displaying a preset second prompt message if it is detected that the focus detection result does not meet the preset live broadcast condition, and the second prompt message is used to prompt the user to re-focus the operation.

[0294] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0295] The memory stores computer-executable instructions;

[0296] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0297] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the live broadcast special effects rendering method described in the first aspect and various possible designs of the first aspect is implemented.

[0298] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the live broadcast special effects rendering method described in the first aspect and various possible designs of the first aspect.

[0299] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0300] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0301] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A live special effect rendering method, characterized in that, Including: Obtaining a live image frame corresponding to virtual reality live content and a preset target special effect; Determining key point information corresponding to at least some target objects in the live image frame; Performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame; Displaying the target image frame.

2. The method according to claim 1, characterized in that, The performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes: Determining at least some target regions where at least some target objects are located in the live image frame according to the key point information corresponding to the at least some target objects; If the target special effect is a special effect applied locally, then for at least some target regions, performing a local rendering operation on the target regions according to the target special effect to obtain a rendering result of the target regions; For at least some target regions, covering the rendering result of the target regions to the live image frame to obtain the target image frame.

3. The method according to claim 2, wherein The performing a local rendering operation on the target regions according to the target special effect includes: If it is detected that the target special effect meets a preset expansion condition, then performing an expansion operation on the target regions according to a preset region expansion algorithm to obtain a region to be rendered; Performing a local rendering operation on the region to be rendered according to the target special effect.

4. The method according to claim 1, characterized in that The performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes: If the target special effect is a special effect applied globally, then performing a special effect rendering operation on the live image frame according to the target special effect to obtain the target image frame.

5. The method according to claim 1, wherein The determining key point information corresponding to at least some target objects in the live image frame includes: Determining, by a preset central processing unit, key point information corresponding to at least some target objects in the live image frame Key point information; The performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes: Performing a special effect rendering operation on the live image frame according to the target special effect and the key point information by a preset graphics processing unit to obtain a target image frame.

6. The method according to claim 1 or 2, characterized in that, The determining key point information corresponding to at least some target objects in the live image frame includes: Performing a size adjustment operation on the live image frame to obtain an adjusted live image frame; Determining, by the central processing unit, key point information corresponding to at least some target objects in the adjusted live image frame.

7. The method according to claim 1 or 2, characterized in that, The determining key point information corresponding to at least some target objects in the live image frame includes: Performing a first scaling operation on the live image frame by a graphics processing unit to obtain a live image frame with a first preset resolution, and sending the live image frame with the first preset resolution to the central processing unit; Detecting, by the central processing unit, a predicted region corresponding to at least some target objects in the live image frame with the first preset resolution according to a preset first detection algorithm, and sending the predicted region corresponding to the at least some target objects to the graphics processing unit; The graphics processor performs a cropping operation on at least part of the target objects in the live image frame according to the predicted region, obtains a raw pixel map corresponding to at least part of the predicted region, and sends the raw pixel map corresponding to at least part of the predicted region to the central processing unit; The central processing unit determines the key points corresponding to the target objects in at least part of the predicted region according to a preset second detection algorithm.

8. The method according to claim 7, characterized in that The sending the raw pixel map corresponding to at least part of the predicted region to the central processing unit includes: The graphics processor performs a second scaling operation on the raw pixel map corresponding to at least part of the predicted region to obtain a raw pixel map with a second preset resolution corresponding to at least part of the predicted region; Send the raw pixel map with the second preset resolution corresponding to at least part of the predicted region to the central processing unit.

9. The method according to claim 1 or 2, characterized in that, The key point information includes the coordinate information of multiple key points corresponding to the target object; the performing a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes: The graphics processor determines a target region where at least part of the target objects are located in the live image frame according to the key point information corresponding to at least part of the target objects; For the target region, a special effect rendering operation is performed on the target region or the live image frame by using a rendering method matching the target special effect to obtain the target image frame.

10. The method according to claim 1, characterized in that, Before obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect, it further includes: The graphics processor obtains a raw image frame corresponding to the virtual reality live content collected by a binocular image acquisition device, and performs a hardware decoding operation and a format conversion operation on the raw image frame to obtain the live image frame.

11. The method according to claim 7, wherein The central processing unit detecting a predicted region corresponding to at least part of the target objects in the live image frame with the first preset resolution according to a preset first detection algorithm includes: Performing a detection operation on the target objects in the live image frame with the first preset resolution by using a preset first detection algorithm to determine a first region where at least part of the target objects are located; For at least two first regions that meet the preset merging condition, determining whether the size of the merged region after merging the at least two first regions is larger than the size of the at least two first regions before merging; If so, determining the first region as the predicted region; If not, determining the merged region as the predicted region.

12. The method according to any one of claims 1-5, 10-11, characterized in that Before obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect, it further includes: In response to a test instruction triggered by the user, obtaining a test image frame corresponding to the virtual reality live content; Performing a test operation on the test image frame by using a test method corresponding to the test type according to the test type corresponding to the test instruction; The obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect includes: When the test image frame meets the preset live conditions, obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect.

13. The method according to claim 12, characterized in that, The test types include a first test type. According to the test type corresponding to the test instruction, a test operation is performed on the test image frame by using a test method corresponding to the test type, including: Performing brightness detection on the test image frame through a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

14. The method according to claim 13, wherein After performing brightness detection on the test image frame through a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result, it further includes: If it is detected that the brightness detection result meets the preset live broadcast condition, acquiring a live image frame corresponding to the virtual reality live broadcast content and a preset target special effect; If it is detected that the brightness detection result does not meet the preset live broadcast condition, displaying a preset first prompt message, where the first prompt message is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold.

15. The method according to claim 12, wherein The test types include a second test type. According to the test type corresponding to the test instruction, a test operation is performed on the test image frame by using a test method corresponding to the test type, including: Detecting the test image frame through a focus test algorithm corresponding to the second test type to obtain a focus detection result.

16. The method according to claim 15, characterized in that, After detecting the test image frame through a focus test algorithm corresponding to the second test type to obtain a focus detection result, it further includes: If it is detected that the focus detection result meets the preset live broadcast condition, acquiring a live image frame corresponding to the virtual reality live broadcast content and a preset target special effect; If it is detected that the focus detection result does not meet the preset live broadcast condition, displaying a preset second prompt message, where the second prompt message is used to prompt the user to perform a focus operation again.

17. A live special effect rendering device, characterized in that, Including: An acquisition module, configured to acquire a live image frame corresponding to the virtual reality live broadcast content and a preset target special effect; A determination module, configured to determine key point information corresponding to at least some target objects in the live image frame; A rendering module, configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame; A display module, configured to display the target image frame.

18. An electronic device, characterized in that, Including: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the live special effect rendering method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, Computer execution instructions are stored in the computer-readable storage medium. When the processor executes the computer execution instructions, the live special effect rendering method according to any one of claims 1 to 16 is implemented.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, the method for live special effect rendering according to any one of claims 1 to 16 is implemented.

Images