Live special effect rendering method, device, equipment, readable storage medium and product

Abstract

The embodiment of the present disclosure provides a live effect rendering method, device, equipment, readable storage medium and product, which comprises the following steps: acquiring a live image frame corresponding to virtual reality live content and a preset target effect; determining key point information corresponding to at least part of a target object in the live image frame; performing an effect rendering operation on the live image frame according to the target effect and the key point information to obtain a target image frame; and displaying the target image frame. Thus, the region for effect processing can be concentrated in the position associated with the key point information, the region for effect processing is effectively reduced, and the efficiency of effect processing is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of image processing, and particularly relate to a live broadcast special effect rendering method and device, equipment, readable storage medium and product. BACKGROUND

[0002] VR panorama live broadcast generally adopts binocular cameras for real-time shooting, and VR live broadcast generally adopts 8k video frames (7680*4320) or above, which is ultra-high definition video frames compared with traditional 2K (2048*unspecified value) and 720P (1280*720). Due to the requirement of time delay in the live broadcast process, the actual time left for special effect rendering is relatively short. Therefore, it is necessary to complete the algorithm and special effect rendering of 8k picture within a limited time to ensure good experience of VR live broadcast. How to ensure fast special effect rendering in the VR live broadcast process has become a technical problem to be solved. SUMMARY

[0003] Embodiments of the present disclosure provide a live broadcast special effect rendering method, device, equipment, readable storage medium and product, which are used to solve the technical problem that the special effect rendering speed of the collected ultra-high definition video frames is slow in the VR live broadcast scene, and the live broadcast effect cannot be guaranteed.

[0004] In a first aspect, embodiments of the present disclosure provide a live broadcast special effect rendering method, comprising:

[0005] obtaining a live broadcast image frame corresponding to virtual reality live broadcast content and a preset target special effect;

[0006] determining key point information corresponding to at least part of target objects in the live broadcast image frame;

[0007] performing a special effect rendering operation on the live broadcast image frame according to the target special effect and the key point information to obtain a target image frame;

[0008] displaying the target image frame.

[0009] In a second aspect, embodiments of the present disclosure provide a live broadcast special effect rendering device, comprising:

[0010] an obtaining module configured to obtain a live broadcast image frame corresponding to virtual reality live broadcast content and a preset target special effect;

[0011] a determining module configured to determine key point information corresponding to at least part of target objects in the live broadcast image frame;

[0012] a rendering module configured to perform a special effect rendering operation on the live broadcast image frame according to the target special effect and the key point information to obtain a target image frame;

[0013] a display module configured to display the target image frame.

[0014] In a third aspect, an electronic device is provided, and the electronic device includes a processor and a memory.

[0015] The memory stores computer-executable instructions.

[0016] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the live special effect rendering method according to the first aspect and various possible designs of the first aspect.

[0017] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the live special effect rendering method according to the first aspect and various possible designs of the first aspect is implemented.

[0018] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program. When a processor executes the computer program, the live special effect rendering method according to the first aspect and various possible designs of the first aspect is implemented.

[0019] The live special effect rendering method, device, equipment, readable storage medium and product provided by the embodiment can concentrate the region of special effect processing in the position associated with the key point information, effectively reduce the region that needs to be processed by the special effect, and further improve the efficiency of the special effect processing. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present disclosure, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0021] Figure 1 The system architecture schematic diagram based on the present disclosure;

[0022] Figure 2 The flowchart of the live special effect rendering method provided by the embodiment of the present disclosure;

[0023] Figure 3 The external expansion schematic diagram provided by the embodiment of the present disclosure;

[0024] Figure 4A flowchart of a live special effect rendering method provided for another embodiment of the present disclosure is shown in FIG. 6;

[0025] Figure 5 An application scenario diagram provided for an embodiment of the present disclosure is shown in FIG. 7;

[0026] Figure 6 A flowchart of a live special effect rendering method provided for another embodiment of the present disclosure is shown in FIG. 6;

[0027] Figure 7 A flowchart of a live special effect rendering method provided for another embodiment of the present disclosure is shown in FIG. 6;

[0028] Figure 8 An interface interaction diagram provided for an embodiment of the present disclosure is shown in FIG. 8;

[0029] Figure 9 Another interface interaction diagram provided for an embodiment of the present disclosure is shown in FIG. 9;

[0030] Figure 10 Another interface interaction diagram provided for an embodiment of the present disclosure is shown in FIG. 9;

[0031] Figure 11 A structure diagram of a live special effect rendering device provided for an embodiment of the present disclosure is shown in FIG. 10;

[0032] Figure 12 A structure diagram of an electronic device provided for an embodiment of the present disclosure is shown in FIG. 11. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described below in connection with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present disclosure.

[0034] In order to solve the technical problem that the speed of special effect rendering on the collected ultra-high-definition video frames is slow in the VR live scene, and the live effect cannot be guaranteed, the present disclosure provides a live special effect rendering method, device, equipment, readable storage medium and product.

[0035] It should be noted that the live special effect rendering method, device, equipment, readable storage medium and product provided by the present disclosure can be applied in the image rendering scene in any VR scene.

[0036] The existing VR panoramic live broadcast adopts a binocular camera to perform real-time shooting, usually adopts an 8K video frame (7680*4320) or above, and belongs to an ultra-high-definition video frame compared with a traditional 2K (2048*unspecified value) and 720P (1280*720). The 8K single-frame data amount is 7680*4320*4 Byte=126 MB, the live broadcast frame rate is generally required to be 30-60 fps, the upper limit of the single-frame time delay is 16 ms-33 ms, and the actual time window left for special effect rendering can be shorter, and the rendering of the algorithm and special effect of the 8K picture needs to be completed within a limited time to ensure a good experience of the VR live broadcast.

[0037] In the process of solving the above technical problems, the inventors have found that, in order to improve the speed of special effect rendering and ensure a good experience of the VR live broadcast, a graphics processor can be used for special effect rendering operation, and a central processing unit can be used for recognition and detection operation. In order to further improve the speed of special effect rendering, the range of special effect rendering can be concentrated on the anchor or the surrounding, so that the actual pixel area to be processed can be reduced. In addition, since the data amount of the live broadcast image frame is large, the time consumption of data transmission between the CPU and the GPU needs to be avoided. Therefore, before the live broadcast image frame acquired by the graphics processor is sent to the central processing unit, the live broadcast image frame can be compressed and / or cropped to reduce the data amount and improve the transmission speed.

[0038] Figure 1 The system architecture based on the present disclosure is shown in FIG. 1. Figure 1 As shown in FIG. 1, the system architecture based on the present disclosure at least includes a binocular image acquisition device 11 and a server 12, wherein the server 12 is provided with a graphics processing unit and a central processing unit, the graphics processing unit and the central processing unit are provided with a live special effect rendering device, and the live special effect rendering device can be written in C / C++, Java, Shell or Python.

[0039] Figure 2 The flowchart of the live special effect rendering method provided by the embodiment of the present disclosure is shown in FIG. 2. Figure 2 As shown in FIG. 2, the method includes the following steps.

[0040] In step 201, a live broadcast image frame corresponding to virtual reality live broadcast content and a preset target special effect are acquired.

[0041] The execution subject of the embodiment is a live special effect rendering device, which can be coupled to a server, and the server is respectively provided with a graphics processing unit and a central processing unit.

[0042] In this embodiment, when a user performs virtual reality (VR) live streaming, the user can select special effects, beautification, filters and other content according to actual needs to improve the live streaming effect. When the target special effect selected by the user is obtained, a special effect rendering operation needs to be performed on the live streaming content according to the target special effect to achieve a decoration effect.

[0043] In VR live streaming, in order to ensure the live streaming effect, a binocular image acquisition device is used to perform a live streaming content acquisition operation, and the live streaming image frame acquired by the binocular image acquisition device is usually an 8k image frame (7680*4320) or above, which has a large size and a long special effect rendering time.

[0044] Correspondingly, in order to perform a special effect rendering operation on live streaming content, a live streaming special effect rendering device can acquire a live streaming image frame corresponding to virtual reality live streaming content. The live streaming image frame can be acquired at a preset time interval, or the live streaming image frame can be acquired at a preset frequency, and the present disclosure does not limit this.

[0045] Correspondingly, in order to perform a special effect rendering operation on the live streaming image frame, a preset target special effect can also be acquired. The target special effect can be selected by a user according to actual needs during live streaming.

[0046] Step 202, determining key point information corresponding to at least part of a target object in the live streaming image frame.

[0047] In this embodiment, in order to improve the speed of special effect rendering and avoid live streaming process lag, the special effect rendering operation can be concentrated around at least part of a target object in the live streaming image frame, where the target object can be a person, an animal, a specific object or the like in the live streaming image frame.

[0048] Therefore, after the live streaming image frame is acquired, key point information corresponding to at least part of a target object in the live streaming image frame can be determined. Optionally, the key point information corresponding to at least part of a target object in the live streaming image frame can be determined according to a preset detection algorithm, where the key point information can be coordinate information of a key position in the target object.

[0049] Step 203, performing a special effect rendering operation on the live streaming image frame according to the target special effect and the key point information to obtain a target image frame.

[0050] In this embodiment, after the key point information is acquired, the target special effect can be used to perform a special effect rendering operation on the live image frame according to the key point information to obtain a target image frame. Thus, the special effect rendering operation is not required to be performed on all positions of the live image frame, and the efficiency of the special effect rendering is improved on the basis of optimizing the display effect of the target object.

[0051] In step 204, the target image frame is displayed.

[0052] In this embodiment, after the special effect rendering operation on the live image frame is completed and the target image frame is obtained, the target image frame can be displayed.

[0053] Optionally, the target image frame is distributed to a preset terminal device for display. The preset terminal device can be at least part of virtual reality devices for watching the VR live broadcast, so that a user watches the VR live broadcast through the virtual display device.

[0054] Alternatively, if the live special effect rendering apparatus is coupled in the terminal device, the target image frame can be directly controlled to be displayed on a preset display interface of the terminal device.

[0055] It should be noted that, since the server is respectively provided with a graphics processing unit and a central processing unit, the graphics processing unit can be used to perform the special effect rendering operation, and the central processing unit can be used to identify the key points. Alternatively, the central processing unit can be used to perform the special effect rendering operation, and the graphics processing unit can be used to identify the key points. The present disclosure does not limit this.

[0056] The live special effect rendering method provided in this embodiment can determine key point information in a live image frame corresponding to VR live broadcast content after the live image frame is acquired, and perform a special effect rendering operation on positions associated with the key point information in the live image frame according to the key point information. Thus, the area for special effect processing can be concentrated on the positions associated with the key point information, the area for special effect processing is effectively reduced, and the efficiency of the special effect processing is improved.

[0057] In actual applications, different target special effects can correspond to different display effects, and correspondingly have different rendering positions. For example, the rendering positions of special effects such as beautifying, face stickers, and head decorations can be faces or heads. The rendering position of special effects such as filters and global display is the entire live image frame. Therefore, different rendering modes can be used for rendering according to different target special effects.

[0058] Optionally, in any of the above embodiments, step 203 includes:

[0059] According to the key point information corresponding to the at least partial target object, a target region in which the at least partial target object is located in the live image frame is determined.

[0060] If the target special effect is a special effect applied to a local part, for at least partial target regions, a local rendering operation is performed on the target regions according to the target special effect, to obtain a target region rendering result.

[0061] For at least partial target regions, the target region rendering result is overlaid into the live image frame, to obtain the target image frame.

[0062] In this embodiment, the target special effect can be a special effect applied to a local part. For example, it can be a beauty special effect, which is a special effect applied to a face only. Or it can be a headwear special effect, which is a special effect applied to a head only. Therefore, for the special effect applied to a local part, a special effect rendering operation can be performed on the target regions only, and no special effect rendering operation is performed on other positions in the live image frame. Thus, the region for special effect processing can be concentrated on the positions associated with the key point information, effectively reducing the range of special effect rendering and improving the speed of special effect rendering.

[0063] Specifically, when the target special effect is a special effect applied to a local part, a local rendering operation can be performed on each target region according to the target special effect, to obtain a target region rendering result. After the special effect rendering is completed, the target region rendering result can be overlaid into the live image frame, to obtain the target image frame.

[0064] Further, based on any of the above embodiments, the local rendering operation performed on the target region according to the target special effect includes:

[0065] If it is detected that the target special effect satisfies a preset outward expansion condition, an outward expansion operation is performed on the target region according to a preset region outward expansion algorithm, to obtain a to-be-rendered region.

[0066] A local rendering operation is performed on the to-be-rendered region according to the target special effect.

[0067] In this embodiment, to ensure the rendering effect of the edge of the target region, an outward expansion operation can be performed on the target region. Specifically, if it is detected that the target special effect satisfies a preset outward expansion condition, an outward expansion operation is performed on the target region according to a preset region outward expansion algorithm, to obtain a to-be-rendered region. The preset outward expansion condition can be that when the target special effect is a special effect applied to a face or other preset position, the outward expansion operation can be performed. A local rendering operation is performed on the to-be-rendered region after the outward expansion according to the target special effect.

[0068] Figure 3 An outward expansion schematic diagram provided by the embodiments of the present disclosure is shown in FIG. 2.Figure 3 As shown, in order to obtain a better rendering effect, after the target region 31 is obtained, an outward expansion operation can be performed on the target region 31 to obtain a to-be-rendered region 32.

[0069] The live special effect rendering method provided in this embodiment can guarantee the effect of special effect rendering and improve the quality of live streaming by performing an outward expansion operation on a target region when a target special effect meets a preset outward expansion condition.

[0070] Optionally, based on any of the above embodiments, step 203 includes:

[0071] If the target special effect is a special effect applied to the whole, performing a special effect rendering operation on the live image frame according to the target special effect to obtain the target image frame.

[0072] In this embodiment, the target special effect can also include a special effect applied to the whole, such as a filter, raindrops displayed on the whole, and the like. Therefore, when the target special effect is a special effect applied to the whole, a special effect rendering operation can be performed on the live image frame according to the target special effect to obtain the target image frame.

[0073] The live special effect rendering method provided in this embodiment can concentrate the region for special effect processing on the position associated with the key point information by performing a special effect rendering operation on the position associated with the key point information in the live image frame according to the key point information, effectively reducing the region that needs to be processed, and further improving the efficiency of special effect processing.

[0074] It should be noted that, since the server is respectively provided with a graphics processor and a central processor, the central processor can be used to perform the key point identification operation according to the processing characteristics of different processors, and the graphics processor can be used to perform the special effect rendering processing based on the key point information.

[0075] Further, based on any of the above embodiments, step 202 includes:

[0076] Determining, by a preset central processor, key point information corresponding to at least part of a target object in the live image frame;

[0077] Step 203 includes:

[0078] Performing, by a preset graphics processor, a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame.

[0079] In the embodiment, the graphic processor is in communication connection with the central processor and the binocular image acquisition device respectively, so as to acquire the live image frame acquired by the binocular image acquisition device, detect by the central processor, and render special effect by the graphic processor.

[0080] Therefore, after acquiring the live image frame, the graphic processor can send the live image frame to the central processor. Correspondingly, after acquiring the live image frame, the central processor determines the key point information corresponding to at least part of the target object in the live image frame according to the preset detection algorithm, wherein the key point information can be the coordinate information of the key position in the target object. And the key point information is fed back to the graphic processor.

[0081] After acquiring the key point information, the graphic processor can render the live image frame according to the key point information by using the rendering mode corresponding to the target special effect.

[0082] The live special effect rendering method provided in the embodiment can calculate the key point information in the live image frame by using the central processor after acquiring the live image frame corresponding to the virtual reality live content, and render the position associated with the key point information in the live image frame according to the key point information by using the graphic processor, so as to concentrate the special effect processing area in the position associated with the key point information, effectively reduce the area that needs to be processed, and further improve the efficiency of special effect processing. In addition, by using the graphic processor to perform special effect processing operation, the transmission of a large amount of live image frames can be effectively avoided, the data transmission time is reduced, and the efficiency of special effect processing can be further improved.

[0083] Further, on the basis of any of the above embodiments, step 202 comprises:

[0084] performing size adjustment operation on the live image frame to obtain an adjusted live image frame.

[0085] determining, by the central processor, key point information corresponding to at least part of the target object in the adjusted live image frame.

[0086] In the embodiment, the live image frame is generally 8K (7680*4320) or above, and the size is large. Therefore, the key point recognition operation based on the live image frame takes a long time. In order to ensure the live effect, the live image frame can be subjected to size adjustment operation before the key point information of the live image frame is identified, to obtain an adjusted live image frame. The size adjustment operation can be a size scaling operation on the live image frame, which scales the live image frame to a 1K image frame, and then the key point recognition based on the adjusted live image frame is efficient.

[0087] Further, since the server is respectively provided with a graphic processor and a central processor, the central processor can be used to identify the key point information. In addition, the graphic processor can be used to acquire the live image frame, or the central processor can be used to acquire the live image frame, or the user can set according to the actual demand, and the execution subject of the live image frame acquisition in the embodiment is not limited.

[0088] Therefore, after completing the size adjustment of the live image frame and obtaining the adjusted live image frame, the central processor can be used to determine the key point information corresponding to at least part of the target object in the adjusted live image frame.

[0089] The live effect rendering method provided in the embodiment can effectively reduce the calculation amount in the key point identification process and improve the efficiency of live image frame rendering by adjusting the size of the live image frame after obtaining the live image frame corresponding to the virtual reality live content and using the central processor to calculate the key point information in the live image frame. Further, the virtual reality live can be smooth and not stuck, and the user experience can be improved.

[0090] Figure 4 The flowchart of the live effect rendering method provided in another embodiment of the disclosure is shown in the above embodiment, and as shown in Figure 4 , step 202 includes:

[0091] Step 401, performing a first scaling operation on the live image frame by the graphic processor to obtain a live image frame of a first preset resolution, and sending the live image frame of the first preset resolution to the central processor.

[0092] Step 402, detecting a prediction area corresponding to at least part of the target object in the live image frame of the first preset resolution according to a preset first detection algorithm by the central processor, and sending the prediction area corresponding to the at least part of the target object to the graphic processor.

[0093] Step 403, performing a clipping operation on at least part of the target object in the live image frame according to the prediction area by the graphic processor to obtain an original pixel map corresponding to at least part of the prediction area, and sending the original pixel map corresponding to the at least part of the prediction area to the central processor.

[0094] Step 404, determining a key point corresponding to the target object in at least part of the prediction area according to a preset second detection algorithm by the central processor.

[0095] In this embodiment, since the pixel value of the live image frame is generally (7680*4320) or above, the single frame data amount is 7680*4320*4 Byte=126 MB, and therefore the transmission time length of the live image frame is also relatively long.

[0096] Optionally, the live image frame acquisition and the special effect rendering operation can be performed by the graphics processor, and the key point information identification operation can be performed by the central server. Therefore, after the graphics processor acquires the live image frame, the live image frame needs to be sent to the central processor for key point detection. In order to improve the speed of special effect rendering, the data amount of the transmission data can be reduced in the data transmission process. Specifically, the graphics processor can perform a first scaling operation on the live image frame to obtain a live image frame of a first preset resolution, and send the live image frame of the first preset resolution to the central processor. In actual application, the corresponding first preset resolution can be set according to actual needs, and the present disclosure does not limit this. For example, the 8K live image frame can be scaled to a 1K live image frame.

[0097] After the central processor acquires the live image frame of the first preset resolution, since the content clarity of the live image frame of the first preset resolution is lower than that of the original live image frame, the central processor can perform coarse-grained prediction on the region where the target object in the live image frame of the first preset resolution is located to obtain a predicted region corresponding to at least part of the target object in the live image frame of the first preset resolution. And send the predicted region corresponding to at least part of the target object in the live image frame of the first preset resolution to the graphics processor.

[0098] After the graphics processor acquires the predicted region corresponding to at least part of the target object, the graphics processor can directly perform special effect rendering operation on the predicted region to obtain a target image frame. Optionally, in order to further improve the accuracy of special effect rendering, the graphics processor can also perform cropping operation on the target object according to the predicted region corresponding to at least part of the target object to obtain an original pixel map corresponding to at least part of the predicted region. Wherein, the pixels of the original pixel map are the same as those of the live image frame. Since the size of the original pixel map is much smaller than that of the live image frame, the transmission speed to the central processor is faster.

[0099] Correspondingly, after the central processor acquires the at least part of the original pixel map, the central processor can identify the key point information of the target object in the original pixel map to obtain the key point information of at least part of the target object and feed back to the graphics processor. When the target object is a person, the key point information can be coordinate information of key positions such as the head and facial features of the person.

[0100] Figure 5 The application scenario diagram provided by the embodiment of the present disclosure is as follows:Figure 5 As shown, the graphic processor 51 can perform a first scaling operation on the live image frame 52 to obtain a live image frame 53 of a first preset resolution, and transmit the live image frame 53 of the first preset resolution to the central processor 54. The central processor 54 can perform detection of a prediction region in the live image frame 53 to obtain a prediction region 55 corresponding to at least part of a target object in the live image frame. The prediction region 55 corresponding to at least part of the target object in the live image frame is transmitted to the graphic processor 51, so that the graphic processor 51 can perform a cropping operation on the prediction region 55 corresponding to at least part of the target object in the live image frame, and transmit an original pixel map 56 corresponding to the at least part of the prediction region after cropping to the central processor 54. The central processor 54 can perform detection of a key point in the original pixel map 56 corresponding to the at least part of the prediction region, and feed back the key point to the graphic processor 51, so that the graphic processor 51 performs a special effect rendering operation on the live image frame according to the key point information to obtain a target image frame.

[0101] Further, on the basis of any of the above embodiments, step 404 comprises:

[0102] performing, by the graphic processor, a second scaling operation on the original pixel map corresponding to the at least part of the prediction region to obtain an original pixel map of a second preset resolution corresponding to the at least part of the prediction region.

[0103] transmitting the original pixel map of the second preset resolution corresponding to the at least part of the prediction region to the central processor.

[0104] In this embodiment, in order to further improve the speed of image transmission, before the original pixel map is transmitted, a second scaling operation can be performed on the original pixel map to obtain an original pixel map of a second preset resolution corresponding to the at least part of the prediction region. The scaling scale of the second scaling operation is smaller than the scaling scale of the first scaling operation, that is, the second preset resolution is greater than the first preset resolution. The original pixel map of the second preset resolution corresponding to the at least part of the prediction region is transmitted to the central processor.

[0105] The live special effect rendering method provided in this embodiment can effectively reduce the data amount of data transmission by transmitting the live image frame of the first preset resolution after scaling to the central processor, and transmitting the original pixel map corresponding to the at least part of the prediction region after cropping to the central processor, thereby improving the speed of data transmission between the graphic processor and the central processor, and further improving the speed of special effect rendering in a VR live scenario.

[0106] Further, on the basis of any of the above embodiments, step 402 comprises:

[0107] detecting, by a preset first detection algorithm, the target object in the live image frame of the first preset resolution to determine a first region in which at least part of the target object is located;

[0108] For at least two first regions satisfying a preset merging condition, it is determined whether a size of a merged region after merging of the at least two first regions is greater than a size of the at least two first regions when not merged.

[0109] If yes, the first region is determined as the prediction region.

[0110] If no, the merged region is determined as the prediction region.

[0111] In the embodiment, in the process of generating the prediction region, in order to reduce the calculation amount of subsequent special effect rendering, a merging operation can be performed on the regions satisfying the preset merging condition. Specifically, the target object in the live image frame of the first preset resolution can be detected by a preset first detection algorithm to determine the first region in which at least part of the target object is located.

[0112] It is determined whether at least part of the first region satisfies a preset merging condition, wherein the preset merging condition includes but is not limited to that a distance between at least two first regions is less than a preset distance threshold, at least two first regions have an intersection, any first region has a large coverage area while first regions around it have small coverage areas, etc.

[0113] For at least two first regions satisfying a preset merging condition, it is determined whether a size of a merged region after merging of the at least two first regions is greater than a size of the at least two first regions when not merged. If yes, the first region is determined as the prediction region. If no, the merged region is determined as the prediction region.

[0114] The live special effect rendering method provided in the embodiment can improve the special effect rendering speed by using a graphics processor to perform special effect rendering operation and using a central processing unit to perform identification and detection operation. In order to further improve the special effect rendering speed, the range of special effect rendering can be concentrated on the anchor or around the anchor, so that the pixel region actually needed to be processed can be reduced, and thus the efficiency of special effect processing can be improved.

[0115] Figure 6 The flowchart of the live special effect rendering method provided in another embodiment of the present disclosure is based on any of the above embodiments, and the key point information includes coordinate information of a plurality of key points corresponding to the target object. As shown in Figure 6 Step 203 includes:

[0116] In step 601, the graphic processor determines a target region in which the at least partial target object is located in the live image frame according to the key point information corresponding to the at least partial target object.

[0117] In step 602, a special effect rendering operation is performed on the target region or the live image frame by using a rendering mode matched with the target special effect corresponding to the target region, to obtain the target image frame.

[0118] In this embodiment, in order to improve the speed of the live image frame special effect rendering and ensure the live effect, the region for special effect processing is concentrated in the position associated with the key point information.

[0119] Therefore, after the key point information corresponding to the at least partial target object is obtained, the graphic processor can determine a target region in which the at least partial target object is located in the live image frame according to the key point information corresponding to the at least partial target object. For each target region, a special effect rendering operation can be performed on the target region or the live image frame according to the rendering mode corresponding to the target special effect preset by the user, to obtain the target image frame.

[0120] The live special effect rendering method provided in this embodiment can improve the efficiency of the special effect rendering by concentrating the rendering region around the target region.

[0121] Further, before step 201, the method further includes:

[0122] The graphic processor obtains an original image frame corresponding to the virtual reality live content collected by the binocular image collection device, and performs a hardware decoding operation and a format conversion operation on the original image frame, to obtain the live image frame.

[0123] In this embodiment, in order to further improve the speed of the special effect rendering and avoid excessive information interaction between the graphic processor and the central processing unit, the preprocessing of the original image frame is performed by the graphic processor.

[0124] Correspondingly, the live special effect rendering device can obtain an original image frame corresponding to the virtual reality live content collected by the binocular image collection device. The original image frame is subjected to a hardware decoding operation and a format conversion operation, to obtain the live image frame.

[0125] The live special effect rendering method provided in this embodiment can effectively avoid the time delay caused by excessive transmission of the live image frame and improve the rendering speed of the live image frame by performing the preprocessing of the original image frame by the graphic processor.

[0126] Figure 7A flowchart of a live special effect rendering method provided for another embodiment of the present disclosure is shown in FIG. 7, which is based on any of the above embodiments. As shown in FIG. 7, before step 201, the method further includes: Figure 7

[0127] Step 701: In response to a test instruction triggered by a user, a test image frame corresponding to the virtual reality live content is obtained.

[0128] Step 702: According to a test type corresponding to the test instruction, a test operation is performed on the test image frame by using a test mode corresponding to the test type.

[0129] Step 201 includes:

[0130] Step 703: When the test image frame meets a preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are obtained.

[0131] In this embodiment, in order to ensure the live effect of the virtual reality live, before performing special effect rendering on the live image frame corresponding to the virtual reality live, the current live effect needs to be tested first.

[0132] Optionally, a preset test control can be displayed on a live display interface, and a user can trigger the test control according to actual needs. In response to a test instruction triggered by the user triggering the test control, a test image frame corresponding to the virtual reality live content can be obtained. The test image frame can be an image frame captured by a preset binocular image capture device, and the test image frame can be composed of an image frame captured by a left image capture device in the binocular image capture device and an image frame captured by a right image capture device in the binocular image capture device.

[0133] Further, in order to ensure the live effect, the test image frame can be tested by different test types. For example, the test type can include a first test type and a second test type. The first test type can be a brightness test, and by performing the brightness test on the test image frame, the viewing effect of the live image frame can be ensured. The second test type can be a focusing test, so as to ensure the definition of the live image frame.

[0134] For different test types, different test modes can be preset. After determining the current test type, the test operation can be performed on the test image frame by using the test mode corresponding to the test type.

[0135] After completing the test operation on the test image frame, a test result can be obtained. After obtaining the test result, if it is detected that the test result meets a preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are obtained.​

[0136] Figure 8 An interface interaction schematic diagram provided by an embodiment of the present disclosure is shown in FIG. 8. As shown in FIG. 8, a preset test control 82 can be displayed on a live display interface 81. In response to a triggering operation of the test control 82 by a user, a test image frame 83 and a test result 84 can be displayed in a preset display area of the live display interface 81. Figure 8

[0137] The live effect rendering method provided by the embodiment can effectively ensure the display effect of virtual reality live streaming, thereby improving user experience.

[0138] Optionally, on the basis of any of the above embodiments, the test type includes a first test type, and step 702 includes:

[0139] The test image frame is subjected to brightness detection by using a brightness detection algorithm corresponding to the first test type, and a brightness detection result is obtained.

[0140] In the embodiment, the test type includes a first test type, which can be brightness testing. The test image frame is subjected to brightness testing, thereby ensuring the viewing effect of the live image frame.

[0141] Optionally, a preset brightness test control can be displayed on the live display interface. The user can trigger the first test type of test operation by triggering the brightness test control. When it is determined that the type of the current test operation is the first test type, the test image frame is subjected to brightness detection by using a brightness detection algorithm corresponding to the first test type, and a brightness detection result is obtained.

[0142] Any algorithm capable of realizing brightness detection can be used to realize brightness detection of the test image frame, and the present disclosure does not limit this.

[0143] Figure 9 Another interface interaction schematic diagram provided by an embodiment of the present disclosure is shown in FIG. 9. As shown in FIG. 9, a preset brightness test control 92 can be displayed on a live display interface 91. In response to a triggering operation of the brightness test control 92 by a user, a test image frame 93 and a brightness test result 94 can be displayed in a preset display area of the live display interface 91. Figure 9 Further, on the basis of any of the above embodiments, after the test image frame is subjected to brightness detection by using a brightness detection algorithm corresponding to the first test type, and a brightness detection result is obtained, the method further includes:

[0144]

[0145] ​​If it is detected that the brightness detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are acquired.

[0146] If it is detected that the brightness detection result does not meet the preset live broadcast condition, preset first prompt information is displayed, and the first prompt information is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold.

[0147] In this embodiment, after the brightness detection result is acquired, if it is detected that the brightness detection result meets the preset live broadcast condition, a live broadcast image frame corresponding to the virtual reality live broadcast content and a preset target special effect are acquired, so as to perform subsequent special effect rendering operation on the live broadcast image frame. Alternatively, if it is detected that the brightness detection result meets the preset live broadcast condition, prompt information indicating that the test is successful can also be displayed, so as to prompt the user to perform subsequent live broadcast operation. Conversely, if it is detected that the brightness detection result does not meet the preset live broadcast condition, preset first prompt information is displayed, and the first prompt information is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold. For example, the first prompt information can be: the current brightness is insufficient, please adjust the room light to 600 lumens.

[0148] The live broadcast special effect rendering method provided in this embodiment can guarantee the live broadcast effect of the virtual reality live broadcast, by detecting the brightness of the test image frame before performing special effect rendering on the live broadcast image frame, and performing subsequent live broadcast image frame rendering operation when the brightness meets the preset live broadcast condition.

[0149] Alternatively, on the basis of any of the above embodiments, the test type includes a second test type, and step 702 includes:

[0150] The test image frame is detected by using a focusing test algorithm corresponding to the second test type, to obtain a focusing detection result.

[0151] In this embodiment, the test type includes a second test type, and the second test type can be a focusing test, so as to guarantee the definition of the live broadcast image frame. When it is determined that the current test type is the second test type, the test image frame can be detected by using a focusing test algorithm corresponding to the second test type, to obtain a focusing detection result. Any algorithm that can realize focusing detection can be used to perform the test operation on the test image frame, and the present disclosure does not limit this.

[0152] Figure 10 Another interface interaction schematic diagram provided in the embodiments of the present disclosure is as follows: Figure 10As shown, the preset focus test control 1002 can be displayed on the live display interface 1001. In response to a triggering operation of the focus test control 1002 by the user, the test image frame 1003 and the focus test result 1004 can be displayed in a preset display area of the live display interface 1001.

[0153] Further, based on any of the above embodiments, after the focus detection result is obtained by detecting the test image frame through the focus test algorithm corresponding to the second test type, the method further includes:

[0154] If it is detected that the focus detection result meets the preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are obtained.

[0155] If it is detected that the focus detection result does not meet the preset live condition, a preset second prompt information is displayed, and the second prompt information is used to prompt the user to perform the focus operation again.

[0156] In this embodiment, if it is detected that the focus detection result meets the preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are obtained for subsequent special effect rendering operation on the live image frame. Alternatively, if it is detected that the focus detection result meets the preset live condition, a prompt information indicating that the test is successful can also be displayed to prompt the user to perform subsequent live operation. Conversely, if it is detected that the focus detection result does not meet the preset live condition, a preset second prompt information is displayed, and the second prompt information is used to prompt the user to perform the focus operation again.

[0157] Alternatively, the adjustment manner can also be determined according to the focus detection result, and the adjustment manner is displayed in the second prompt information, so that the user adjusts the focus according to the adjustment manner. For example, the second prompt information can be: focus failed, left camera adjusted backward.

[0158] The live special effect rendering method provided in this embodiment can detect whether the focus is completed before the live image frame is subjected to special effect rendering. If it is detected that the focus detection result meets the preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are obtained for subsequent special effect rendering operation on the live image frame. Therefore, the definition of the live image frame can be ensured, and the user experience is further improved.

[0159] Figure 11 The structure diagram of the live special effect rendering device provided in this embodiment of the present disclosure is applied to a graphics processor, such as Figure 11As shown, the apparatus includes: an acquisition module 1101, a determination module 1102, a rendering module 1103, and a display module 1104. The acquisition module 1101 is configured to acquire a live image frame corresponding to a virtual reality live broadcast content and a preset target special effect. The determination module 1102 is configured to determine key point information corresponding to at least part of a target object in the live image frame. The rendering module 1103 is configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information, to obtain a target image frame. The display module 1104 is configured to display the target image frame.

[0160] Further, on the basis of any of the above embodiments, the rendering module is configured to: determine a target region in which at least part of the target object is located in the live image frame according to the key point information corresponding to at least part of the target object. If the target special effect is a special effect applied to a local part, perform a local rendering operation on at least part of the target region according to the target special effect, to obtain a target region rendering result. Cover the target region rendering result to the live image frame for at least part of the target region, to obtain the target image frame.

[0161] Further, on the basis of any of the above embodiments, the rendering module is configured to: if it is detected that the target special effect satisfies a preset expansion condition, perform an expansion operation on the target region according to a preset region expansion algorithm, to obtain a to-be-rendered region. Perform a local rendering operation on the to-be-rendered region according to the target special effect.

[0162] Further, on the basis of any of the above embodiments, the rendering module is configured to: if the target special effect is a special effect applied to a global part, perform a special effect rendering operation on the live image frame according to the target special effect, to obtain the target image frame.

[0163] Further, on the basis of any of the above embodiments, the determination module is configured to determine the key point information corresponding to at least part of the target object in the live image frame by a preset central processing unit. The rendering module is configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information by a preset graphics processing unit, to obtain a target image frame.

[0164] Further, on the basis of any of the above embodiments, the determination module is configured to perform a size adjustment operation on the live image frame, to obtain an adjusted live image frame. The central processing unit is configured to determine the key point information corresponding to at least part of the target object in the adjusted live image frame.

[0165] Further, on the basis of any of the above embodiments, the determining module is configured to: perform, by the graphics processor, a first scaling operation on the live image frame to obtain a live image frame of a first preset resolution, and send the live image frame of the first preset resolution to the central processor; detect, by the central processor, a prediction region corresponding to at least part of the target object in the live image frame of the first preset resolution according to a preset first detection algorithm, and send the prediction region corresponding to the at least part of the target object to the graphics processor; perform, by the graphics processor, a cropping operation on the at least part of the target object in the live image frame according to the prediction region to obtain an original pixel map corresponding to at least part of the prediction region, and send the original pixel map corresponding to the at least part of the prediction region to the central processor; and determine, by the central processor, a key point corresponding to the target object in at least part of the prediction region according to a preset second detection algorithm.

[0166] Further, on the basis of any of the above embodiments, the determining module is configured to: perform, by the graphics processor, a second scaling operation on the original pixel map corresponding to the at least part of the prediction region to obtain an original pixel map of a second preset resolution corresponding to the at least part of the prediction region, and send the original pixel map of the second preset resolution corresponding to the at least part of the prediction region to the central processor.

[0167] Further, on the basis of any of the above embodiments, the key point information includes coordinate information of a plurality of key points corresponding to the target object. The rendering module is configured to: determine, by the graphics processor, a target region in which the at least part of the target object is located in the live image frame according to the key point information corresponding to the at least part of the target object, and perform, for the target region, a special effect rendering operation on the target region or the live image frame by using a rendering mode matched with the target special effect to obtain the target image frame.

[0168] Further, on the basis of any of the above embodiments, the device further includes a preprocessing module configured to: acquire, by the graphics processor, an original image frame corresponding to virtual reality live content collected by a binocular image collection device, and perform a hardware decoding operation and a format conversion operation on the original image frame to obtain the live image frame.

[0169] Further, on the basis of any of the above embodiments, the rendering module is configured to: perform a detection operation on the target object in the live image frame of the first preset resolution by using a first preset detection algorithm to determine a first region in which at least part of the target object is located. For at least two first regions that satisfy a preset merging condition, determine whether a size of a merged region after the at least two first regions are merged is greater than a size of the at least two first regions before the at least two first regions are merged. If yes, determine the first region as the prediction region. If no, determine the merged region as the prediction region.

[0170] Further, on the basis of any of the above embodiments, the device further comprises an acquisition module configured to acquire a test image frame corresponding to the virtual reality live content in response to a test instruction triggered by the user.

[0171] a test module configured to perform a test operation on the test image frame by using a test method corresponding to a test type corresponding to the test instruction; and the acquisition module is further configured to acquire a live image frame corresponding to the virtual reality live content and a preset target special effect when the test image frame satisfies a preset live condition.

[0172] Further, on the basis of any of the above embodiments, the test type comprises a first test type, and the test module is configured to: perform a brightness detection on the test image frame by using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0173] Further, on the basis of any of the above embodiments, the device further comprises a processing module configured to: acquire a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the brightness detection result satisfies a preset live condition; and the processing module is further configured to display a preset first prompt information if it is detected that the brightness detection result does not satisfy the preset live condition, the first prompt information being configured to prompt the user to adjust a brightness of a current location to a preset brightness threshold.

[0174] Further, on the basis of any of the above embodiments, the test type comprises a second test type, and the test module is configured to: perform a detection on the test image frame by using a focusing test algorithm corresponding to the second test type to obtain a focusing detection result.

[0175] Further, based on any of the above embodiments, the device further includes a processing module configured to acquire a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the focus detection result meets the preset live condition.

[0176] The device provided in the embodiment can be used to execute the technical solutions of the above method embodiments, and has similar implementation principles and technical effects. Details are not described herein.

[0177] To implement the above embodiments, the electronic device provided in the embodiment includes a processor and a memory.

[0178] The memory stores computer execution instructions.

[0179] The processor executes the computer execution instructions stored in the memory, so that the processor executes the live special effect rendering method according to any of the above embodiments.

[0180] Figure 12 The structure diagram of the electronic device provided in the embodiment of the present disclosure is shown in FIG. 12. Figure 12 As shown in FIG. 12, the electronic device 1200 can be a terminal device or a server. The terminal device can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 12 The electronic device shown in the figure is only an example, and should not bring any limitation to the functions and use range of the embodiment of the present disclosure.

[0181] As shown in FIG. 12, the electronic device 1200 can be a terminal device or a server. The terminal device can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a personal digital assistant (PDA), a tablet computer (PAD), a portable multimedia player (PMP), a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 12As shown, the electronic device 1200 can include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1201 that can perform various appropriate actions and processes according to programs stored in a Read Only Memory (ROM) 1202 or loaded into a Random Access Memory (RAM) 1203 from a storage device 1208. Various programs and data required by the electronic device 1200 for operation are also stored in the RAM 1203. The processing device 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0182] Generally, the following devices can be connected to the I / O interface 1205: input devices 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 1207 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, etc.; storage devices 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication devices 1209 can allow the electronic device 1200 to communicate wirelessly or wired with other devices to exchange data. Although Figure 12 The electronic device 1200 is shown with various devices, but it should be understood that not all of the shown devices are required to be implemented or present. More or fewer devices can alternatively be implemented or present.

[0183] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 1209, or installed from the storage devices 1208, or installed from the ROM 1202. When the computer program is executed by the processing device 1201, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0184] It should be noted that the computer readable medium in the above disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the above two. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0185] The embodiment of the present disclosure further provides a computer readable storage medium, which stores computer execution instructions. When a processor executes the computer execution instructions, the live special effect rendering method according to any of the above embodiments is implemented.

[0186] The embodiment of the present disclosure further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method of live special effect rendering according to any of the above embodiments is implemented.

[0187] The computer readable medium described above can be included in the electronic device described above; or can exist separately and not be assembled into the electronic device.

[0188] The computer readable medium described above carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0189] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0190] The computer program instructions can also be loaded onto a computer or other programmable information processing apparatus to cause a series of operations to be performed on the computer or other programmable information processing apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable information processing apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0191] The units described in the embodiments of the present disclosure can be implemented by hardware, software, or a combination thereof. Among them, the name of the unit does not constitute a limitation to the unit itself in some cases. For example, the first obtaining unit can also be described as "a unit for obtaining at least two Internet protocol addresses".

[0192] The functions described in this specification above can be performed at least in part by one or more hardware logic components. For example, non-limiting examples of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0193] ​In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0194] In a first aspect, according to one or more embodiments of the present disclosure, a live special effect rendering method is provided, comprising:

[0195] obtaining a live image frame corresponding to a virtual reality live content and a preset target special effect;

[0196] determining key point information corresponding to at least part of a target object in the live image frame;

[0197] performing a special effect rendering operation on the live image frame according to the target special effect and the key point information, to obtain a target image frame;

[0198] displaying the target image frame.

[0199] According to one or more embodiments of the present disclosure, the performing a special effect rendering operation on the live image frame according to the target special effect and the key point information, to obtain a target image frame, comprises:

[0200] determining a target region in which at least part of the target object is located in the live image frame according to the key point information corresponding to at least part of the target object;

[0201] if the target special effect is a special effect applied to a local part, performing a local rendering operation on the target region according to the target special effect, to obtain a target region rendering result, for at least part of the target region;

[0202] covering the target region rendering result to the live image frame to obtain the target image frame, for at least part of the target region.

[0203] According to one or more embodiments of the present disclosure, the performing a local rendering operation on the target region according to the target special effect comprises:

[0204] If it is detected that the target special effect satisfies a preset external expansion condition, an external expansion operation is performed on the target region according to a preset region external expansion algorithm, and a to-be-rendered region is obtained;

[0205] According to the target special effect, a local rendering operation is performed on the to-be-rendered region.

[0206] According to one or more embodiments of the present disclosure, the special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes:

[0207] If the target special effect is a special effect applied to the whole, a special effect rendering operation is performed on the live image frame according to the target special effect to obtain the target image frame.

[0208] According to one or more embodiments of the present disclosure, the determination of the key point information corresponding to at least part of the target objects in the live image frame includes:

[0209] The key point information corresponding to at least part of the target objects in the live image frame is determined by a preset central processing unit;

[0210] The special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes:

[0211] The special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame is performed by a preset graphics processing unit.

[0212] According to one or more embodiments of the present disclosure, the determination of the key point information corresponding to at least part of the target objects in the live image frame includes:

[0213] A size adjustment operation is performed on the live image frame to obtain an adjusted live image frame;

[0214] The key point information corresponding to at least part of the target objects in the adjusted live image frame is determined by the central processing unit.

[0215] According to one or more embodiments of the present disclosure, the determination of the key point information corresponding to at least part of the target objects in the live image frame includes:

[0216] A first scaling operation is performed on the live image frame by the graphics processing unit to obtain a live image frame of a first preset resolution, and the live image frame of the first preset resolution is sent to the central processing unit.

[0217] detect, by the central processor, a prediction region corresponding to at least part of a target object in the live image frame of the first preset resolution according to a preset first detection algorithm, and send the prediction region corresponding to the at least part of the target object to the graphics processor;

[0218] perform, by the graphics processor, a cropping operation on the at least part of the target object in the live image frame according to the prediction region, obtain an original pixel map corresponding to at least part of the prediction region, and send the original pixel map corresponding to the at least part of the prediction region to the central processor;

[0219] determine, by the central processor, a key point corresponding to a target object in at least part of the prediction region according to a preset second detection algorithm.

[0220] According to one or more embodiments of the present disclosure, the sending of the original pixel map corresponding to the at least part of the prediction region to the central processor comprises:

[0221] perform, by the graphics processor, a second scaling operation on the original pixel map corresponding to the at least part of the prediction region, to obtain an original pixel map of a second preset resolution corresponding to the at least part of the prediction region;

[0222] send the original pixel map of the second preset resolution corresponding to the at least part of the prediction region to the central processor.

[0223] According to one or more embodiments of the present disclosure, the key point information comprises coordinate information of a plurality of key points corresponding to a target object; and the performing of the special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame comprises:

[0224] determine, by the graphics processor, a target region in which the at least part of the target object is located in the live image frame according to the key point information corresponding to the at least part of the target object;

[0225] perform, for the target region, a special effect rendering operation on the target region or the live image frame by using a rendering mode matched with the target special effect, to obtain the target image frame.

[0226] According to one or more embodiments of the present disclosure, before the obtaining of the live image frame corresponding to the virtual reality live content and the preset target special effect, the method further comprises:

[0227] obtain, by the graphics processor, an original image frame corresponding to the virtual reality live content collected by a binocular image collection device, and perform a hardware decoding operation and a format conversion operation on the original image frame to obtain the live image frame.

[0228] According to one or more embodiments of the present disclosure, the detecting, by the central processor, a prediction region corresponding to at least part of a target object in the live image frame of the first preset resolution according to a preset first detection algorithm comprises:

[0229] detecting, by a preset first detection algorithm, the target object in the live image frame of the first preset resolution to determine a first region in which the at least part of the target object is located;

[0230] for at least two first regions satisfying a preset merging condition, determining whether a size of a merged region after merging of the at least two first regions is greater than a size of the at least two first regions before merging;

[0231] if yes, determining the first region as the prediction region;

[0232] if no, determining the merged region as the prediction region.

[0233] According to one or more embodiments of the present disclosure, before the acquiring of the live image frame corresponding to the virtual reality live content and the preset target special effect, the method further comprises:

[0234] acquiring a test image frame corresponding to the virtual reality live content in response to a test instruction triggered by a user;

[0235] performing a test operation on the test image frame by using a test mode corresponding to a test type corresponding to the test instruction;

[0236] the acquiring of the live image frame corresponding to the virtual reality live content and the preset target special effect comprises:

[0237] when the test image frame satisfies a preset live condition, acquiring the live image frame corresponding to the virtual reality live content and the preset target special effect.

[0238] According to one or more embodiments of the present disclosure, the test type comprises a first test type, and the performing of the test operation on the test image frame by using the test mode corresponding to the test type corresponding to the test instruction comprises:

[0239] performing a brightness detection on the test image frame by using a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

[0240] According to one or more embodiments of the present disclosure, after the performing of the brightness detection on the test image frame by using the brightness detection algorithm corresponding to the first test type to obtain the brightness detection result, the method further comprises:

[0241] If it is detected that the brightness detection result meets the preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are acquired.

[0242] If it is detected that the brightness detection result does not meet the preset live condition, preset first prompt information is displayed, and the first prompt information is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold.

[0243] According to one or more embodiments of the present disclosure, the test type includes a second test type, and the test type corresponding to the test instruction is used to perform a test operation on the test image frame by using a test method corresponding to the test type, including:

[0244] The test image frame is detected by using a focusing test algorithm corresponding to the second test type to obtain a focusing detection result.

[0245] According to one or more embodiments of the present disclosure, after the test image frame is detected by using the focusing test algorithm corresponding to the second test type to obtain the focusing detection result, the method further includes:

[0246] If it is detected that the focusing detection result meets the preset live condition, a live image frame corresponding to the virtual reality live content and a preset target special effect are acquired.

[0247] If it is detected that the focusing detection result does not meet the preset live condition, preset second prompt information is displayed, and the second prompt information is used to prompt the user to perform a focusing operation again.

[0248] In a second aspect, according to one or more embodiments of the present disclosure, a live special effect rendering device is provided, including:

[0249] An acquisition module is configured to acquire a live image frame corresponding to virtual reality live content and a preset target special effect.

[0250] A determination module is configured to determine key point information corresponding to at least part of a target object in the live image frame.

[0251] A rendering module is configured to perform a special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame.

[0252] A display module is configured to display the target image frame.

[0253] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0254] determine a target region in which the at least partial target object is located in the live image frame according to the key point information corresponding to the at least partial target object;

[0255] if the target special effect is a special effect applied to a local part, perform a local rendering operation on the target region according to the target special effect to obtain a target region rendering result for at least partial target regions;

[0256] overlay the target region rendering result to the live image frame to obtain the target image frame for at least partial target regions.

[0257] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0258] if it is detected that the target special effect satisfies a preset expansion condition, perform an expansion operation on the target region according to a preset region expansion algorithm to obtain a to-be-rendered region;

[0259] perform a local rendering operation on the to-be-rendered region according to the target special effect.

[0260] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0261] if the target special effect is a special effect applied to a global part, perform a special effect rendering operation on the live image frame according to the target special effect to obtain the target image frame.

[0262] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0263] determine, by a preset central processing unit, key point information corresponding to at least partial target objects in the live image frame;

[0264] The rendering module is configured to:

[0265] perform a special effect rendering operation on the live image frame according to the target special effect and the key point information by a preset graphics processing unit to obtain a target image frame.

[0266] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0267] perform a size adjustment operation on the live image frame to obtain an adjusted live image frame;

[0268] determine, by the central processing unit, key point information corresponding to at least partial target objects in the adjusted live image frame.

[0269] According to one or more embodiments of the present disclosure, the determining module is configured to:

[0270] The first scaling operation is performed on the live image frame by the graphic processor, and a live image frame of a first preset resolution is obtained, and the live image frame of the first preset resolution is sent to the central processor.

[0271] The central processor detects a prediction region corresponding to at least part of the target objects in the live image frame of the first preset resolution according to a preset first detection algorithm, and sends the prediction region corresponding to the at least part of the target objects to the graphic processor.

[0272] The graphic processor performs a cropping operation on at least part of the target objects in the live image frame according to the prediction region, and obtains an original pixel map corresponding to at least part of the prediction region, and sends the original pixel map corresponding to at least part of the prediction region to the central processor.

[0273] The central processor determines a key point corresponding to a target object in at least part of the prediction region according to a preset second detection algorithm.

[0274] According to one or more embodiments of the present disclosure, the determination module is configured to:

[0275] The graphic processor performs a second scaling operation on the original pixel map corresponding to at least part of the prediction region, and obtains an original pixel map of a second preset resolution corresponding to at least part of the prediction region.

[0276] The original pixel map of the second preset resolution corresponding to at least part of the prediction region is sent to the central processor.

[0277] According to one or more embodiments of the present disclosure, the key point information includes coordinate information of a plurality of key points corresponding to the target objects; and the rendering module is configured to:

[0278] The graphic processor determines a target region in which at least part of the target objects are located in the live image frame according to the key point information corresponding to at least part of the target objects.

[0279] For the target region, a special effect rendering operation is performed on the target region or the live image frame by using a rendering mode matched with the target special effect, and the target image frame is obtained.

[0280] According to one or more embodiments of the present disclosure, the device further includes a preprocessing module configured to:

[0281] The graphic processor acquires an original image frame corresponding to virtual reality live content collected by a binocular image collection device, and performs a hardware decoding operation and a format conversion operation on the original image frame, and obtains the live image frame.

[0282] According to one or more embodiments of the present disclosure, the rendering module is configured to:

[0283] detecting, by a preset first detection algorithm, a target object in the live image frame of the first preset resolution, to determine a first region in which at least part of the target object is located;

[0284] for at least two first regions that satisfy a preset merging condition, determining whether a size of a merged region after merging of the at least two first regions is greater than a size of the at least two first regions before merging;

[0285] if yes, determining the first region as the prediction region;

[0286] if no, determining the merged region as the prediction region.

[0287] According to one or more embodiments of the present disclosure, the apparatus further comprises an acquisition module configured to acquire a test image frame corresponding to the virtual reality live content in response to a test instruction triggered by a user; and a test module configured to perform a test operation on the test image frame by using a test method corresponding to a test type corresponding to the test instruction. The acquisition module is further configured to acquire a live image frame corresponding to the virtual reality live content and a preset target special effect when the test image frame satisfies a preset live condition.

[0288] According to one or more embodiments of the present disclosure, the test type comprises a first test type, and the test module is configured to perform a brightness detection on the test image frame by using a brightness detection algorithm corresponding to the first test type, to obtain a brightness detection result.

[0289] According to one or more embodiments of the present disclosure, the apparatus further comprises a processing module configured to acquire a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the brightness detection result satisfies a preset live condition; and the processing module is further configured to display a preset first prompt information if it is detected that the brightness detection result does not satisfy the preset live condition, the first prompt information being configured to prompt the user to adjust a brightness of a current location to a preset brightness threshold.

[0290] According to one or more embodiments of the present disclosure, the test type comprises a second test type, and the test module is configured to perform a detection on the test image frame by using a focusing test algorithm corresponding to the second test type, to obtain a focusing detection result.

[0291] According to one or more embodiments of the present disclosure, the device further includes a processing module configured to acquire a live image frame corresponding to the virtual reality live content and a preset target special effect if it is detected that the focus detection result meets the preset live condition.

[0292] In a third aspect, according to one or more embodiments of the present disclosure, an electronic device is provided, including at least one processor and a memory.

[0293] The memory stores computer-executable instructions.

[0294] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the live special effect rendering method according to the first aspect and various possible designs of the first aspect.

[0295] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, and the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the live special effect rendering method according to the first aspect and various possible designs of the first aspect is implemented.

[0296] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program. When a processor executes the computer program, the live special effect rendering method according to the first aspect and various possible designs of the first aspect is implemented.

[0297] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the applied technology. It should be understood by those skilled in the art that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combinations of the above technical features or their equivalent features without departing from the disclosed concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

[0298] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. Certain of the operations described in the discussion are combinable into a single operation, and certain operations can be separated into several operations. In some embodiments, the operations described in the discussion can be performed in an order different than presented in the discussion. In some embodiments, the operations described in the discussion can be performed concurrently. Also, while several specific implementation details are discussed in the discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0299] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A live effect rendering method, characterized by, The method comprises the following steps: obtaining a live image frame corresponding to virtual reality live content and a preset target special effect; determining key point information corresponding to at least part of target objects in the live image frame; performing special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame; displaying the target image frame; the step of determining the key point information corresponding to at least part of the target objects in the live image frame comprises the following steps: performing first scaling operation on the live image frame by a graphics processor to obtain a live image frame with a first preset resolution, and sending the live image frame with the first preset resolution to a central processing unit; detecting a prediction area corresponding to at least part of the target objects in the live image frame with the first preset resolution according to a preset first detection algorithm by the central processing unit, and sending the prediction area corresponding to the at least part of the target objects to the graphics processor; performing cropping operation on at least part of the target objects in the live image frame according to the prediction area by the graphics processor to obtain an original pixel map corresponding to at least part of the prediction area, and sending the original pixel map corresponding to at least part of the prediction area to the central processing unit; determining the key points corresponding to the target objects in at least part of the prediction area according to a preset second detection algorithm by the central processing unit.

2. The method of claim 1, wherein, the step of performing special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame comprises the following steps: determining a target area where at least part of the target objects are located in the live image frame according to the key point information corresponding to at least part of the target objects; if the target special effect is a special effect applied to a local part, performing local rendering operation on the target area according to the target special effect to obtain a target area rendering result for at least part of the target area; covering the target area rendering result to the live image frame for at least part of the target area to obtain the target image frame.

3. The method of claim 2, wherein, the step of performing local rendering operation on the target area according to the target special effect comprises the following steps: if it is detected that the target special effect meets a preset expansion condition, performing expansion operation on the target area according to a preset area expansion algorithm to obtain a to-be-rendered area; performing local rendering operation on the to-be-rendered area according to the target special effect.

4. The method of claim 1, wherein, the step of performing special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame comprises the following steps: if the target special effect is a special effect applied to a global part, performing special effect rendering operation on the live image frame according to the target special effect to obtain the target image frame.

5. The method of claim 1, wherein, the step of sending the original pixel map corresponding to at least part of the prediction area to the central processing unit comprises the following steps: performing second scaling operation on the original pixel map corresponding to at least part of the prediction area by a graphics processor to obtain an original pixel map corresponding to at least part of the prediction area with a second preset resolution; sending the original pixel map corresponding to at least part of the prediction area with the second preset resolution to the central processing unit.

6. The method of claim 1 or 2, wherein, The key point information includes coordinate information of a plurality of key points corresponding to the target object; and the special effect rendering operation on the live image frame according to the target special effect and the key point information to obtain a target image frame includes: determining, by a graphics processor, a target region where the at least part of the target objects are located in the live image frame according to the key point information corresponding to the at least part of the target objects; performing a special effect rendering operation on the target region or the live image frame in a rendering mode matched with the target special effect for the target region to obtain the target image frame.

7. The method of claim 1, wherein, Before the live image frame corresponding to the virtual reality live content and the preset target special effect are obtained, the method further includes: obtaining, by a graphics processor, an original image frame corresponding to the virtual reality live content collected by a binocular image collection device, and performing a hardware decoding operation and a format conversion operation on the original image frame to obtain the live image frame.

8. The method of claim 1, wherein, The detecting, by the central processing unit, a prediction region corresponding to the at least part of the target objects in the live image frame of the first preset resolution according to the preset first detection algorithm includes: detecting, by a preset first detection algorithm, the target objects in the live image frame of the first preset resolution to determine a first region where the at least part of the target objects are located; for at least two first regions that meet a preset merging condition, determining whether a size of a merged region after the at least two first regions are merged is greater than a size of the at least two first regions when the at least two first regions are not merged; if yes, determining the first region as the prediction region; if no, determining the merged region as the prediction region.

9. The method of any one of claims 1-4, 7-8, wherein, Before the live image frame corresponding to the virtual reality live content and the preset target special effect are obtained, the method further includes: obtaining a test image frame corresponding to the virtual reality live content in response to a test instruction triggered by a user; performing a test operation on the test image frame in a test mode corresponding to a test type corresponding to the test instruction according to the test type; The live image frame corresponding to the virtual reality live content and the preset target special effect are obtained, including: when the test image frame meets a preset live condition, obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect.

10. The method of claim 9, wherein, The test type includes a first test type, and the test operation on the test image frame in the test mode corresponding to the test type corresponding to the test instruction according to the test type includes: performing a brightness detection on the test image frame by a brightness detection algorithm corresponding to the first test type to obtain a brightness detection result.

11. The method of claim 10, wherein, After the brightness detection on the test image frame by the brightness detection algorithm corresponding to the first test type to obtain the brightness detection result, the method further includes: if it is detected that the brightness detection result meets the preset live condition, obtaining the live image frame corresponding to the virtual reality live content and the preset target special effect. If it is detected that the brightness detection result does not satisfy the preset live broadcast condition, preset first prompt information is displayed, and the first prompt information is used to prompt the user to adjust the brightness of the current location to a preset brightness threshold.

12. The method of claim 9, wherein, The test type includes a second test type, and the test image frame is tested according to the test type corresponding to the test instruction by using a test mode corresponding to the test type, including: The test image frame is detected by using a focusing test algorithm corresponding to the second test type to obtain a focusing detection result.

13. The method of claim 12, wherein, After the test image frame is detected by using the focusing test algorithm corresponding to the second test type to obtain the focusing detection result, the method further includes: If it is detected that the focusing detection result satisfies the preset live broadcast condition, a live broadcast image frame corresponding to virtual reality live broadcast content and a preset target special effect are obtained. If it is detected that the focusing detection result does not satisfy the preset live broadcast condition, preset second prompt information is displayed, and the second prompt information is used to prompt the user to re-perform a focusing operation.

14. A live effect rendering apparatus, comprising: The method includes: An acquisition module is configured to acquire a live broadcast image frame corresponding to virtual reality live broadcast content and a preset target special effect. A determination module is configured to determine key point information corresponding to at least part of target objects in the live broadcast image frame. A rendering module is configured to perform special effect rendering operation on the live broadcast image frame according to the target special effect and the key point information to obtain a target image frame. A display module is configured to display the target image frame. The determination module is configured to perform first scaling operation on the live broadcast image frame by using a graphics processing unit to obtain a live broadcast image frame of a first preset resolution, and send the live broadcast image frame of the first preset resolution to a central processing unit. The central processing unit is configured to detect a prediction region corresponding to at least part of target objects in the live broadcast image frame of the first preset resolution according to a preset first detection algorithm, and send the prediction region corresponding to the at least part of target objects to the graphics processing unit. The graphics processing unit is configured to perform cropping operation on at least part of target objects in the live broadcast image frame according to the prediction region to obtain an original pixel map corresponding to at least part of the prediction region, and send the original pixel map corresponding to at least part of the prediction region to the central processing unit. The central processing unit is configured to determine key points corresponding to target objects in at least part of the prediction region according to a preset second detection algorithm.

15. An electronic device, comprising: The method includes: A processor and a memory; The memory stores computer execution instructions; The processor executes the computer execution instructions stored in the memory, so that the processor executes the live broadcast special effect rendering method in any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, the live broadcast special effect rendering method in any one of claims 1 to 13 is implemented.

17. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the live broadcast special effect rendering method in any one of claims 1 to 13.

Images