Live image frame processing method, device, equipment, readable storage medium and product

Abstract

The embodiment of the present disclosure provides a live image frame processing method, device, equipment, readable storage medium and product, which comprises the following steps: obtaining an image frame processing request; obtaining a live image frame corresponding to virtual reality live content according to the image frame processing request; determining a to-be-processed region in the live image frame, and performing a first processing operation on the to-be-processed region to obtain a to-be-processed image frame; performing a second processing operation on the to-be-processed image frame according to a first color lookup table corresponding to the live image frame to obtain a target image frame after processing; and displaying the target image frame. Thus, the image frame processing operation in the VR scene can be realized. In addition, the first processing operation is performed on the to-be-processed region, and the second processing operation is performed on the whole, so that the calculation amount in the image processing process can be effectively reduced, the efficiency of the image beautification operation is improved, and thus the high-quality live effect can be ensured.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, device, readable storage medium, and product for processing live image frames. Background Technology

[0002] During live streaming, to achieve a better viewing experience, beautification technology can be used to optimize the content based on user actions. Existing beautification technologies can be applied to both PC and mobile devices, enabling beautification in 2D scenes.

[0003] With the development of technology, Virtual Reality (VR) technology is gradually entering users' lives. Users can use VR technology to achieve 3D VR live streaming. However, existing beautification technology cannot achieve beautification operations in VR scenes. Summary of the Invention

[0004] This disclosure provides a live image frame processing method, apparatus, device, readable storage medium, and product to solve the technical problem that existing beautification technologies cannot achieve image optimization operations in VR scenes.

[0005] In a first aspect, embodiments of this disclosure provide a method for processing live image frames, including:

[0006] Get image frame processing request;

[0007] According to the image frame processing request, obtain the live image frame corresponding to the virtual reality live content;

[0008] The region to be processed in the live image frame is determined, and a first processing operation is performed on the region to be processed to obtain the image frame to be processed.

[0009] A global second processing operation is performed on the image frame to be processed to obtain the processed target image frame;

[0010] Display the target image frame.

[0011] Secondly, embodiments of this disclosure provide a live image frame processing system, including: a terminal device, a binocular image acquisition device, and a virtual reality device; wherein...

[0012] The binocular image acquisition device is used to acquire live image frames corresponding to virtual reality live content.

[0013] The terminal device is used to acquire an image frame processing request, acquire a live image frame corresponding to the virtual reality live content according to the image frame processing request; determine the area to be processed in the live image frame, and perform a first processing operation on the area to be processed to obtain an image frame to be processed; perform a global second processing operation on the image frame to be processed to obtain a processed target image frame; and send the target image frame to the virtual reality device.

[0014] The virtual reality device is used to display the target image frame.

[0015] Thirdly, embodiments of this disclosure provide a live image frame processing apparatus, comprising:

[0016] The acquisition module is used to acquire image frame processing requests;

[0017] The image frame acquisition module is used to acquire the live image frame corresponding to the virtual reality live content according to the image frame processing request.

[0018] The determining module is used to determine the region to be processed in the live image frame and perform a first processing operation on the region to be processed to obtain the image frame to be processed.

[0019] The processing module is used to perform a global second processing operation on the image frame to be processed to obtain the processed target image frame;

[0020] The sending module is used to display the target image frame.

[0021] Fourthly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0022] The memory stores computer-executed instructions;

[0023] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the live image frame processing method as described in the first aspect and various possible designs of the first aspect.

[0024] Fifthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the live image frame processing method described in the first aspect and various possible designs of the first aspect.

[0025] In a sixth aspect, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the live image frame processing method as described in the first aspect and various possible designs of the first aspect.

[0026] The live image frame processing method, apparatus, device, readable storage medium, and product provided in this embodiment obtain live image frames corresponding to virtual reality live content according to an image frame processing request, perform a first processing operation on the area to be processed in the live image frame, perform a global second processing operation on the image frame after the first processing operation, and send the target image frame after the second processing operation to a preset terminal device for display, thereby realizing image processing operations in VR scenes. Furthermore, by performing a first processing operation on the area to be processed and then performing a global second processing operation, the computational load in the image processing process can be effectively reduced, the efficiency of image processing can be improved, and thus a high-quality live broadcast effect can be guaranteed. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of the live image frame processing system provided in the embodiments of this disclosure;

[0029] Figure 2 A schematic flowchart of a live image frame processing method provided in an embodiment of this disclosure;

[0030] Figure 3 A schematic flowchart of a live image frame processing method provided in another embodiment of this disclosure;

[0031] Figure 4 A schematic flowchart of a live image frame processing method provided in another embodiment of this disclosure;

[0032] Figure 5 A schematic flowchart of a live image frame processing method provided in another embodiment of this disclosure;

[0033] Figure 6 This is a schematic diagram of the interface interaction provided in the embodiments of this disclosure;

[0034] Figure 7 This is a schematic diagram of the structure of the live image frame processing apparatus provided in the embodiments of this disclosure;

[0035] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0037] To address the technical problem that existing technologies cannot achieve image optimization operations in VR scenarios, this disclosure provides a live image frame processing method, apparatus, device, readable storage medium, and product.

[0038] It should be noted that the live image frame processing method, apparatus, device, readable storage medium and product provided in this disclosure can be applied to any live image frame beautification scenario.

[0039] Existing beautification technologies are generally applied to 2D scenes and cannot beautify live image frames in VR scenes.

[0040] In solving the aforementioned technical problems, the inventors discovered through research that a live image frame processing system can be constructed, comprising a terminal device, a binocular image acquisition device, and a virtual reality device. Based on this system, the binocular image acquisition device simultaneously acquires real-time image data (4K per image) of the broadcaster from different angles, compressing the two images into a single 8K input image. The 8K image is then input to the terminal device for face recognition and rendering processing; this terminal device can be a personal computer (PC). The processed video image is then pushed to the virtual reality device. Finally, the virtual reality device's display client receives the live video stream and plays it in 3D, allowing users to watch a beautified 3D live stream.

[0041] Figure 1 This is a schematic diagram of the structure of the live image frame processing system provided in the embodiments of this disclosure, as shown below. Figure 1 As shown, the live image frame processing system may include a terminal device 11, a binocular image acquisition device 12, and a virtual reality device 13. The terminal device 11 is communicatively connected to the binocular image acquisition device 12 and the virtual reality device 13, respectively, so as to be able to interact with the binocular image acquisition device 12 and the virtual reality device 13.

[0042] Based on the above system architecture, the binocular image acquisition device 12 is used to acquire live image frames corresponding to virtual reality live content.

[0043] The terminal device 11 is used to acquire an image frame processing request, acquire a live image frame corresponding to the virtual reality live content according to the image frame processing request; determine the area to be processed in the live image frame, and perform a first processing operation on the area to be processed to obtain an image frame to be processed; perform a second processing operation on the image frame to be processed to obtain a processed target image frame; and send the target image frame to the virtual reality device 13.

[0044] The virtual reality device 13 is used to display the target image frame.

[0045] Based on the above system architecture, it is possible to perform beautification operations on live image frames in 3D scenes, thereby enabling users to watch beautified 3D live streams in virtual reality devices.

[0046] Figure 2 This is a flowchart illustrating the live image frame processing method provided in this embodiment of the disclosure, as shown below. Figure 2 As shown, the method includes:

[0047] Step 201: Obtain image frame processing request.

[0048] In this embodiment, the executing entity is a terminal device, which can be communicatively connected to both a binocular image acquisition device and a virtual reality device, thereby enabling information interaction with both. Optionally, the terminal device can be a PC host. Alternatively, the terminal device can be any device capable of image processing; this disclosure does not impose any limitations on this.

[0049] Optionally, the terminal device can also communicate with the user's live streaming device. When the user is live streaming in VR, they can select beautification technology to enhance the live streaming content according to their actual needs. Correspondingly, when the user triggers the beautification option on the terminal device, an image frame processing request can be generated based on the trigger operation, and the image frame processing request can be sent to the terminal device.

[0050] As one feasible approach, the live stream content can be automatically enhanced based on the user's live stream activation. Upon receiving the user's activation request, an image frame processing request can be automatically generated and sent to the terminal device.

[0051] Accordingly, the terminal device can obtain the image frame processing request.

[0052] Step 202: Obtain the live image frame corresponding to the virtual reality live content according to the image frame processing request.

[0053] In this embodiment, when an image frame processing request is received, the live image frame corresponding to the virtual reality live content can be obtained.

[0054] In VR live streaming scenarios, binocular image acquisition devices are typically used to capture live content. These devices obtain real-time images of the streamer from different angles, with each camera outputting a 4K image. Therefore, specific live image frames corresponding to the virtual reality live streaming content can be acquired.

[0055] Step 203: Determine the region to be processed in the live image frame, and perform a first processing operation on the region to be processed to obtain the image frame to be processed.

[0056] In this embodiment, during the enhancement of live image frames, skin smoothing and whitening operations can be performed on the users within the live image frames. However, since live image frames in VR live streaming scenarios are often large in size, the computational load for enhancing live image frames is substantial.

[0057] To improve the speed of beautifying live image frames, when performing skin smoothing operations on users in live image frames, the smoothing operation can be performed only on the user's face and exposed limbs in the live image frame, while other parts of the live image frame are not smoothed.

[0058] Therefore, after acquiring the live image frame, the first step is to identify the region to be processed within the live image frame. Optionally, this region to be processed can specifically be the user's face, exposed limbs, etc., in the live image frame. Alternatively, in different application scenarios, if the user-triggered request is for special effects processing on a target area, for example, if the user-triggered request is for adding a sticker to the head, then the region to be processed can specifically be the user's head in the live image frame.

[0059] After the region to be processed is identified, the first processing operation can be performed on the region to be processed to obtain the image frame to be processed. When the image frame processing request is a beautification request, this first processing operation can be a skin smoothing operation on the region to be processed.

[0060] Step 204: Perform a global second processing operation on the image frame to be processed to obtain the processed target image frame.

[0061] In this embodiment, after completing the first processing operation on the region to be processed, a global second processing operation can be performed on the image frame to be processed. Specifically, when the image frame processing request is a beautification request, this second processing operation can be a whitening operation.

[0062] In practical applications, the image frame to be processed may include the background in addition to the user. When performing a global second processing operation on the image frame, both the user and the background need to be optimized simultaneously. Therefore, to ensure the optimization effect of the second processing operation, a second color lookup table (LUT) corresponding to the live image frame can be pre-set. Specifically, the LUT corresponding to the live image frame can be adjusted. During the adjustment process, a set of LUTs that only affects skin tones and minimizes the impact on the background is determined, and this LUT is used as the second color lookup table.

[0063] Furthermore, a global color mapping operation can be performed on the image frame to be processed based on the first color lookup table to complete the second processing operation on the image frame to be processed and obtain the processed target image frame.

[0064] Specifically, the second global processing operation for the image frame to be processed can be image processing for the entire image frame to be processed.

[0065] Step 205: Display the target image frame.

[0066] In this embodiment, after the optimization operation of the live image frame is completed, the target image frame can be displayed.

[0067] Optionally, the target image frame can be sent to a preset terminal device for display, so that users can watch the beautified 3D live stream on the preset terminal device. Optionally, the preset terminal device can specifically be a virtual reality device.

[0068] Alternatively, after obtaining the target image frame, the target image frame can be displayed on a preset display interface in the terminal device.

[0069] Furthermore, based on any of the above embodiments, step 202 includes:

[0070] Acquire two image frames to be stitched together from at least two cameras.

[0071] The two image frames to be stitched together are stitched together to obtain the live image frame.

[0072] In this embodiment, at least two cameras can capture two image frames to be stitched together, each of which has a 4K image size. To facilitate subsequent live image frame optimization, the two image frames can be stitched together horizontally into a single 8K live image frame. The stitched 8K live image frame is then directly input into the terminal device for special effects processing without requiring distortion correction.

[0073] Optionally, the two cameras can be two cameras installed on a binocular image acquisition device. Specifically, it can acquire two image frames to be stitched together, captured by the two cameras in the binocular image acquisition device.

[0074] The live image frame processing method provided in this embodiment obtains the live image frame corresponding to the virtual reality live content according to the image frame processing request, performs a first processing operation on the area to be processed in the live image frame, performs a global second processing operation on the image frame after the first processing operation, and sends the target image frame after the second processing operation to the virtual reality device for display, thereby realizing image processing operations in VR scenes. Furthermore, by performing a first processing operation on the area to be processed and then performing a global second processing operation, the computational load in the image processing process can be effectively reduced, the efficiency of image processing can be improved, and thus a high-quality live broadcast effect can be guaranteed.

[0075] Figure 3 This is a flowchart illustrating a live image frame processing method according to another embodiment of the present disclosure. Based on any of the above embodiments, such as... Figure 3 As shown, step 203 includes:

[0076] Step 301: Perform a recognition operation on the live image frame to obtain the key points corresponding to each target object in the live image frame.

[0077] Step 302: Determine the target area of ​​each target object based on the key points and the preset target mask.

[0078] Step 303: Determine the non-processed area in the target area, adjust the pixel value corresponding to the non-processed area to a preset value, and obtain the area to be processed.

[0079] In this embodiment, after acquiring the live image frame, a preset key point recognition algorithm can be used to identify the key points in the live image frame, thereby obtaining the key points corresponding to each target object in the live image frame. It should be noted that since the input live image frame is composed of two image frames to be stitched together from the left and right angles, if there are N target objects in the image frame to be stitched, the live image frame will detect the facial key points corresponding to 2N target objects and output 2N sets of facial key points.

[0080] Furthermore, the target area of ​​each target object can be determined based on this key point and the preset target mask. This target mask includes multiple layers, with different layers recording the eyebrow, nostril, mouth, and eye areas respectively.

[0081] Understandably, the initial beautification operation often needs to be applied to the user's skin area. Therefore, to improve the subsequent optimization effect and reduce the subsequent computational load, non-processed areas within the target region can be identified. These non-processed areas are then reduced to obtain the areas to be processed. These non-processed areas include, but are not limited to, the user's hair, eyes, and eyebrows.

[0082] Optionally, in order to perform the deletion operation on the non-processed area, the pixel values ​​of the non-processed area can be adjusted to preset values ​​to obtain the area to be processed.

[0083] The live image frame processing method provided in this embodiment identifies key points in the live image frame and determines the target area where the target object is located based on the key points. This enables a first beautification operation to be performed based on the target area, reducing the scope of the first beautification operation and the computational load of the live image frame beautification.

[0084] Furthermore, based on any of the above embodiments, step 303 includes:

[0085] The live image frame is converted from an RGB image to an HSV image, and the hue and brightness of the HSV image are used as constraint information to determine the first unprocessed region in the target area.

[0086] By adjusting the layer corresponding to the target mask, a second unprocessed area in the target region is determined.

[0087] The pixel values ​​corresponding to the first unprocessed region and the second unprocessed region in the target region are set to zero to obtain the region to be processed.

[0088] In this embodiment, to perform the deletion operation on unprocessed areas, the live image frame can first be converted from an RGB image to an HSV image. HSV (Hue, Saturation, Value) is a color space created based on the intuitive characteristics of color. By using the hue and brightness of the HSV image as constraint information, a first unprocessed area that differs from the user's skin can be filtered out. This first unprocessed area includes, but is not limited to, non-skin areas such as hair and glasses.

[0089] Furthermore, the target mask can include multiple layers, with different layers recording the eyebrow, nostril, mouth, and eye areas respectively. By adjusting the different layers of the target mask, a second unprocessed area within the target region can be filtered. This second unprocessed area includes, but is not limited to, the eye, nostril, and mouth areas.

[0090] After identifying the first unprocessed area and the second unprocessed area, the first unprocessed area and the second unprocessed area in the target area can be deleted to obtain the area to be processed, which only includes the skin.

[0091] Optionally, the pixel values ​​corresponding to the first unprocessed region and the second unprocessed region can be adjusted to zero to achieve the deletion operation of the first unprocessed region and the second unprocessed region.

[0092] The live image frame processing method provided in this embodiment reduces the scope of the first beautification operation by deleting the first and second unprocessed regions in the target area, effectively reducing the computational load of live image frame beautification, improving the efficiency of live image frame beautification, and thus ensuring high-quality live streaming results.

[0093] Furthermore, based on any of the above embodiments, step 203 includes:

[0094] Determine the facial region in the live image frame;

[0095] A first processing operation is performed on at least one of the blemish area, shadow area, and highlight area in the facial region to obtain the image frame to be processed.

[0096] In this embodiment, the first processing operation can specifically be a facial processing operation. Specifically, the facial region in the live image frame can be determined, and any facial recognition method can be used to identify and extract the facial region. The first processing operation is performed on at least one of the blemish area, shadow area, and highlight area in the facial region to obtain the image frame to be processed. The blemish area can be an area with spots or acne, the highlight area can be an overexposed area, and the shadow area can be an area with low darkness.

[0097] By whitening the entire live stream image frame and smoothing the skin on the facial area, the live stream image frame can be effectively beautified, thereby improving the live streaming effect of virtual reality.

[0098] Figure 4 This is a flowchart illustrating a live image frame processing method according to another embodiment of the present disclosure. Based on any of the above embodiments, such as... Figure 4 As shown, step 203 includes:

[0099] Step 401: Identify the defective areas within the area to be processed, perform correction operations on the defective areas, and obtain a first texture map.

[0100] Step 402: Perform color correction on the first texture map to obtain the second texture map.

[0101] Step 403: Determine the high-frequency region information corresponding to the live image frame based on the second texture map and the live image frame, and determine the underlying skin texture information corresponding to the live image frame based on the second texture map.

[0102] Step 404: Perform a fusion operation on the high-frequency region information and the underlying skin texture information to obtain the image frame to be processed.

[0103] In this embodiment, the skin smoothing operation on the live image frame may specifically include optimization of blemishes such as blemishes and correction of the overall skin. Therefore, after obtaining the area to be processed, blemish areas within the area to be processed can first be identified, and correction operations can be performed on the blemish areas to obtain a first texture map.

[0104] Furthermore, color correction can be performed on the first texture image to brighten skin curves and remove yellow and red hues, resulting in a second texture image. Based on the second texture image and the live image frame, the high-frequency region information corresponding to the live image frame can be determined, as can the underlying skin texture information corresponding to the live image frame. After layer segmentation, textures and details can be modified on the high-frequency layer without compromising the original colors, while brightness and darkness blocks can be modified on the low-frequency layer without destroying details. The high-frequency region information and the underlying skin texture information are then fused to obtain the image frame to be processed.

[0105] The live image frame processing method provided in this embodiment optimizes the defective areas within the processing area and corrects the colors, thereby effectively optimizing the processing area and achieving an aesthetic effect. Furthermore, by determining the high-frequency region information and the underlying skin texture information corresponding to the live image frame, and after layer segmentation, textures and details can be modified on the high-frequency layer without damaging the original colors, and light and dark blocks can be modified on the low-frequency layer without destroying details, further improving the optimization effect of the live image frame.

[0106] Furthermore, based on any of the above embodiments, step 401 includes:

[0107] The region to be processed in the live image frame is blurred by a bilateral filtering algorithm to obtain a blurred image frame.

[0108] Calculate the difference between the blurred image frame and the live image frame.

[0109] If the difference in any region is within a preset range, then the region is determined to be a defective region.

[0110] In this embodiment, when identifying defective regions, the region to be processed in the live image frame can first be blurred. Specifically, a bilateral filtering algorithm can be used to blur the region to be processed in the live image frame to obtain a blurred image frame. The viewport size of the bilateral filtering algorithm is (0.45 * image height, 0.45 * image width). The filter kernel size is 10 pixels.

[0111] Bilateral filtering is a nonlinear filter that can preserve edges and smooth noise reduction. Like other filtering principles, bilateral filtering uses a weighted average method, representing the intensity of a pixel by the weighted average of the brightness values ​​of its surrounding pixels. The weighted average is based on a Gaussian distribution. The weights in bilateral filtering consider not only the Euclidean distance between pixels but also the radiative differences within the pixel domain (the similarity between pixels in the convolution kernel and the center pixel, and color intensity). Both weights are considered simultaneously when calculating the center pixel.

[0112] Furthermore, the difference between the blurred image frame and the live image frame can be calculated. To enable the identification of defective regions, a range of differences can be preset. For any region, if the difference falls within this range, the region is considered a defective region.

[0113] Furthermore, based on any of the above embodiments, the step of correcting the defective area to obtain the first texture map includes:

[0114] The defective area is filled using a preset pixel average value to obtain the first texture image.

[0115] In this embodiment, since the defective area is generally an area with a high pixel value or a low pixel value, the average pixel value can be preset. The average pixel value can be the average pixel value corresponding to the pixels in the live image frame.

[0116] The defective areas are filled using the average pixel value to optimize them and obtain the first texture map.

[0117] The live image frame processing method provided in this embodiment blurs the area to be processed and calculates the difference between the blurred image frame and the live image frame, thereby accurately locating the defective area within the area to be processed. Then, it can fill the defective area based on the preset pixel average value to correct the defective area and ensure the beautification effect of the live image frame.

[0118] Furthermore, based on any of the above embodiments, step 402 includes:

[0119] Obtain a preset color channel from the first texture image as the grayscale image corresponding to the first texture image, and obtain a preset first color lookup table corresponding to the first texture image.

[0120] The target processing area in the grayscale image is determined based on a preset grayscale value range.

[0121] The target processing area is color-corrected according to the first color lookup table to obtain the second texture map.

[0122] In this embodiment, for the first texture image, since the first texture image is an RGB image, a preset color channel in the first texture image can be used as the corresponding grayscale image. For example, the blue channel can be used as the corresponding grayscale image of the first texture image.

[0123] After obtaining the grayscale image corresponding to the first texture image, the light and dark details in the first texture image can be determined based on the grayscale image, thereby identifying the highlight areas, shadow areas, etc.

[0124] In practical applications, different images exhibit different lighting and shadow characteristics. For example, live stream image frames captured in bright locations may suffer from overexposure, thus requiring adjustment of the highlight areas. Conversely, live stream image frames captured in low-light locations may suffer from low clarity due to excessive darkness, necessitating adjustment of the shadow areas.

[0125] Therefore, grayscale value ranges can be preset for different application scenarios. The target processing area in the grayscale image is determined based on this grayscale value range. A preset first color lookup table corresponding to the first texture image is obtained. This first color lookup table can achieve the purpose of brightening skin curves and removing yellow and red hues. This first color lookup table can be the LUT image with the best optimization effect during the adjustment process. This disclosure does not limit the process of determining the first color lookup table. Therefore, color correction operations can be performed on the target processing area according to the first color lookup table to obtain a second texture image. Optionally, color mapping can be performed on the target processing area according to the first color lookup table to achieve color correction.

[0126] The live image frame processing method provided in this embodiment obtains a preset color channel from a first texture image as the corresponding grayscale image, and determines the target processing area in the grayscale image according to a preset grayscale value range. This enables accurate optimization of the target optimization area, making the optimization effect more in line with the user's personalized needs. Furthermore, by performing color correction on the target processing area using a first color lookup table corresponding to the first texture image, the method can achieve skin curve brightening and color removal (removing yellow and red hues), thus improving the optimization effect of the live image frame.

[0127] Furthermore, based on any of the above embodiments, step 403 includes:

[0128] A first joint bilateral filtering operation is performed on the second texture map to obtain a first blurring result.

[0129] The difference between the first blurring result and the live image frame is calculated to obtain the high-frequency region information.

[0130] A second joint bilateral filtering operation is performed on the second texture map to obtain a second blurring result.

[0131] The second blurring result is determined as the underlying skin texture information corresponding to the live image frame.

[0132] The radius of the filter kernel in the second joint bilateral filtering operation is greater than the radius of the filter kernel in the first joint bilateral filtering operation.

[0133] In this embodiment, to further improve the optimization effect, after obtaining the second texture map, a first joint bilateral filtering operation can be performed on the second texture map to obtain a first blurring result. This first joint bilateral filtering operation can be a small-radius joint bilateral filtering process. Further, the difference between the first blurring result and the live image frame can be calculated to obtain high-frequency region information. A second joint bilateral filtering operation is then performed on the second texture map to obtain a second blurring result. The second blurring result is determined as the underlying skin texture information corresponding to the live image frame. This second joint bilateral filtering operation can be a large-radius joint bilateral filtering process. Optionally, the radius of the filter kernel of the second joint bilateral filtering operation is larger than the radius of the filter kernel of the first joint bilateral filtering operation. Preferably, the radius of the filter kernel of the second joint bilateral filtering operation can be twice the radius of the filter kernel of the first joint bilateral filtering operation.

[0134] Furthermore, based on any of the above embodiments, step 404 includes:

[0135] The high-frequency region information and the underlying skin texture information are fused using a linear light mixing mode to obtain the image frame to be processed.

[0136] In this embodiment, a linear light mixing mode can be used to fuse high-frequency region information and underlying skin texture information to obtain the image frame to be processed.

[0137] Furthermore, based on any of the above embodiments, after step 404, the method further includes:

[0138] The image frame to be processed is sharpened to obtain the processed image frame.

[0139] Step 204 includes:

[0140] A second processing operation is performed on the processed image frame to obtain the processed target image frame.

[0141] In this embodiment, after obtaining the image frame to be processed, the image frame to be processed can be sharpened to obtain the final skin smoothing result, thus obtaining the processed image frame to be processed. Further, a second processing operation can be performed based on the processed image frame to obtain the processed target image frame.

[0142] The live image frame processing method provided in this embodiment divides an image into a high-frequency layer and a low-frequency layer. This allows for the modification of textures and details on the high-frequency layer without damaging the original colors, and the modification of light and dark color blocks on the low-frequency layer without destroying details, thereby improving the optimization effect of live image frames.

[0143] Figure 5 This is a flowchart illustrating a live image frame processing method according to another embodiment of the present disclosure. Based on any of the above embodiments, step 201 includes:

[0144] Step 501: In response to the user's trigger operation on the preset image processing control, a preset image processing page is displayed, wherein the image processing page includes at least one image processing item.

[0145] Step 502: In response to the user's selection operation on the image processing page, obtain the image frame processing request.

[0146] In this embodiment, preset image processing controls can be displayed on the display interface in advance. In response to a user's triggering operation on the preset image processing controls, a preset image processing page can be displayed, wherein the image processing page includes at least one image processing item. For example, in practical applications, the image processing item may include at least skin smoothing and whitening items.

[0147] After the image processing page is displayed, users can select the image processing items that need optimization based on their actual needs. An image frame processing request is then generated based on at least one selected image processing item. Subsequently, targeted image processing operations can be performed on the live stream image frames according to this request.

[0148] For example, if a user selects the skin smoothing and whitening options, the live image frame can be processed locally for skin smoothing and globally for whitening based on the image frame processing request.

[0149] Furthermore, based on any of the above embodiments, step 501 includes:

[0150] Display at least one preset image processing selection control, wherein the image processing selection control includes a first image processing control and a second image processing control.

[0151] In response to the user's triggering operation on the first image processing control, a preset image processing page is displayed.

[0152] In response to the user's triggering operation on the second image processing control, the live image frame corresponding to the virtual reality live content is displayed.

[0153] In this embodiment, at least one image processing selection control can be pre-displayed on the display interface, wherein the image processing selection control includes a first image processing control and a second image processing control. For example, in a practical application, a selection box for whether to enable beauty mode can be displayed on the display interface, and the first image processing control and the second image processing control can be displayed in the selection box, wherein the first image processing control can be "yes" and the second image processing control can be "no".

[0154] In response to a user's triggering of the first image processing control, indicating that the user needs to perform image processing operations, a preset image processing page can be displayed, allowing the user to select the image processing item to be optimized according to their actual needs. Conversely, in response to a user's triggering of the second image processing control, indicating that the user does not currently need to perform image processing operations, the live image frames corresponding to the virtual reality live content can be directly sent to a display device for display, where the display device can be a virtual reality device.

[0155] The live image frame processing method provided in this embodiment displays a preset image processing page in response to the user's trigger operation on the preset image processing control. This allows the user to determine the processing items for personalized live image frames based on the image processing page, thereby making the display effect of the processed target image frame more in line with the user's actual needs and improving the user experience.

[0156] Furthermore, based on any of the above embodiments, step 502 includes:

[0157] In response to the user's selection of one or more image processing items on the image processing page, the adjustment parameters corresponding to each image processing item are determined respectively.

[0158] The corresponding adjustment parameters for each image processing item are obtained based on the adjustment parameters for each image processing item.

[0159] In this embodiment, for each image processing item, the user can also select the corresponding adjustment parameters. Optionally, a higher adjustment parameter indicates a greater intensity of image processing, and vice versa.

[0160] Therefore, after displaying the image processing page, in response to the user's selection of one or more image processing items on the page, the adjustment parameters corresponding to each image processing item can be determined. The parameters corresponding to each image processing item are then used to obtain the corresponding settings for each image processing item.

[0161] Optionally, the adjustment parameters can be determined through a preset adjustment control or a preset parameter input box; this disclosure does not impose any restrictions on this.

[0162] By determining the adjustment parameters corresponding to each image processing item, image processing operations on live image frames can be performed more accurately.

[0163] Furthermore, based on any of the above embodiments, each image processing item corresponds to at least one image processing sub-item and adjustment controls corresponding to each image processing sub-item; the step of determining the adjustment parameters corresponding to each image processing item in response to the user's selection operation of one or more image processing items on the image processing page includes:

[0164] In response to a user's trigger operation on the adjustment controls corresponding to one or more image processing sub-items, the adjustment parameters corresponding to one or more image processing sub-items are obtained, and the adjustment parameters corresponding to the one or more image processing sub-items are determined as the adjustment parameters corresponding to each image processing item.

[0165] In this embodiment, each image processing project corresponds to at least one image processing sub-project and an adjustment control corresponding to each image processing sub-project. Users can determine the adjustment parameters by dragging and dropping the adjustment controls.

[0166] Specifically, in response to a user's triggering operation on the adjustment controls corresponding to one or more image processing sub-items, adjustment parameters corresponding to one or more image processing sub-items can be obtained, and the adjustment parameters corresponding to one or more image processing sub-items can be determined as the adjustment parameters corresponding to each image processing item.

[0167] Figure 6 This is a schematic diagram of the interface interaction provided in the embodiments of this disclosure, such as... Figure 6As shown, a preset first image processing control 61 and a second image processing control 62 can be displayed on the display interface. In response to the user's trigger operation on the first image processing control 61, an image processing page 63 can be displayed. The image processing page 63 includes multiple image processing items 64, each image processing item 64 includes at least one image processing sub-item 65, and each image processing sub-item 65 corresponds to an adjustment control 66.

[0168] The live image frame processing method provided in this embodiment sets adjustment controls at the display position associated with each image processing sub-item, so that users can quickly adjust and determine the adjustment parameters based on the adjustment controls, thereby making the current image processing effect more in line with the user's personalized needs.

[0169] Furthermore, based on any of the above embodiments, the image processing item includes at least a first processing item and a second processing item; after step 202, it further includes:

[0170] If the image frame processing request includes the first processing item and the second processing item, then the region to be processed in the live image frame is determined, and the first processing operation is performed on the region to be processed to obtain the image frame to be processed; and the second global processing operation is performed on the image frame to be processed to obtain the processed target image frame.

[0171] If the image frame processing request includes the first processing item, then the region to be processed in the live image frame is determined, and the first processing operation is performed on the region to be processed to obtain the target image frame.

[0172] If the image frame processing request includes the second processing item, then the live image frame is subjected to a global second processing operation to obtain the processed target image frame.

[0173] In this embodiment, the image processing project includes at least a first processing project and a second processing project. For example, in a practical application, the first processing project could be a skin smoothing project, and the second processing project could be a skin whitening project.

[0174] After obtaining the image frame processing request and the live image frame, if the image frame processing request includes a first processing item and a second processing item, the region to be processed in the live image frame can be determined, and the first processing operation can be performed on the region to be processed to obtain the image frame to be processed; then, the second global processing operation can be performed on the image frame to be processed to obtain the processed target image frame. Continuing with the previous example, if the image frame processing request includes skin smoothing and whitening items, then local skin smoothing and global whitening operations can be performed on the live image frame to obtain the target image frame.

[0175] If the image frame processing request includes a first processing item, the region to be processed in the live image frame is determined, and the first processing operation is performed on the region to be processed to obtain the target image frame. Continuing with the previous example, if the image frame processing request includes a skin smoothing item, a local skin smoothing operation can be performed on the live image frame to obtain the target image frame.

[0176] If the image frame processing request includes a second processing item, then a global second processing operation is performed on the live image frame to obtain the processed target image frame. Continuing with the previous example, if the image frame processing request includes a whitening item, then a global whitening operation can be performed on the live image frame to obtain the target image frame.

[0177] The live image frame processing method provided in this embodiment performs corresponding live image frame processing operations based on the user's selection of image processing items and the generated live image frame request. This enables the display effect of virtual reality live streaming to better meet the user's personalized needs and improve the live streaming effect.

[0178] Figure 7 This is a schematic diagram of the structure of the live image frame processing apparatus provided in the embodiments of this disclosure, as shown below. Figure 7 As shown, the device includes: an acquisition module 71, an image frame acquisition module 72, a determination module 73, a processing module 74, and a sending module 75. The acquisition module 71 is used to acquire an image frame processing request. The image frame acquisition module 72 is used to acquire a live image frame corresponding to the virtual reality live content according to the image frame processing request. The determination module 73 is used to determine the region to be processed in the live image frame and perform a first processing operation on the region to be processed to obtain a image frame to be processed. The processing module 74 is used to perform a global second processing operation on the image frame to be processed to obtain a processed target image frame. The sending module 75 is used to display the target image frame.

[0179] Furthermore, based on any of the above embodiments, the image frame acquisition module is used to: acquire two image frames to be stitched together from at least two cameras; and perform a stitching operation on the two image frames to be stitched together to obtain the live image frame.

[0180] Further, based on any of the above embodiments, the determining module is configured to: perform a recognition operation on the live image frame to obtain key points corresponding to each target object in the live image frame; determine the target region where each target object is located based on the key points and a preset target mask; determine the non-processed region within the target region; adjust the pixel values ​​corresponding to the non-processed region to a preset value to obtain the region to be processed.

[0181] Further, based on any of the above embodiments, the determining module is configured to: convert the live image frame from an RGB image to an HSV image, use the hue and brightness of the HSV image as constraint information, and determine a first unprocessed region in the target area. By adjusting the layer corresponding to the target mask, a second unprocessed region in the target area is determined. The pixel values ​​corresponding to the first unprocessed region and the second unprocessed region in the target area are set to zero to obtain the region to be processed.

[0182] Furthermore, based on any of the above embodiments, the determining module is used to: determine the facial region in the live image frame; perform a first processing operation on at least one of the blemish region, shadow region, and highlight region in the facial region to obtain the image frame to be processed.

[0183] Further, based on any of the above embodiments, the determining module is configured to: identify defective regions within the area to be processed; perform a correction operation on the defective regions to obtain a first texture map; perform a color correction operation on the first texture map to obtain a second texture map; determine high-frequency region information corresponding to the live image frame based on the second texture map and the live image frame; and determine underlying skin texture information corresponding to the live image frame based on the second texture map; and perform a fusion operation on the high-frequency region information and the underlying skin texture information to obtain the image frame to be processed.

[0184] Furthermore, based on any of the above embodiments, the determining module is configured to: blur the region to be processed in the live image frame using a bilateral filtering algorithm to obtain a blurred image frame; calculate the difference between the blurred image frame and the live image frame; and determine that the region is a defective region if the difference between any region is within a preset difference range.

[0185] Furthermore, based on any of the above embodiments, the determining module is used to: fill the defective area with a preset pixel average value to obtain the first texture map.

[0186] Further, based on any of the above embodiments, the determining module is configured to: obtain a preset color channel in the first texture image as a grayscale image corresponding to the first texture image, and obtain a preset first color lookup table corresponding to the first texture image. Determine the target processing area in the grayscale image according to a preset grayscale value range. Perform color correction operation on the target processing area according to the first color lookup table to obtain the second texture image.

[0187] Further, based on any of the above embodiments, the determining module is configured to: perform a first joint bilateral filtering operation on the second texture map to obtain a first blurring result; calculate the difference between the first blurring result and the live image frame to obtain the high-frequency region information; perform a second joint bilateral filtering operation on the second texture map to obtain a second blurring result; and determine the second blurring result as the underlying skin texture information corresponding to the live image frame. Wherein, the radius of the filter kernel of the second joint bilateral filtering operation is greater than the radius of the filter kernel of the first joint bilateral filtering operation.

[0188] Furthermore, based on any of the above embodiments, the determining module is used to: perform a fusion operation on the high-frequency region information and the underlying skin texture information through a linear light mixing mode to obtain the image frame to be processed.

[0189] Furthermore, based on any of the above embodiments, the apparatus further includes: a sharpening module, used to sharpen the image frame to be processed to obtain a processed image frame to be processed. The processing module is used to: perform a global second processing operation on the processed image frame to be processed to obtain a processed target image frame.

[0190] Furthermore, based on any of the above embodiments, the processing module is used to: perform a global second processing operation on the image frame to be processed according to the second color lookup table corresponding to the image frame to be processed, so as to obtain the processed target image frame.

[0191] Furthermore, based on any of the above embodiments, the acquisition module is configured to: display a preset image processing page in response to a user's trigger operation on a preset image processing control, wherein the image processing page includes at least one image processing item; and obtain the image frame processing request in response to a user's selection operation on the image processing page.

[0192] Furthermore, based on any of the above embodiments, the acquisition module is configured to: display at least one preset image processing selection control, wherein the image processing selection control includes a first image processing control and a second image processing control; display a preset image processing page in response to the user's trigger operation on the first image processing control; and display live image frames corresponding to the virtual reality live streaming content in response to the user's trigger operation on the second image processing control.

[0193] Furthermore, based on any of the above embodiments, the acquisition module is configured to: in response to a user's selection operation of one or more image processing items in the image processing page, determine the adjustment parameters corresponding to each image processing item; and obtain the corresponding parameters of each image processing item based on the adjustment parameters of each image processing item.

[0194] Furthermore, based on any of the above embodiments, each image processing item corresponds to at least one image processing sub-item and an adjustment control corresponding to each image processing sub-item; the acquisition module is used to: in response to the user's trigger operation on the adjustment control corresponding to one or more image processing sub-items, obtain the adjustment parameters corresponding to one or more image processing sub-items, and determine the adjustment parameters corresponding to the one or more image processing sub-items as the adjustment parameters corresponding to each image processing item.

[0195] Further, based on any of the above embodiments, the image processing item includes at least a first processing item and a second processing item; the apparatus further includes: a processing module, configured to, if the image frame processing request includes the first processing item and the second processing item, determine the region to be processed in the live image frame, perform a first processing operation on the region to be processed to obtain a image frame to be processed; and perform a global second processing operation on the image frame to be processed to obtain a processed target image frame; a processing module, configured to, if the image frame processing request includes the first processing item, determine the region to be processed in the live image frame, perform a first processing operation on the region to be processed to obtain the target image frame; and a processing module, configured to, if the image frame processing request includes the second processing item, perform a global second processing operation on the live image frame to obtain a processed target image frame.

[0196] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0197] To implement the above embodiments, this disclosure also provides an electronic device, including a processor and a memory.

[0198] The memory stores computer-executed instructions.

[0199] The processor executes computer execution instructions stored in the memory, causing the processor to perform the live image frame processing method as described in any of the above embodiments.

[0200] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 8As shown, the electronic device 800 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0201] like Figure 8 As shown, the electronic device 800 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. The RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0202] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0203] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of embodiments of this disclosure.

[0204] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0205] This disclosure also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the live image frame processing method as described in any of the above embodiments.

[0206] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the information display method as described in any of the above embodiments.

[0207] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0208] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0209] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0210] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0211] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0212] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0213] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0214] In a first aspect, according to one or more embodiments of this disclosure, a live image frame processing method is provided, comprising:

[0215] Get image frame processing request;

[0216] According to the image frame processing request, obtain the live image frame corresponding to the virtual reality live content;

[0217] The region to be processed in the live image frame is determined, and a first processing operation is performed on the region to be processed to obtain the image frame to be processed.

[0218] A global second processing operation is performed on the image frame to be processed to obtain the processed target image frame;

[0219] Display the target image frame.

[0220] According to one or more embodiments of this disclosure, obtaining the live image frame corresponding to the virtual reality live content includes:

[0221] Acquire two image frames to be stitched together from at least two cameras;

[0222] The two image frames to be stitched together are stitched together to obtain the live image frame.

[0223] According to one or more embodiments of this disclosure, determining the region to be processed in the live image frame includes:

[0224] The live image frame is subjected to a recognition operation to obtain the key points corresponding to each target object in the live image frame;

[0225] The target area of ​​each target object is determined based on the key points and the preset target mask.

[0226] Identify the non-processed areas within the target region, adjust the pixel values ​​corresponding to the non-processed areas to preset values, and obtain the region to be processed.

[0227] According to one or more embodiments of this disclosure, determining the non-processed region within the target region and adjusting the pixel value corresponding to the non-processed region to a preset value includes:

[0228] The live image frame is converted from an RGB image to an HSV image, and the hue and brightness of the HSV image are used as constraint information to determine the first unprocessed region in the target area.

[0229] By adjusting the layer corresponding to the target mask, a second unprocessed area in the target region is determined;

[0230] The pixel values ​​corresponding to the first unprocessed region and the second unprocessed region in the target region are set to zero to obtain the region to be processed.

[0231] According to one or more embodiments of this disclosure, determining the region to be processed in the live image frame and performing a first processing operation on the region to be processed to obtain the image frame to be processed includes:

[0232] Determine the facial region in the live image frame;

[0233] A first processing operation is performed on at least one of the blemish area, shadow area, and highlight area in the facial region to obtain the image frame to be processed.

[0234] According to one or more embodiments of this disclosure, performing a first processing operation on the region to be processed to obtain a frame of image to be processed includes:

[0235] Identify defective areas within the area to be processed, perform correction operations on the defective areas, and obtain a first texture map;

[0236] Perform color correction on the first texture image to obtain the second texture image;

[0237] The high-frequency region information corresponding to the live image frame is determined based on the second texture map and the live image frame, and the underlying skin texture information corresponding to the live image frame is determined based on the second texture map.

[0238] The high-frequency region information and the underlying skin texture information are fused to obtain the image frame to be processed.

[0239] According to one or more embodiments of this disclosure, identifying defective areas within the area to be processed includes:

[0240] The region to be processed in the live image frame is blurred by a bilateral filtering algorithm to obtain a blurred image frame.

[0241] Calculate the difference between the blurred image frame and the live image frame;

[0242] If the difference in any region is within a preset range, then the region is determined to be a defective region.

[0243] According to one or more embodiments of this disclosure, the step of correcting the defective region to obtain a first texture map includes:

[0244] The defective area is filled using a preset pixel average value to obtain the first texture image.

[0245] According to one or more embodiments of this disclosure, performing a color correction operation on the first texture image to obtain a second texture image includes:

[0246] Obtain a preset color channel in the first texture image as the grayscale image corresponding to the first texture image, and obtain a preset first color lookup table corresponding to the first texture image;

[0247] The target processing area in the grayscale image is determined based on a preset grayscale value range;

[0248] The target processing area is color-corrected according to the first color lookup table to obtain the second texture map.

[0249] According to one or more embodiments of this disclosure, determining the high-frequency region information corresponding to the live image frame based on the second texture map and the live image frame, and determining the underlying skin texture information corresponding to the live image frame based on the second texture map, includes:

[0250] A first joint bilateral filtering operation is performed on the second texture map to obtain a first blurring result;

[0251] Calculate the difference between the first blurring result and the live image frame to obtain the high-frequency region information;

[0252] A second joint bilateral filtering operation is performed on the second texture map to obtain a second blurring result;

[0253] The second blurring result is determined as the underlying skin texture information corresponding to the live image frame;

[0254] The radius of the filter kernel in the second joint bilateral filtering operation is greater than the radius of the filter kernel in the first joint bilateral filtering operation.

[0255] According to one or more embodiments of this disclosure, the step of fusing the high-frequency region information and the underlying skin texture information to obtain the image frame to be processed includes:

[0256] The high-frequency region information and the underlying skin texture information are fused using a linear light mixing mode to obtain the image frame to be processed.

[0257] According to one or more embodiments of this disclosure, after performing the fusion operation on the high-frequency region information and the underlying skin texture information to obtain the image frame to be processed, the method further includes:

[0258] The image frame to be processed is sharpened to obtain the processed image frame;

[0259] The second global processing operation on the image frame to be processed to obtain the processed target image frame includes:

[0260] A global second processing operation is performed on the processed image frame to obtain the processed target image frame.

[0261] According to one or more embodiments of this disclosure, performing a global second processing operation on the image frame to be processed to obtain a processed target image frame includes:

[0262] The second global processing operation is performed on the image frame to be processed according to the second color lookup table corresponding to the image frame to be processed, so as to obtain the processed target image frame.

[0263] According to one or more embodiments of this disclosure, the image frame processing request includes:

[0264] In response to a user's triggering operation on a preset image processing control, a preset image processing page is displayed, wherein the image processing page includes at least one image processing item;

[0265] In response to the user's selection on the image processing page, the image frame processing request is obtained.

[0266] According to one or more embodiments of this disclosure, displaying a preset image processing page in response to a user's triggering operation on a preset image processing control includes:

[0267] Display at least one preset image processing selection control, wherein the image processing selection control includes a first image processing control and a second image processing control;

[0268] In response to the user's trigger operation on the first image processing control, a preset image processing page is displayed;

[0269] In response to the user's triggering operation on the second image processing control, the live image frame corresponding to the virtual reality live content is displayed.

[0270] According to one or more embodiments of this disclosure, obtaining the image frame processing request in response to a user's selection operation on the image processing page includes:

[0271] In response to the user's selection of one or more image processing items on the image processing page, the adjustment parameters corresponding to each image processing item are determined respectively;

[0272] The corresponding adjustment parameters for each image processing item are obtained based on the adjustment parameters for each image processing item.

[0273] According to one or more embodiments of this disclosure, each image processing item corresponds to at least one image processing sub-item and adjustment controls corresponding to each image processing sub-item; the step of determining adjustment parameters corresponding to each image processing item in response to a user's selection operation on one or more image processing items on the image processing page includes:

[0274] In response to a user's trigger operation on the adjustment controls corresponding to one or more image processing sub-items, the adjustment parameters corresponding to one or more image processing sub-items are obtained, and the adjustment parameters corresponding to the one or more image processing sub-items are determined as the adjustment parameters corresponding to each image processing item.

[0275] According to one or more embodiments of this disclosure, the image processing project includes at least a first processing project and a second processing project; after obtaining the live image frame corresponding to the virtual reality live content according to the image frame processing request, the process further includes:

[0276] If the image frame processing request includes the first processing item and the second processing item, then the region to be processed in the live image frame is determined, and the first processing operation is performed on the region to be processed to obtain the image frame to be processed; and the second global processing operation is performed on the image frame to be processed to obtain the processed target image frame.

[0277] If the image frame processing request includes the first processing item, then the region to be processed in the live image frame is determined, and the first processing operation is performed on the region to be processed to obtain the target image frame.

[0278] If the image frame processing request includes the second processing item, then the live image frame is subjected to a global second processing operation to obtain the processed target image frame.

[0279] Secondly, according to one or more embodiments of this disclosure, a live image frame processing system is provided, comprising: a terminal device, a binocular image acquisition device, and a virtual reality device; wherein,

[0280] The binocular image acquisition device is used to acquire live image frames corresponding to virtual reality live content.

[0281] The terminal device is used to acquire an image frame processing request, acquire a live image frame corresponding to the virtual reality live content according to the image frame processing request; determine the area to be processed in the live image frame, and perform a first processing operation on the area to be processed to obtain an image frame to be processed; perform a global second processing operation on the image frame to be processed to obtain a processed target image frame; and send the target image frame to the virtual reality device.

[0282] The virtual reality device is used to display the target image frame.

[0283] Thirdly, according to one or more embodiments of this disclosure, a live image frame processing apparatus is provided, comprising:

[0284] The acquisition module is used to acquire image frame processing requests;

[0285] The image frame acquisition module is used to acquire the live image frame corresponding to the virtual reality live content according to the image frame processing request.

[0286] The determining module is used to determine the region to be processed in the live image frame and perform a first processing operation on the region to be processed to obtain the image frame to be processed.

[0287] The processing module is used to perform a global second processing operation on the image frame to be processed to obtain the processed target image frame;

[0288] The sending module is used to display the target image frame.

[0289] According to one or more embodiments of this disclosure, the image frame acquisition module is used for:

[0290] Acquire two image frames to be stitched from at least two cameras;

[0291] The two image frames to be stitched together are stitched together to obtain the live image frame.

[0292] According to one or more embodiments of this disclosure, the determining module is used to:

[0293] The live image frame is subjected to a recognition operation to obtain the key points corresponding to each target object in the live image frame;

[0294] The target area of ​​each target object is determined based on the key points and the preset target mask.

[0295] Identify the non-processed areas within the target region, adjust the pixel values ​​corresponding to the non-processed areas to preset values, and obtain the region to be processed.

[0296] According to one or more embodiments of this disclosure, the determining module is used to:

[0297] The live image frame is converted from an RGB image to an HSV image, and the hue and brightness of the HSV image are used as constraint information to determine the first unprocessed region in the target area.

[0298] By adjusting the layer corresponding to the target mask, a second unprocessed area in the target region is determined;

[0299] The pixel values ​​corresponding to the first unprocessed region and the second unprocessed region in the target region are set to zero to obtain the region to be processed.

[0300] According to one or more embodiments of this disclosure, the determining module is used to:

[0301] Identify the facial region in the live image frame;

[0302] A first processing operation is performed on at least one of the blemish area, shadow area, and highlight area in the facial region to obtain the image frame to be processed.

[0303] According to one or more embodiments of this disclosure, the determining module is used to:

[0304] Identify defective areas within the area to be processed, perform correction operations on the defective areas, and obtain a first texture map;

[0305] Perform color correction on the first texture image to obtain the second texture image;

[0306] The high-frequency region information corresponding to the live image frame is determined based on the second texture map and the live image frame, and the underlying skin texture information corresponding to the live image frame is determined based on the second texture map.

[0307] The high-frequency region information and the underlying skin texture information are fused to obtain the image frame to be processed.

[0308] According to one or more embodiments of this disclosure, the determining module is used to:

[0309] The region to be processed in the live image frame is blurred by a bilateral filtering algorithm to obtain a blurred image frame.

[0310] Calculate the difference between the blurred image frame and the live image frame;

[0311] If the difference in any region is within a preset range, then the region is determined to be a defective region.

[0312] According to one or more embodiments of this disclosure, the determining module is used to:

[0313] The defective area is filled using a preset pixel average value to obtain the first texture image.

[0314] According to one or more embodiments of this disclosure, the determining module is used to:

[0315] Obtain a preset color channel in the first texture image as the grayscale image corresponding to the first texture image, and obtain a preset first color lookup table corresponding to the first texture image;

[0316] The target processing area in the grayscale image is determined based on a preset grayscale value range;

[0317] The target processing area is color-corrected according to the first color lookup table to obtain the second texture map.

[0318] According to one or more embodiments of this disclosure, the determining module is used to:

[0319] A first joint bilateral filtering operation is performed on the second texture map to obtain a first blurring result;

[0320] Calculate the difference between the first blurring result and the live image frame to obtain the high-frequency region information;

[0321] A second joint bilateral filtering operation is performed on the second texture map to obtain a second blurring result;

[0322] The second blurring result is determined as the underlying skin texture information corresponding to the live image frame;

[0323] The radius of the filter kernel in the second joint bilateral filtering operation is greater than the radius of the filter kernel in the first joint bilateral filtering operation.

[0324] According to one or more embodiments of this disclosure, the determining module is used to:

[0325] The high-frequency region information and the underlying skin texture information are fused using a linear light mixing mode to obtain the image frame to be processed.

[0326] According to one or more embodiments of this disclosure, the apparatus further includes:

[0327] The sharpening module is used to sharpen the image frame to be processed, and obtain the processed image frame.

[0328] The processing module is used for:

[0329] A global second processing operation is performed on the processed image frame to obtain the processed target image frame.

[0330] According to one or more embodiments of this disclosure, the processing module is configured to: perform a global second processing operation on the image frame to be processed according to a second color lookup table corresponding to the image frame to be processed, and obtain a processed target image frame.

[0331] According to one or more embodiments of this disclosure, the acquisition module is configured to:

[0332] In response to a user's triggering operation on a preset image processing control, a preset image processing page is displayed, wherein the image processing page includes at least one image processing item;

[0333] In response to the user's selection on the image processing page, the image frame processing request is obtained.

[0334] According to one or more embodiments of this disclosure, the acquisition module is configured to:

[0335] Display at least one preset image processing selection control, wherein the image processing selection control includes a first image processing control and a second image processing control;

[0336] In response to the user's trigger operation on the first image processing control, a preset image processing page is displayed;

[0337] In response to the user's triggering operation on the second image processing control, the live image frame corresponding to the virtual reality live content is displayed.

[0338] According to one or more embodiments of this disclosure, the acquisition module is configured to:

[0339] In response to the user's selection of one or more image processing items on the image processing page, the adjustment parameters corresponding to each image processing item are determined respectively;

[0340] The corresponding adjustment parameters for each image processing item are obtained based on the adjustment parameters for each image processing item.

[0341] According to one or more embodiments of this disclosure, each image processing item corresponds to at least one image processing sub-item and adjustment controls corresponding to each image processing sub-item; the acquisition module is used for:

[0342] In response to a user's trigger operation on the adjustment controls corresponding to one or more image processing sub-items, the adjustment parameters corresponding to one or more image processing sub-items are obtained, and the adjustment parameters corresponding to the one or more image processing sub-items are determined as the adjustment parameters corresponding to each image processing item.

[0343] According to one or more embodiments of this disclosure, the image processing item includes at least a first processing item and a second processing item; the apparatus further includes:

[0344] The processing module is configured to, if the image frame processing request includes the first processing item and the second processing item, determine the region to be processed in the live image frame, perform a first processing operation on the region to be processed to obtain the image frame to be processed, and perform a global second processing operation on the image frame to be processed to obtain the processed target image frame.

[0345] The processing module is configured to determine the region to be processed in the live image frame if the image frame processing request includes the first processing item, and perform a first processing operation on the region to be processed to obtain the target image frame.

[0346] The processing module is configured to perform a global second processing operation on the live image frame if the image frame processing request includes the second processing item, so as to obtain the processed target image frame.

[0347] Fourthly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0348] The memory stores computer-executed instructions;

[0349] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the live image frame processing method as described in the first aspect and various possible designs of the first aspect.

[0350] Fifthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the live image frame processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0351] Sixthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the live image frame processing method as described in the first aspect and various possible designs of the first aspect.

[0352] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0353] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0354] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for processing live image frames, characterized in that, include: Get image frame processing request; According to the image frame processing request, obtain the live image frame corresponding to the virtual reality live content; The target area of ​​each target object is determined based on the key points corresponding to each target object in the live image frame and the preset target mask. The target mask includes multiple layers, and different layers record the eyebrow, nostril, mouth and eye areas respectively. The live image frame is converted from an RGB image to an HSV image, and the hue and brightness of the HSV image are used as constraint information to determine the first unprocessed region in the target area. By adjusting the layer corresponding to the target mask, a second unprocessed area in the target region is determined; Set the pixel values ​​corresponding to the first unprocessed region and the second unprocessed region to zero to obtain the region to be processed, which is the facial skin region. A first processing operation is performed on the region to be processed to obtain the image frame to be processed; A global second processing operation is performed on the image frame to be processed to obtain the processed target image frame; Display the target image frame.

2. The method according to claim 1, characterized in that, The process of acquiring the live image frames corresponding to the virtual reality live content includes: Acquire two image frames to be stitched from at least two cameras; The two image frames to be stitched together are stitched together to obtain the live image frame.

3. The method according to claim 1, characterized in that, Also includes: The live image frame is subjected to a recognition operation to obtain the key points corresponding to each target object in the live image frame.

4. The method according to any one of claims 1-3, characterized in that, The first processing operation on the region to be processed to obtain the image frame to be processed includes: Identify the facial region in the live image frame; A first processing operation is performed on at least one of the blemish area, shadow area, and highlight area in the facial region to obtain the image frame to be processed.

5. The method according to any one of claims 1-3, characterized in that, The first processing operation on the region to be processed to obtain the image frame to be processed includes: Identify defective areas within the area to be processed, perform correction operations on the defective areas, and obtain a first texture map; Perform color correction on the first texture image to obtain the second texture image; Based on the second texture map and the live image frame, determine the high-frequency region information corresponding to the live image frame, and based on the second texture map, determine the underlying skin texture information corresponding to the live image frame; The high-frequency region information and the underlying skin texture information are fused to obtain the image frame to be processed.

6. The method according to claim 5, characterized in that, The identification of defective areas within the area to be processed includes: The region to be processed in the live image frame is blurred by a bilateral filtering algorithm to obtain a blurred image frame. Calculate the difference between the blurred image frame and the live image frame; If the difference in any region is within a preset range, then the region is determined to be a defective region.

7. The method according to claim 5, characterized in that, The step of correcting the defective area to obtain the first texture map includes: The defective area is filled using a preset pixel average value to obtain the first texture image.

8. The method according to claim 5, characterized in that, The step of performing a color correction operation on the first texture image to obtain a second texture image includes: Obtain a preset color channel in the first texture image as the grayscale image corresponding to the first texture image, and obtain a preset first color lookup table corresponding to the first texture image; The target processing area in the grayscale image is determined based on a preset grayscale value range; The target processing area is color-corrected according to the first color lookup table to obtain the second texture map.

9. The method according to claim 5, characterized in that, The step of determining the high-frequency region information corresponding to the live image frame based on the second texture map and the live image frame, and determining the underlying skin texture information corresponding to the live image frame based on the second texture map, includes: A first joint bilateral filtering operation is performed on the second texture map to obtain a first blurring result; Calculate the difference between the first blurring result and the live image frame to obtain the high-frequency region information; A second joint bilateral filtering operation is performed on the second texture map to obtain a second blurring result; The second blurring result is determined as the underlying skin texture information corresponding to the live image frame; The radius of the filter kernel in the second joint bilateral filtering operation is greater than the radius of the filter kernel in the first joint bilateral filtering operation.

10. The method according to any one of claims 1-3, characterized in that, The step of performing a global second processing operation on the image frame to be processed to obtain the processed target image frame includes: The second global processing operation is performed on the image frame to be processed according to the second color lookup table corresponding to the image frame to be processed, so as to obtain the processed target image frame.

11. The method according to any one of claims 1-3, characterized in that, The image frame acquisition processing request includes: In response to a user's triggering operation on a preset image processing control, a preset image processing page is displayed, wherein the image processing page includes at least one image processing item; In response to the user's selection on the image processing page, the image frame processing request is obtained.

12. The method according to claim 11, characterized in that, The step of displaying a preset image processing page in response to a user's triggering operation on a preset image processing control includes: Display at least one preset image processing selection control, wherein the image processing selection control includes a first image processing control and a second image processing control; In response to the user's trigger operation on the first image processing control, a preset image processing page is displayed; In response to the user's triggering operation on the second image processing control, the live image frame corresponding to the virtual reality live content is displayed.

13. The method according to claim 11, characterized in that, The step of obtaining the image frame processing request in response to a user's selection operation on the image processing page includes: In response to the user's selection of one or more image processing items on the image processing page, the adjustment parameters corresponding to each image processing item are determined; and the image frame processing request corresponding to each image processing item is obtained based on the adjustment parameters corresponding to each image processing item.

14. The method according to claim 13, characterized in that, Each image processing item corresponds to at least one image processing sub-item and adjustment controls for each sub-item; the step of determining the adjustment parameters corresponding to each image processing item in response to the user's selection of one or more image processing items on the image processing page includes: In response to a user's trigger operation on the adjustment controls corresponding to one or more image processing sub-items, the adjustment parameters corresponding to one or more image processing sub-items are obtained, and the adjustment parameters corresponding to the one or more image processing sub-items are determined as the adjustment parameters corresponding to each image processing item.

15. A live streaming image frame processing system, characterized in that, include: Terminal equipment, binocular image acquisition devices, and virtual reality equipment; among them, The binocular image acquisition device is used to acquire live image frames corresponding to virtual reality live content. The terminal device is used to acquire image frame processing requests, acquire live image frames corresponding to virtual reality live content according to the image frame processing requests; determine the target region of each target object based on key points corresponding to each target object in the live image frame and a preset target mask, wherein the target mask includes multiple layers, and different layers record the eyebrow, nostril, mouth and eye regions respectively; convert the live image frame from RGB image to HSV image, and use the hue and brightness of the HSV image as constraint information to determine the first non-processed region in the target region; determine the second non-processed region in the target region by adjusting the layers corresponding to the target mask; set the pixel values ​​corresponding to the first non-processed region and the second non-processed region to zero to obtain the region to be processed, wherein the region to be processed is the facial skin region; and perform a first processing operation on the region to be processed to obtain a image frame to be processed; perform a global second processing operation on the image frame to be processed to obtain a processed target image frame; and send the target image frame to the virtual reality device. The virtual reality device is used to display the target image frame.

16. A live image frame processing apparatus, characterized in that, include: The acquisition module is used to acquire image frame processing requests; The image frame acquisition module is used to acquire the live image frame corresponding to the virtual reality live content according to the image frame processing request. The determination module is used to determine the target area of ​​each target object based on the key points corresponding to each target object in the live image frame and the preset target mask. The target mask includes multiple layers, and different layers record the eyebrow, nostril, mouth and eye areas respectively. The live image frame is converted from an RGB image to an HSV image, and the hue and brightness of the HSV image are used as constraint information to determine the first unprocessed region in the target area. By adjusting the layer corresponding to the target mask, a second unprocessed area in the target region is determined; Set the pixel values ​​corresponding to the first unprocessed region and the second unprocessed region to zero to obtain the region to be processed, which is the facial skin region. A first processing operation is performed on the region to be processed to obtain the image frame to be processed; The processing module is used to perform a global second processing operation on the image frame to be processed to obtain the processed target image frame; The sending module is used to display the target image frame.

17. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the live image frame processing method as described in any one of claims 1 to 14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the live image frame processing method as described in any one of claims 1 to 14.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method for processing live image frames as described in any one of claims 1 to 14.

Images