A picture quality enhancement method and device, electronic equipment and storage medium

Abstract

The application provides a picture quality enhancement method and device, electronic equipment and a storage medium. The method comprises the following steps: decoding a video stream sent by a service end to obtain a plurality of continuous picture frames to be processed; identifying a first target region of a current picture frame to be processed which needs to be subjected to picture quality enhancement; and performing picture quality enhancement on the first target region of the current picture frame to be processed to obtain a picture quality enhanced picture of the current frame. In the scheme, the first target region of the current picture frame to be processed which needs to be subjected to picture quality enhancement is identified, and then picture quality enhancement is performed on the first target region of the current picture frame to be processed, so that the picture quality enhanced picture of the current frame is obtained, and picture quality enhancement does not need to be performed on all regions of the picture, thereby reducing the calculation time consumption.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image quality enhancement method, apparatus, electronic device, and storage medium. Background Technology

[0002] To improve the quality of videos played by the player, users can use AI super-resolution computing to upscale standard definition (SD) and low definition (LHD) videos to UHD. However, AI super-resolution computing is computationally intensive and requires a considerable amount of time to perform image enhancement. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide an image quality enhancement method, apparatus, electronic device, and storage medium to reduce the computation time required for image quality enhancement.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0005] The first aspect of this invention discloses an image quality enhancement method, the method comprising:

[0006] Decode the video stream sent by the server to obtain a series of multiple frames of images to be processed;

[0007] Identify the first target region in the current frame of the image to be processed that requires image quality enhancement;

[0008] Image quality enhancement is performed on the first target region of the image to be processed in the current frame to obtain the image quality enhanced image of the current frame.

[0009] Preferably, image quality enhancement is performed on the first target region of the image to be processed in the current frame to obtain an enhanced image of the current frame, including:

[0010] Crops the image corresponding to the first target region from the image to be processed in the current frame;

[0011] Image quality enhancement is performed on the image corresponding to the first target region;

[0012] The image to be processed in the current frame is merged with the image corresponding to the first target region after image quality enhancement to obtain the image quality enhanced image of the current frame.

[0013] Preferably, identifying the first target region of the image to be processed in the current frame that requires image quality enhancement includes:

[0014] The image to be processed in the current frame is divided into binary blocks to obtain the corresponding first block binary image;

[0015] The values ​​in the first segmented binary image are initialized to 0 to obtain the second segmented binary image;

[0016] Based on the second target region for image quality enhancement required for the image to be processed in the previous frame, the first target region for image quality enhancement required for the image to be processed in the current frame is identified from the second block binary image.

[0017] Preferably, based on the second target region requiring image quality enhancement of the image to be processed in the previous frame, the first target region requiring image quality enhancement of the image to be processed in the current frame is identified from the second block binary image, including:

[0018] Determine whether the value of the second target region in the image to be processed in the previous frame that needs image quality enhancement is empty;

[0019] Calculate the first starting point based on the judgment result;

[0020] In the second block binary graph, the points around the first starting point are traversed to obtain the boundary points of the first target region of the image to be processed in the current frame that needs to be enhanced in terms of image quality. The boundary points of the first target region include the left boundary point, the right boundary point, the upper boundary point, and the lower boundary point.

[0021] Preferably, the calculation of the first starting point based on the judgment result includes:

[0022] If the value of the second target region is empty, the first starting point is calculated based on the first center point of the second block binary map;

[0023] If the value of the second target region is not empty, calculate the fuzzy value of the second center point of the second target region and the points around the second center point;

[0024] The second block binary image is updated using the fuzzy values ​​of the second center point and the points surrounding the second center point;

[0025] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are all specified values, then the second center point is determined as the first starting point.

[0026] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are not all the specified values, the first starting point is calculated based on the first center point.

[0027] Preferably, the process of calculating the first starting point based on the first center point includes:

[0028] Starting from the first center point, traverse outwards to obtain the target point;

[0029] Determine whether the target point meets the preset starting point requirements;

[0030] If the condition is not met, return to the step of traversing the surrounding area starting from the first center point to obtain the target point;

[0031] If the conditions are met, the target point is determined as the first starting point.

[0032] A second aspect of this invention discloses an image enhancement device, the device comprising:

[0033] The decoding unit is used to decode the video stream sent by the server to obtain a series of multiple frames of images to be processed.

[0034] The recognition unit is used to identify the first target area of ​​the image to be processed in the current frame that needs to be enhanced in terms of image quality;

[0035] An enhancement unit is used to enhance the image quality of the first target region of the image to be processed in the current frame, so as to obtain an enhanced image of the current frame.

[0036] Preferably, the enhancement unit includes:

[0037] The cropping module is used to crop the image corresponding to the first target region from the image to be processed in the current frame;

[0038] The enhancement module is used to enhance the image quality of the image corresponding to the first target area;

[0039] The fusion module is used to fuse the image to be processed in the current frame with the image corresponding to the first target region after image quality enhancement, so as to obtain the image quality enhanced image of the current frame.

[0040] A third aspect of the present invention discloses an electronic device, comprising: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the image enhancement method disclosed in the first aspect of the present invention.

[0041] A fourth aspect of the present invention discloses a storage medium storing computer-executable instructions for executing the image enhancement method disclosed in the first aspect of the present invention.

[0042] Based on the above embodiments of the present invention, an image quality enhancement method, apparatus, electronic device, and storage medium are provided. The method involves: decoding a video stream sent by a server to obtain multiple consecutive frames of images to be processed; identifying a first target region in the current frame of the image to be processed that requires image quality enhancement; and enhancing the first target region of the current frame of the image to be processed to obtain an enhanced image of the current frame. In this solution, the first target region in the current frame of the image to be processed that requires image quality enhancement is identified, and then the first target region of the current frame of the image to be processed is enhanced separately to obtain an enhanced image of the current frame. This eliminates the need to enhance the entire area of ​​the image, reducing computational time. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0044] Figure 1 A flowchart of an image quality enhancement method provided in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of obtaining an enhanced image provided in an embodiment of the present invention;

[0046] Figure 3 A flowchart for identifying a first target region provided in an embodiment of the present invention;

[0047] Figure 4 This is an example diagram of the first block binary image provided in an embodiment of the present invention;

[0048] Figure 5 An example diagram of the first target area provided in an embodiment of the present invention;

[0049] Figure 6 Another flowchart for identifying the first target region provided in an embodiment of the present invention;

[0050] Figure 7 This is a structural block diagram of an image enhancement device provided in an embodiment of the present invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0053] To improve the quality of videos played by the player, users watching standard definition (SD) or low definition (LD) videos can use AI super-resolution computing to upscale them to ultra-high definition (UHD). This saves users data usage while allowing them to watch higher-quality videos. However, AI super-resolution computing is computationally intensive and requires a considerable amount of time to perform image enhancement.

[0054] To address the aforementioned issues, embodiments of the present invention provide an image quality enhancement method, apparatus, electronic device, and storage medium. The method identifies a first target region of the current frame's image to be processed that requires image quality enhancement, and then performs image quality enhancement solely on this first target region, thereby obtaining an enhanced image of the current frame. This eliminates the need to enhance the entire image, reducing computational time.

[0055] Overall, this solution only enhances the image quality of the physical regions (the first target region, mainly the visible areas of the human face and the real object image region) using AI super-resolution calculation, while other regions (such as virtual and blurred regions) are enlarged proportionally, thus saving computational resources. Furthermore, for images spanning multiple consecutive frames, the enhanced image quality region of the previous frame (i.e., the second target region) can be referenced to further improve the separation efficiency of the first target region in the current frame. Detailed implementation processes are described in the following embodiments.

[0056] See Figure 1 The flowchart of an image quality enhancement method provided by an embodiment of the present invention is shown. The method includes:

[0057] Step S101: Decode the video stream sent by the server to obtain a series of multiple frames of images to be processed.

[0058] In the specific implementation step S101, the video stream sent by the server is obtained, and the video stream is decoded by the player to obtain a series of multiple frames of images to be processed.

[0059] Step S102: Identify the first target area in the current frame of the image to be processed that needs image quality enhancement.

[0060] In the specific implementation step S102, referring to the second target region of the previous frame of the image to be processed that needs image quality enhancement, the first target region of the current frame of the image to be processed that needs image quality enhancement is identified. The first target region is the region in the current frame of the image to be processed that needs image quality enhancement (e.g., AI super-resolution calculation). Similarly, the second target region is the region in the previous frame of the image to be processed that underwent image quality enhancement.

[0061] Step S103: Enhance the image quality of the first target region of the image to be processed in the current frame to obtain the enhanced image quality of the current frame.

[0062] In the specific implementation step S103, the first target region of the image to be processed in the current frame is enhanced to obtain the enhanced image of the current frame. The enhanced image is the final image after AI super-resolution calculation.

[0063] In some embodiments, the image enhancement image of the current frame replaces the "image to be processed in the current frame" (equivalent to the source image) and is added to the rendering queue for image rendering.

[0064] By using the above method, each frame of the image to be processed in the video stream is processed using steps S101 to S103, thereby completing the image quality enhancement of the video stream.

[0065] In this embodiment of the invention, a first target region of the current frame image to be processed that needs image quality enhancement is identified, and then the first target region of the current frame image to be processed is enhanced separately to obtain the image quality enhanced image of the current frame. It is not necessary to enhance the image quality of the entire image, thus reducing the computation time.

[0066] Regarding the above embodiments of the present invention Figure 1 For the step S103, "obtain the image quality enhancement image of the current frame", please refer to [link / reference]. Figure 2 The flowchart for obtaining an enhanced image according to an embodiment of the present invention is shown, including the following steps:

[0067] Step S201: Crop the image corresponding to the first target region from the image to be processed in the current frame.

[0068] In the specific implementation step S201, the first target region is mapped back to the image to be processed in the current frame, and the image corresponding to the first target region is extracted.

[0069] Step S202: Enhance the image quality of the image corresponding to the first target area.

[0070] In the specific implementation step S202, select an AI super-resolution model with A times image quality enhancement. For example, select an AI super-resolution model with 1.5 times or 2 times image quality enhancement according to actual needs.

[0071] The image quality of the first target region is enhanced by using an AI super-resolution model. The image of the first target region after the image quality enhancement is the image after AI super-resolution.

[0072] Step S203: Merge the image to be processed in the current frame with the image corresponding to the first target area after image quality enhancement to obtain the image quality enhanced image of the current frame.

[0073] In the specific implementation step S203, it is determined whether image quality enhancement has been performed on the image corresponding to the first target area. If image quality enhancement has not been performed, the image to be processed in the current frame (source image) can be rendered directly.

[0074] If image enhancement is performed, the "image to be processed in the current frame" and the "image corresponding to the first target area after image enhancement" are merged to obtain the image enhancement image of the current frame.

[0075] Specifically, the "image corresponding to the first target area that has undergone image enhancement" is the image corresponding to the "image with blurred background areas excluded," while the area without image enhancement is the "image with blurred background areas." Therefore, during fusion, based on the "image corresponding to the first target area that has undergone image enhancement," a bilinear scaling algorithm is used to scale the pixels corresponding to the "image with blurred background areas" by a factor of A to fill the pixel values ​​of the area without image enhancement (which can be understood as the area outside the first target area), thereby completing the entire image and obtaining the image enhancement image of the current frame (the final image after AI super-resolution calculation).

[0076] The above embodiments of the present invention Figure 2 This is an explanation of how to obtain an enhanced image of the current frame.

[0077] It should be noted that when identifying the first target area, the calculation of the fuzzy value is involved. The fuzzy value can be calculated by the following formulas (1)-(4).

[0078] L(x,y)=I(x,y) H(1);

[0079] (2);

[0080] (3);

[0081] (4);

[0082] In formulas (1) to (4), I(x,y) represents the points around point (x,y), H is a matrix, and L(x,y) represents the value obtained by multiplying the points around point (x,y) with matrix H. The average value of L(x,y) is calculated for all points in the image, where M and N are the dimensions of the image; This represents the average (i.e., variance) of the sum of squares of all points calculated using "L(x,y)" minus the mean. It can represent the blur value of an image. When the value is less than 100, it can be represented as fuzzy.

[0083] Regarding the above embodiments of the present invention Figure 1 The first target area involved in step S102, see [link / reference]. Figure 3 The flowchart illustrating the identification of a first target region provided by an embodiment of the present invention includes the following steps:

[0084] Step S301: Divide the current frame image to be processed into binary blocks to obtain the corresponding first block binary image.

[0085] It should be noted that binary block partitioning refers to dividing an image into multiple image blocks according to a predetermined block size, and then performing binarization processing on each image block, quantizing the value of each image block to 0 or 1 (which may be updated to 2 later), and finally forming a block binary image.

[0086] It should be further noted that the resolution of video streams is usually greater than 640. 360 high resolution, first block binary image (i.e., binary region template, size n) When m) is applied to the video stream (i.e., during the current playback), the first block binary image will not change during playback, and the first block binary image can reduce redundant region blur calculations.

[0087] In the specific implementation step S301, the resolution size of the image to be processed in the current frame is obtained as w. h, proportionally scale this resolution to 640 The scaling factor of 360 is s, therefore w1 = w s, h1=h s.

[0088] Divide w1 and h1 into blocks using squares of size stride (stride value between 10 and 20), resulting in n = (w1 + stride - 1) / stride and m = (h1 + stride - 1) / stride. This yields the following: Figure 4 The dimension shown is n The first block of the binary graph of m.

[0089] For example, when enhancing the image quality of a 720p resolution image, w=1280, h=720. At this time, the scaling factor s is 0.5, w1=640, h1=360. When stride=15, n=43, m=24.

[0090] Step S302: Initialize the values ​​in the first block binary graph to 0 to obtain the second block binary graph.

[0091] In the specific implementation step S302, the values ​​in the first block binary graph are initialized to 0, that is, all the values ​​in the first block binary graph are set to 0, thereby obtaining the second block binary graph.

[0092] Step S303: Based on the second target region for image quality enhancement required in the previous frame of the image to be processed, identify the first target region for image quality enhancement required in the current frame of the image to be processed from the second block binary image.

[0093] In the specific implementation step S303, it is determined whether the value of the second target area that needs to be enhanced in the previous frame of the image to be processed is empty, and the first starting point is calculated based on the determination result.

[0094] The specific implementation of calculating the first starting point based on the judgment result is as follows: If the value of the second target region is empty, the first starting point is calculated based on the first center point of the second block binary image (how to calculate the first starting point will be explained later). Specifically, when the value of the second target region is empty, it indicates that there is no "second target region of the previous frame to be processed" for reference. At this time, the first center point (n / 2, m / 2) of the second block binary image is taken and the first starting point is recalculated.

[0095] If the value of the second target region is not empty, calculate the fuzzy value of the second center point of the second target region and the points around the second center point.

[0096] Specifically, when the value of the second target region is not empty, the second center point (x, y) of the "second target region of the previous frame image to be processed" is taken, along with the points (x-1, y), (x+1, y), (x, y-1), and (x, y+1) surrounding the second center point. The points (x-1, y), (x+1, y), (x, y-1), and (x, y+1) surrounding the second center point are the four points above, below, left, and right of (x, y) centered at (x, y).

[0097] Using the formulas (1) to (4) above, calculate the fuzzy values ​​of the second center point (x,y) and its surrounding points (x-1,y), (x+1,y), (x,y-1), and (x,y+1), respectively. These fuzzy values ​​are denoted as f(x,y), f(x-1,y), f(x+1,y), f(x,y-1), and f(x,y+1), respectively.

[0098] The second block binary image is updated using the blur values ​​of the second center point and the points surrounding the second center point. Specifically, the second block binary image corresponding to the current frame image to be processed is updated using the blur values ​​calculated above: f(x,y), f(x-1,y), f(x+1,y), f(x,y-1), f(x,y+1).

[0099] Taking "updating the second block binary graph using f(x,y)" as an example, the following explains how to update the second block binary graph: Determine if the value of f(x,y) is greater than F (F is a settable empirical value, for example, the default value of F is 100). If it is greater than F, set the value at position (x,y) in the second block binary graph to 2; otherwise, set the value at position (x,y) in the second block binary graph to 1. Continue this process to update the second block binary graph.

[0100] Determine whether the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are all specified values ​​(e.g., the specified value is 2).

[0101] That is, determine whether the values ​​at positions (x,y), (x-1,y), (x+1,y), (x,y-1), and (x,y+1) in the updated second block binary graph are all the specified values.

[0102] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary image are all specified values, it means that the "second target area of ​​the previous frame image to be processed" can be referenced, and the second center point (x,y) is taken as the first starting point.

[0103] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary image are not all the specified values, it indicates that there is no "second target region of the previous frame image to be processed" for reference, and the first starting point is calculated based on the first center point (n / 2, m / 2).

[0104] For example: Determine if the values ​​at positions (x,y), (x-1,y), (x+1,y), (x,y-1), and (x,y+1) in the updated second block binary image are all 2. If not, it indicates that there is no "second target region of the previous frame's image to be processed" for reference, and the first starting point is recalculated using (n / 2, m / 2). If yes, it means that the "second target region of the previous frame's image to be processed" is available for reference (i.e., the first starting point does not need to be recalculated), and the second center point (x,y) is used as the first starting point.

[0105] The above is an explanation of "calculating the first starting point based on the judgment result".

[0106] After determining the first starting point in the above way, the points around the first starting point are traversed in the second block binary image to obtain the boundary points of the first target area of ​​the current frame image to be processed, which needs to be enhanced in terms of image quality. The boundary points of the first target area include the left boundary point, the right boundary point, the upper boundary point, and the lower boundary point.

[0107] Specifically, we first define the first starting point as (x, y). We then traverse the points to the left of the first starting point (x, y) in the second block binary graph to find the left boundary point. The specific traversal method is as follows: traverse the column (x-1, y), and then simultaneously traverse (x-1, yj) and (x-1, y + j) from top to bottom, taking the values ​​of (x-1, yj) and (x-1, y + j) at the corresponding positions in the second block binary graph.

[0108] If the values ​​of (x-1, yj) and (x-1, y+j) at the corresponding positions in the second block binary image are 0, then the corresponding fuzzy values ​​are calculated using the fuzzy value calculation formulas given in formulas (1) to (4) above, and the values ​​of (x-1, yj) and (x-1, y+j) at the corresponding positions in the second block binary image are updated according to the calculated fuzzy values.

[0109] If the ratio of "(x-1, yj) and (x-1, y +j) at the corresponding positions in the second block binary graph" is 0, then the "(x-1, yj) and (x-1, y +j) at the corresponding positions in the second block binary graph" are directly adopted.

[0110] If a point in (x-1, yj) and (x-1, y+j) has a value of 2 (a specified value), then skip directly traversing the column (x-2, y); if all values ​​of 1 are traversed in the current column, then the current column xi is the left boundary point of the first target region.

[0111] Using the same method as described above to obtain the left boundary point, the right boundary point, upper boundary point, and lower boundary point of the first target area can be obtained, and the first target area can then be identified.

[0112] For example, using the above traversal method, we can obtain... Figure 5 The first target region shown is the region enclosed by "1", which is the first target region that has been identified.

[0113] In some embodiments, regarding the aforementioned "calculating the first starting point based on the first center point", the specific implementation of calculating the first starting point based on the first center point is as follows: traverse the surrounding area with the first center point as the starting point to obtain the target point; determine whether the target point meets the preset starting point requirements; if not, return to the step of traversing the surrounding area with the first center point as the starting point to obtain the target point; if it meets the requirements, determine the target point as the first starting point.

[0114] Specifically, starting from the first center point (n / 2, m / 2), we traverse outwards to obtain the target point, which is temporarily represented by (x, y).

[0115] To determine whether the target point (x,y) meets the preset starting point requirements, the determination method is as follows: First, take the values ​​at positions (x,y), (x-1,y), (x+1,y), (x,y-1), and (x,y+1) from the second block binary image. For the value at a certain position, if the value at that position is 0, calculate the fuzzy value at that position using the fuzzy value calculation formula given in formulas (1) to (4) above, and update the value at that position in the second block binary image according to the calculated fuzzy value (the update rule is to determine whether f is greater than F; if it is greater, the value at that position is 2; otherwise, the value at that position is 1); if the value at that position is not 0, directly use the value at that position.

[0116] Then, it is determined whether the values ​​at positions (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the second binary image block are all 2. If so, the target point meets the starting point requirement, and the target point (x, y) is determined as the first starting point. If not, the traversal and judgment process described above continues until the traversal is complete. If the first starting point has not been determined after the traversal, it can be determined that the current frame of the image to be processed does not have a first target region (equivalent to no region where image quality enhancement can be performed), and the process ends.

[0117] The above embodiments of the present invention Figure 3 This is an explanation of how to identify the first target area.

[0118] To better understand how to identify the first target area, through Figure 6 The flowchart shown is used as an example. Figure 6 The given flowchart includes the following steps:

[0119] Step 1: Initialize the values ​​in the first block binary graph to 0 to obtain the second block binary graph.

[0120] Step 2: When the value of the second target region is empty, it means that there is no "second target region of the previous frame image to be processed" for reference. At this time, take the first center point (n / 2, m / 2) of the second block binary image and recalculate the first starting point, then execute step 4.

[0121] Step 3: If the value of the second target region is not empty, take the second center point (x, y) of the "second target region of the previous frame image to be processed" and perform the first condition judgment.

[0122] The first condition is determined as follows: Calculate the fuzzy values ​​of the second center point (x, y) and its surrounding points (x-1, y), (x+1, y), (x, y-1), and (x, y+1). Update the second block binary image using the fuzzy values ​​of the second center point and its surrounding points. Determine whether the values ​​of the second center point and its surrounding points in the updated second block binary image are all specified values ​​(e.g., the specified value is 2).

[0123] If the values ​​of the second center point and the points around the second center point in the updated second block binary image are all specified values, it means that the "second target area of ​​the previous frame image to be processed" can be referenced, that is, the first condition is met. Then, the second center point (x,y) is taken as the first starting point, and step 5 is executed.

[0124] If the values ​​of the second center point and the points around the second center point in the updated second block binary image are not all the specified values, it indicates that there is no "second target region of the previous frame image to be processed" for reference, that is, the first condition is not met. The first starting point is calculated based on the first center point (n / 2, m / 2), and step 4 is executed.

[0125] Step 4: Starting from the first center point (n / 2, m / 2), traverse outwards to obtain the target point, which is temporarily represented by (x, y).

[0126] To determine whether the target point (x,y) meets the preset starting point requirements, the determination method is as follows: First, take the values ​​at positions (x,y), (x-1,y), (x+1,y), (x,y-1), and (x,y+1) from the second block binary image. For the value at a certain position, if the value at that position is 0, calculate the fuzzy value at that position using the fuzzy value calculation formula given in formulas (1) to (4) above, and update the value at that position in the second block binary image according to the calculated fuzzy value (the update rule is to determine whether f is greater than F; if it is greater, the value at that position is 2; otherwise, the value at that position is 1); if the value at that position is not 0, directly use the value at that position.

[0127] Then, determine if the values ​​at positions (x, y), (x-1, y), (x+1, y), (x, y-1), and (x, y+1) in the second binary image block are all 2. If yes, the target point meets the starting point requirement, and the target point (x, y) is determined as the first starting point, and step 5 is executed. If not, continue the traversal and judgment process described above until the traversal is complete. If the first starting point has not been determined after the traversal, it can be determined that the current frame of the image to be processed does not have a first target region (equivalent to no region where image quality enhancement can be performed), and the process ends.

[0128] Step 5: First, traverse the points to the left of the first starting point (x, y) in the second block binary graph to find the left boundary point. Specifically, traverse the column (x-1, y), and then simultaneously traverse (x-1, yj) and (x-1, y + j) from top to bottom, taking the values ​​of (x-1, yj) and (x-1, y + j) at the corresponding positions in the second block binary graph.

[0129] If the values ​​of (x-1, yj) and (x-1, y+j) at the corresponding positions in the second block binary image are 0, then the corresponding fuzzy values ​​are calculated using the fuzzy value calculation formulas given in formulas (1) to (4) above, and the values ​​of (x-1, yj) and (x-1, y+j) at the corresponding positions in the second block binary image are updated according to the calculated fuzzy values.

[0130] If the ratio of "(x-1, yj) and (x-1, y +j) at the corresponding positions in the second block binary graph" is 0, then the "(x-1, yj) and (x-1, y +j) at the corresponding positions in the second block binary graph" are directly adopted.

[0131] If a point in (x-1, yj) and (x-1, y+j) has a value of 2 (a specified value), then skip directly traversing the column (x-2, y); if all values ​​of 1 are traversed in the current column, then the current column xi is the left boundary point of the first target region.

[0132] Using the same method as described above to obtain the left boundary point, the right boundary point, upper boundary point, and lower boundary point of the first target area can be obtained, and the first target area can then be identified.

[0133] It should be noted that, Figure 6 The execution principle of each step has been explained in detail in the above embodiments, which can be found therein, and will not be repeated here.

[0134] Corresponding to the image quality enhancement method provided in the above embodiments of the present invention, see also... Figure 7 The present invention also provides a structural block diagram of an image quality enhancement device, which includes: a decoding unit 701, an identification unit 702 and an enhancement unit 703;

[0135] Decoding unit 701 is used to decode the video stream sent by the server to obtain a series of multiple frames of images to be processed.

[0136] The recognition unit 702 is used to identify the first target area of ​​the current frame image to be processed that needs to be enhanced in terms of image quality.

[0137] The enhancement unit 703 is used to enhance the image quality of the first target region of the image to be processed in the current frame, so as to obtain the image quality enhanced image of the current frame.

[0138] In this embodiment of the invention, a first target region of the current frame image to be processed that needs image quality enhancement is identified, and then the first target region of the current frame image to be processed is enhanced separately to obtain the image quality enhanced image of the current frame. It is not necessary to enhance the image quality of the entire image, thus reducing the computation time.

[0139] Preferred, combined Figure 7 The enhancement unit 703, as shown, includes a trimming module, an enhancement module, and a fusion module. The execution principles of each module are as follows:

[0140] The cropping module is used to crop the image corresponding to the first target region from the image to be processed in the current frame.

[0141] The enhancement module is used to enhance the image quality of the image corresponding to the first target area.

[0142] In its specific implementation, the enhancement module is used to: select an AI super-resolution model with A times image quality enhancement; and enhance the image quality of the image corresponding to the first target region through the AI ​​super-resolution model.

[0143] The fusion module is used to fuse the image to be processed in the current frame with the image corresponding to the first target area that has been enhanced to obtain the enhanced image of the current frame.

[0144] Preferred, combined Figure 7 The content shown indicates that the identification unit 702 includes a block module, an initialization module, and an identification module. The execution principle of each module is as follows:

[0145] The segmentation module is used to perform binary segmentation on the current frame of the image to be processed, and obtain the corresponding first segment binary image.

[0146] An initialization module is used to initialize the values ​​in the first block binary graph to 0 in order to obtain the second block binary graph.

[0147] The recognition module is used to identify the first target region that needs to be enhanced in the current frame of the image to be processed from the second block binary image, based on the second target region that needs to be enhanced in the image to be processed according to ... binary image.

[0148] In some embodiments, the recognition module is specifically used to: determine whether the value of the second target region that needs to be enhanced in the previous frame of the image to be processed is empty; calculate the first starting point based on the determination result; traverse the points around the first starting point in the second block binary map to obtain the boundary points of the first target region that needs to be enhanced in the current frame of the image to be processed, wherein the boundary points of the first target region include the left boundary point, the right boundary point, the upper boundary point, and the lower boundary point.

[0149] The implementation method for calculating the first starting point based on the judgment result is as follows: if the value of the second target region is empty, the first starting point is calculated based on the first center point of the second block binary map;

[0150] If the value of the second target region is not empty, calculate the fuzzy value of the second center point of the second target region and the points around the second center point;

[0151] Update the second block binary image using the fuzzy values ​​of the second center point and the points surrounding the second center point;

[0152] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are all specified values, then the second center point is determined as the first starting point.

[0153] If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are not all specified values, the first starting point is calculated based on the first center point.

[0154] Furthermore, the implementation method for calculating the first starting point based on the first center point is as follows: traverse the surrounding area with the first center point as the starting point to obtain the target point; determine whether the target point meets the preset starting point requirements; if not, return to the step of traversing the surrounding area with the first center point as the starting point to obtain the target point; if it meets the requirements, determine the target point as the first starting point.

[0155] Preferably, the present invention also provides an electronic device, including: a processor and a memory, the processor and the memory being connected via a bus; wherein, the processor is used to call and execute a program stored in the memory; the memory is used to store the program, the program being used to implement the image enhancement method provided in the above method embodiments.

[0156] Preferably, the present invention also provides a storage medium storing computer-executable instructions for executing the image enhancement method provided in the above-described method embodiments.

[0157] In summary, the embodiments of the present invention provide an image quality enhancement method, apparatus, electronic device, and storage medium, which identify a first target region of the current frame image to be processed that needs image quality enhancement, and then perform image quality enhancement on the first target region of the current frame image to be processed separately, thereby obtaining an image quality enhanced image of the current frame. It is not necessary to perform image quality enhancement on the entire region of the image, thus reducing the computation time.

[0158] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0159] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0160] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing image quality, characterized in that, The method includes: Decode the video stream sent by the server to obtain a series of multiple frames of images to be processed; Identify the first target region in the current frame of the image to be processed that requires image quality enhancement; Image quality enhancement is performed on the first target region of the image to be processed in the current frame to obtain the image quality enhanced image of the current frame.

2. The method according to claim 1, characterized in that, Image quality enhancement is performed on the first target region of the image to be processed in the current frame to obtain an enhanced image of the current frame, including: Crops the image corresponding to the first target region from the image to be processed in the current frame; Image quality enhancement is performed on the image corresponding to the first target region; The image to be processed in the current frame is merged with the image corresponding to the first target region after image quality enhancement to obtain the image quality enhanced image of the current frame.

3. The method according to claim 1, characterized in that, Identify the first target region in the current frame of the image to be processed that requires image quality enhancement, including: The image to be processed in the current frame is divided into binary blocks to obtain the corresponding first block binary image; The values ​​in the first segmented binary image are initialized to 0 to obtain the second segmented binary image; Based on the second target region for image quality enhancement required for the image to be processed in the previous frame, the first target region for image quality enhancement required for the image to be processed in the current frame is identified from the second block binary image.

4. The method according to claim 3, characterized in that, Based on the second target region for image quality enhancement required for the image to be processed in the previous frame, the first target region for image quality enhancement required for the image to be processed in the current frame is identified from the second block binary image, including: Determine whether the value of the second target region in the image to be processed in the previous frame that needs image quality enhancement is empty; Calculate the first starting point based on the judgment result; In the second block binary graph, the points around the first starting point are traversed to obtain the boundary points of the first target region of the image to be processed in the current frame that needs to be enhanced in terms of image quality. The boundary points of the first target region include the left boundary point, the right boundary point, the upper boundary point, and the lower boundary point.

5. The method according to claim 4, characterized in that, The first starting point is calculated based on the judgment result, including: If the value of the second target region is empty, the first starting point is calculated based on the first center point of the second block binary map; If the value of the second target region is not empty, calculate the fuzzy value of the second center point of the second target region and the points around the second center point; The second block binary image is updated using the fuzzy values ​​of the second center point and the points surrounding the second center point; If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are all specified values, then the second center point is determined as the first starting point. If the values ​​of the second center point and the points surrounding the second center point in the updated second block binary graph are not all the specified values, the first starting point is calculated based on the first center point.

6. The method according to claim 5, characterized in that, The process of calculating the first starting point based on the first center point includes: Starting from the first center point, traverse outwards to obtain the target point; Determine whether the target point meets the preset starting point requirements; If the condition is not met, return to the step of traversing the surrounding area starting from the first center point to obtain the target point; If the conditions are met, the target point is determined as the first starting point.

7. An image enhancement device, characterized in that, The device includes: The decoding unit is used to decode the video stream sent by the server to obtain a series of multiple frames of images to be processed. The recognition unit is used to identify the first target area of ​​the image to be processed in the current frame that needs to be enhanced in terms of image quality; An enhancement unit is used to enhance the image quality of the first target region of the image to be processed in the current frame, so as to obtain an enhanced image of the current frame.

8. The apparatus according to claim 7, characterized in that, The enhancement unit includes: The cropping module is used to crop the image corresponding to the first target region from the image to be processed in the current frame; The enhancement module is used to enhance the image quality of the image corresponding to the first target area; The fusion module is used to fuse the image to be processed in the current frame with the image corresponding to the first target region after image quality enhancement, so as to obtain the image quality enhanced image of the current frame.

9. An electronic device, characterized in that, include: A processor and a memory are connected via a bus; wherein the processor is used to call and execute a program stored in the memory; The memory is used to store a program for implementing the image enhancement method as described in any one of claims 1-6.

10. A storage medium, characterized in that, The storage medium stores computer-executable instructions for performing the image enhancement method as described in any one of claims 1-6.

Images