Enhancement method, device and equipment of image frame and storage medium

Abstract

Embodiments of the present application provide an image frame enhancement method, device, equipment and storage medium, a plurality of local areas of a current image frame are analyzed and processed to obtain a pixel histogram of each local area, the pixel histogram includes a plurality of first pixel intervals and a first number of pixel points corresponding to each first pixel interval; a target scene category of each local area is determined according to the pixel histogram of each local area; a target mapping curve of each local area is determined according to the target scene category of each local area; and data processing is performed on the pixel points of each local area according to the target mapping curve of each local area to obtain a target image frame corresponding to the current image frame. The effectiveness of image frame enhancement is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a method, apparatus, device and storage medium for enhancing image frames. Background Technology

[0002] Image enhancement refers to improving the quality, sharpness, and visual effect of images through various techniques and algorithms. Among these, tone mapping methods are commonly used for color image enhancement. Tone mapping methods can include global tone mapping algorithms and local tone mapping methods.

[0003] In existing technologies, local tone mapping methods can simultaneously consider the spatial location and brightness information of pixels during image mapping, resulting in enhanced images with good local contrast and detail. However, in image frames with on-screen display (OSD) characteristics, local tone mapping methods are prone to negative effects such as color banding and halos, making the image enhancement less effective. Summary of the Invention

[0004] This application provides an image frame enhancement method, apparatus, device, and storage medium, which improves the effectiveness of image enhancement.

[0005] In a first aspect, embodiments of this application provide a method for enhancing an image frame, comprising:

[0006] The current image frame is analyzed and processed to obtain a pixel histogram for each local region. The pixel histogram includes multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval.

[0007] Based on the pixel histogram of each local region, the target scene category of each local region is determined;

[0008] Based on the target scene category of each local region, determine the target mapping curve for each local region;

[0009] Based on the target mapping curve of each local region, data processing is performed on the pixels of each local region to obtain the target image frame corresponding to the current image frame.

[0010] In one possible implementation, determining the target mapping curve for each local region based on the target scene category of each local region includes:

[0011] Based on the target scene category of each local region, an initial mapping curve is generated for each local region;

[0012] Determine whether the current image frame is a screen display OSD type;

[0013] If so, the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region;

[0014] If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region.

[0015] In one possible implementation, determining whether the current image frame is of OSD type includes:

[0016] Obtain multiple target scene categories in the current image frame;

[0017] Determine the number of regions corresponding to each target scene category in the local area;

[0018] The target scene category with the largest number of regions is determined as the first scene category;

[0019] Based on the number of regions in the first scene category, determine whether the current image frame is of OSD type.

[0020] In one possible implementation, determining whether the current image frame is of OSD type based on the number of regions in the first scene category includes:

[0021] If the number of regions in the first scene category is greater than the first threshold and multiple preset conditions are met, then the current image frame is of OSD type.

[0022] The multiple preset conditions include:

[0023] The difference between the scene number of the first scene category and the scene number of the second scene category is less than the second threshold, and the second scene category is the scene category with the fewest regions.

[0024] The first scene category is the OSD scene category;

[0025] The current image frame and the historical image frame switch scenes, and the historical image frame is a non-OSD type;

[0026] The saturation of the current image frame is less than the third threshold.

[0027] In one possible implementation, for any local region, the initial mapping curve of the local region is filtered to obtain the target mapping curve of the local region, including:

[0028] A first mapping curve and a second mapping curve are determined from the initial mapping curve. The first mapping curve is the mapping curve corresponding to the first scene category, and the second mapping curve is the mapping curve corresponding to the second scene category.

[0029] Based on the first mapping curve, the second mapping curve, and the filtering parameters, determine the adjustment mapping curve corresponding to the second mapping curve;

[0030] The sum of the second mapping curve and the adjusted mapping curve is determined as the third mapping curve for the second scene category;

[0031] The second mapping curve in the initial mapping curve is replaced with the third mapping curve to obtain the target mapping curve.

[0032] In one possible implementation, for any given local region, the target scene category of the local region is determined based on its pixel histogram, including:

[0033] Based on the first number of pixels corresponding to each first pixel interval, the second interval weight of the plurality of second pixel intervals is determined, wherein the interval range of the second pixel interval is greater than the interval range of the first pixel interval;

[0034] By comparing and analyzing the weights of the second intervals in each second pixel interval, the target scene category of the local region is determined.

[0035] In one possible implementation, for any second pixel interval; the second interval weight of the second pixel interval is determined based on the first number of pixels corresponding to each first pixel interval, including:

[0036] Obtain the first interval weight of each first pixel interval corresponding to the second pixel interval, and the first number and adjacent weight of the adjacent pixel intervals of the second pixel interval;

[0037] The first weight is determined by the sum of the products of the first quantity corresponding to each first pixel interval corresponding to the second pixel interval and the region weight.

[0038] The sum of the products of the first number of neighboring pixel regions and the neighbor weight is determined as the second weight;

[0039] The second interval weight is obtained by dividing the sum of the first weight and the second weight by the sum of the first interval weight and the adjacent weights.

[0040] In one possible implementation, data processing is performed on the pixels of each local region according to the target mapping curve of each local region to obtain the target image frame corresponding to the current image frame, including:

[0041] Based on the target mapping curve of each local region, the pixels of each local region are mapped to obtain a mapped image frame;

[0042] Based on the region weight of each local region, the mapped image frame is subjected to region interpolation to obtain the difference image frame;

[0043] Based on the current image frame and the historical image frames, anti-flicker processing is performed on the difference image frame to obtain the target image frame.

[0044] In one possible implementation, the target image frame is obtained by performing anti-flicker processing on the difference image frame based on the current image frame and the historical image frames, including:

[0045] Obtain the first adjustment weight, the second adjustment weight, and the third adjustment weight;

[0046] The target image frame is obtained by adjusting and integrating the current image frame, the historical image frame, and the difference image frame using the first adjustment weight, the second adjustment weight, and the third adjustment weight.

[0047] Secondly, embodiments of this application provide an image frame enhancement device, including an analysis and processing module, a first determination module, a second determination module, and a data processing module:

[0048] The analysis and processing module is used to analyze and process multiple local regions of the current image frame to obtain a pixel histogram of each local region. The pixel histogram includes multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval.

[0049] The first determining module is used to determine the target scene category of each local region based on the pixel histogram of each local region;

[0050] The second determining module is used to determine the target mapping curve of each local region according to the target scene category of each local region;

[0051] The data processing module is used to process the pixels of each local region according to the target mapping curve of each local region to obtain the target image frame corresponding to the current image frame.

[0052] In one possible implementation, the second determining module is specifically used for:

[0053] Based on the target scene category of each local region, an initial mapping curve is generated for each local region;

[0054] Determine whether the current image frame is a screen display OSD type;

[0055] If so, the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region;

[0056] If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region.

[0057] In one possible implementation, the second determining module is specifically used for:

[0058] Obtain multiple target scene categories in the current image frame;

[0059] Determine the number of regions corresponding to each target scene category in the local area;

[0060] The target scene category with the largest number of regions is determined as the first scene category;

[0061] Based on the number of regions in the first scene category, determine whether the current image frame is of OSD type.

[0062] In one possible implementation, the second determining module is specifically used for:

[0063] If the number of regions in the first scene category is greater than the first threshold and multiple preset conditions are met, then the current image frame is of OSD type.

[0064] The multiple preset conditions include:

[0065] The difference between the scene number of the first scene category and the scene number of the second scene category is less than the second threshold, and the second scene category is the scene category with the fewest regions.

[0066] The first scene category is the OSD scene category;

[0067] The current image frame and the historical image frame switch scenes, and the historical image frame is a non-OSD type;

[0068] The saturation of the current image frame is less than the third threshold.

[0069] In one possible implementation, for any given local region; the second determining module is specifically used for:

[0070] A first mapping curve and a second mapping curve are determined from the initial mapping curve. The first mapping curve is the mapping curve corresponding to the first scene category, and the second mapping curve is the mapping curve corresponding to the second scene category.

[0071] Based on the first mapping curve, the second mapping curve, and the filtering parameters, determine the adjustment mapping curve corresponding to the second mapping curve;

[0072] The sum of the second mapping curve and the adjusted mapping curve is determined as the third mapping curve for the second scene category;

[0073] The second mapping curve in the initial mapping curve is replaced with the third mapping curve to obtain the target mapping curve.

[0074] In one possible implementation, for any given local region, the first determining module is specifically used for:

[0075] Based on the first number of pixels corresponding to each first pixel interval, the second interval weight of the plurality of second pixel intervals is determined, wherein the interval range of the second pixel interval is greater than the interval range of the first pixel interval;

[0076] By comparing and analyzing the weights of the second intervals in each second pixel interval, the target scene category of the local region is determined.

[0077] In one possible implementation, for any second pixel interval; the first determining module is specifically used for:

[0078] Obtain the first interval weight of each first pixel interval corresponding to the second pixel interval, and the first number and adjacent weight of the adjacent pixel intervals of the second pixel interval;

[0079] The first weight is determined by the sum of the products of the first quantity corresponding to each first pixel interval corresponding to the second pixel interval and the region weight.

[0080] The sum of the products of the first number of neighboring pixel regions and the neighbor weight is determined as the second weight;

[0081] The second interval weight is obtained by dividing the sum of the first weight and the second weight by the sum of the first interval weight and the adjacent weights.

[0082] In one possible implementation, the data processing module is specifically used for:

[0083] Based on the target mapping curve of each local region, the pixels of each local region are mapped to obtain a mapped image frame;

[0084] Based on the region weight of each local region, the mapped image frame is subjected to region interpolation to obtain the difference image frame;

[0085] Based on the current image frame and the historical image frames, anti-flicker processing is performed on the difference image frame to obtain the target image frame.

[0086] In one possible implementation, the data processing module is specifically used for:

[0087] Obtain the first adjustment weight, the second adjustment weight, and the third adjustment weight;

[0088] The target image frame is obtained by adjusting and integrating the current image frame, the historical image frame, and the difference image frame using the first adjustment weight, the second adjustment weight, and the third adjustment weight.

[0089] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory;

[0090] The memory stores the instructions that the computer executes;

[0091] The processor executes computer execution instructions stored in memory, causing the processor to perform the method of the first aspect.

[0092] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method of the first aspect.

[0093] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method of the first aspect.

[0094] Sixthly, embodiments of this application provide a chip on which a computer program is stored. When the computer program is executed by the chip, it implements the method as described in the first aspect.

[0095] In one possible implementation, the chip is a chip in a chip module.

[0096] This application provides an image frame enhancement method, apparatus, device, and storage medium. It determines the target scene category of each local region using its pixel histogram, determines the target mapping curve for that local region based on the target scene category, and adjusts the pixels of the current image frame using the target mapping curve to obtain the target image frame. By mapping local regions using different mapping curves to obtain the target image frame, the negative effects can be mitigated, improving the effectiveness of image frame enhancement. Attached Figure Description

[0097] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0098] Figure 1 A schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0099] Figure 2 A schematic flowchart illustrating an image frame enhancement method provided in an embodiment of this application;

[0100] Figure 3 A schematic diagram of a pixel histogram provided in an embodiment of this application;

[0101] Figure 4 A schematic diagram of a target mapping curve provided in an embodiment of this application;

[0102] Figure 5 A flowchart illustrating the process of determining a target mapping curve, provided in an embodiment of this application;

[0103] Figure 6 A schematic diagram of a current image frame provided in an embodiment of this application;

[0104] Figure 7 This is a schematic diagram of the architecture for determining OSD type provided in an embodiment of this application;

[0105] Figure 8 A schematic diagram of a process for determining a target image frame provided in an embodiment of this application;

[0106] Figure 9 This is a schematic diagram of a region interpolation process provided in an embodiment of this application;

[0107] Figure 10 A classification diagram illustrating the correspondence provided in the embodiments of this application;

[0108] Figure 11 A schematic diagram of the structure of an image frame enhancement device provided in the application embodiment;

[0109] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0110] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0111] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the embodiments of this application. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0112] It should be noted that the terms "first," "second," and "third" used in the embodiments of this application are used to distinguish similar objects and should not be construed as indicating or implying relative importance or order. Optionally, without departing from the scope of this application, the first terminal device may also be referred to as the second terminal device, and similarly, the second terminal device may also be referred to as the first terminal device.

[0113] It should be further understood that the terms "comprising" or "including" indicate the presence of the aforementioned features, steps, operations, elements, components, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or at least one other feature, step, operation, element, component, type, and / or group. The terms "or," "and / or," and "comprising at least one of the following," as used in this application, can be interpreted as inclusive, or mean any one or any combination thereof. Optionally, "comprising at least one of the following: A, B, C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C," and similarly, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A and B and C."

[0114] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include at least one sub-step or at least one stage. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0115] An image frame refers to a still image in a digital image, typically composed of pixels. In video and image processing, an image frame is the basic unit in a video sequence, with each frame representing a single moment in the video. Image frames can be color or black and white, and have different resolutions and color depths.

[0116] Figure 1 This is a schematic diagram illustrating an application scenario provided by an embodiment of this application. Please refer to [link / reference]. Figure 1 The system includes a terminal device 101. A processing chip can be installed in the terminal device 101, which can enhance image frames. The processing chip can be an image processing chip or a video processing chip. The terminal device 101 may include a camera, an image / video processing platform, and a display device, etc.

[0117] Terminal device 101 can acquire the current image frame in the video, analyze and process multiple local regions in the current image frame to obtain a pixel histogram for each local region. The pixel histogram may include multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval. Terminal device 101 can determine the target scene category for each local region and, based on the scene type of each local region, determine the target mapping curve for each local region. Terminal device 101 can process the pixels of each local region according to the target mapping curve to obtain the target image frame corresponding to the current image frame.

[0118] In existing technologies, local tone mapping methods can simultaneously consider the spatial location and brightness information of pixels during image mapping, resulting in enhanced images with good local contrast and detail. However, in image frames with on-screen display (OSD) characteristics, local tone mapping methods are prone to negative effects such as color banding and halos, making the image enhancement less effective.

[0119] The image frame enhancement method provided in this application can determine the target scene category of each local region through the pixel histogram of each local region, determine the target mapping curve of the local region through the target scene category, and adjust the pixels of the current image frame through the target mapping curve to obtain the target image frame. By mapping the local regions with different mapping curves to obtain the target image frame, the influence of negative effects can be mitigated, and the effectiveness of image frame enhancement can be improved.

[0120] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0121] Figure 2 This is a schematic flowchart illustrating an image frame enhancement method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 The method may include:

[0122] S201. Analyze and process multiple local regions of the current image frame to obtain the pixel histogram of each local region.

[0123] The current image frame may include multiple pixels and the pixel value corresponding to each pixel.

[0124] The current image frame can be divided into multiple local regions. For example, assuming the current image frame has an image size of 1920×1080, it can be divided into 5×5 local regions, each with an image size of 384×216.

[0125] If the region division is not exact, the current image frame can be expanded using edge augmentation before performing region division.

[0126] Furthermore, for any local region, the pixel histogram can be determined as follows: obtain multiple first pixel intervals; among the multiple pixels in the local region, determine the first number of pixels corresponding to each first pixel interval; construct the pixel histogram based on the first number of pixels corresponding to each first pixel interval.

[0127] The range can be divided into multiple first pixel intervals. For example, assuming the range is 0-1023, if the range is divided into 16 first pixel intervals, then first pixel interval 0 is the range of 0-63, first pixel interval 1 is the range of 64-127, ..., first pixel interval 14 is the range of 896-959, and first pixel interval 15 is the range of 960-1023.

[0128] For any given local region, the first number of pixels in each first pixel interval corresponding to that local region can be determined.

[0129] A pixel histogram may include multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval.

[0130] The x-axis of the pixel histogram represents the pixel value, and the y-axis represents the number of pixels.

[0131] Figure 3 This is a schematic diagram of a pixel histogram provided in an embodiment of this application. Please refer to [link / reference]. Figure 3 The pixel histogram Figure 1 This is the pixel histogram of local region 1. Local region 1 has an image size of 384×216, meaning it contains 384×216 pixels. Assuming there are 16 first pixel intervals, ranging from 0 to 15, we can determine the first number of pixels corresponding to each first pixel interval in local region 1 and generate the pixel histogram. Figure 1 pixel histogram Figure 1 The horizontal axis represents the pixel value, and the vertical axis represents the number of pixels.

[0132] S202. Determine the target scene category for each local region based on the pixel histogram of each local region.

[0133] For any local region, the weights of the second intervals of multiple second pixel intervals can be determined based on the first number of pixels corresponding to each first pixel interval in the local region. The weights of the second intervals of each second pixel interval are then compared and analyzed to determine the target scene category of the local region.

[0134] The range of the second pixel interval is greater than the range of the first pixel interval, meaning that the second pixel interval may include at least one of the first pixel intervals.

[0135] For example, suppose there are 16 first pixel intervals, namely first pixel interval 0-15, and suppose there are 4 second pixel intervals, namely second pixel interval 1-4. Then second pixel interval 1 includes first pixel interval 0-3, second pixel interval 2 includes first pixel interval 4-7, second pixel interval 3 includes first pixel interval 8-11, and second pixel interval 4 includes first pixel interval 12-15.

[0136] The weights of the second intervals of multiple second pixel intervals in multiple historical local regions can be compared and analyzed to determine multiple scene categories. After obtaining the weights of the second intervals of multiple second pixel intervals in a local region of the current image frame, the target scene category corresponding to that local region can be determined from multiple scene categories.

[0137] Furthermore, for any second pixel interval, the first interval weight of each first pixel interval corresponding to the second pixel interval, as well as the first number and neighbor weight of the second pixel interval, can be obtained; the sum of the products of the first number and the region weight of each first pixel interval corresponding to the second pixel interval is determined as the first weight; the sum of the products of the first number and the neighbor weight of the neighboring pixel regions is determined as the second weight; the sum of the first weight and the second weight is divided by the sum of the first interval weight and the neighbor weight to obtain the second interval weight of the second pixel interval.

[0138] The first interval weight of each first pixel interval and the neighbor weight of adjacent pixel intervals can be preset.

[0139] At least one first pixel interval preceding the second pixel interval, and / or at least one first pixel interval following the second pixel interval, can be identified as a neighboring pixel region.

[0140] For example, suppose as Figure 3 As shown, the second pixel interval 1 corresponds to the first pixel interval 0-3, and the first pixel interval 4 can be determined as the neighboring pixel interval of the first pixel interval 1; the second pixel interval 2 corresponds to the first pixel interval 4-7, and the first pixel interval 3 and the first pixel interval 8 can be determined as the neighboring pixel intervals.

[0141] For example, suppose as Figure 3 As shown, the second pixel interval 1 corresponds to the first pixel interval 0-3, and the first pixel interval 4 and the first pixel interval 5 can be determined as the neighboring pixel intervals of the first pixel interval 1; the second pixel interval 2 corresponds to the first pixel interval 4-7, and the first pixel interval 2, the first pixel interval 3, the first pixel interval 7 and the first pixel interval 8 can be determined as the neighboring pixel intervals.

[0142] The first pixel interval can have its corresponding neighbor weight and first interval weight.

[0143] For example, suppose the neighboring pixel interval of the second pixel interval 1 is the first pixel interval 4, and the neighbor weight of the first pixel interval 4 is weight 1. The second pixel interval 2 corresponds to the first pixel interval 4, so the first interval weight of the first pixel interval 4 is weight 2. That is, the first pixel interval 4 has its corresponding neighbor weight and first interval weight.

[0144] For example, suppose as Figure 3 As shown, the second pixel interval 1 corresponds to the first pixel interval 0-3. Assuming the neighboring pixel intervals of the second pixel interval 1 are the first pixel interval 4 and the first pixel interval 5, and assuming the first interval weights of the first pixel intervals 0-3 are k0, k1, k2, and k3 respectively, and the first quantities of the first pixel intervals 0-3 are bin0, bin1, bin2, and bin3 respectively, and the neighboring weights of the first pixel intervals 4 and 5 are l4 and l5 respectively, and the first quantities of the first pixel intervals 4 and 5 are bin4 and bin5 respectively, then the second interval weight of the second pixel interval 1 can be determined.

[0145] S203. Determine the target mapping curve for each local area based on the target scene category of each local area.

[0146] The current image frame can include multiple local regions, and the target mapping curve for each local region can be determined. For any local region, the target scene category can be used to adopt its corresponding target mapping curve, making the processing of the current image frame more accurate and improving the effectiveness of image enhancement.

[0147] The target mapping curve can be fitted by multiple mapping curves with different slopes. A mapping curve can correspond to a first pixel interval, that is, the pixels in the first pixel interval can be mapped according to the mapping curve.

[0148] Figure 4 This is a schematic diagram of a target mapping curve provided in an embodiment of this application. Please refer to... Figure 4 The target mapping curve has the input pixel value on the x-axis and the output pixel value on the y-axis. There are 16 first pixel intervals, each with its corresponding mapping curve, which represents the mapping function for that first pixel interval.

[0149] Target mapping curves can be used to indicate how pixels are processed.

[0150] S204. Based on the target mapping curve of each local region, perform data processing on the pixels of each local region to obtain the target image frame corresponding to the current image frame.

[0151] Based on the target mapping curve of each local region, the pixels of each local region can be mapped to obtain a mapped image frame.

[0152] In some embodiments, the mapped image frame can be determined as the target image frame.

[0153] In some embodiments, the mapped image frame may be processed in at least one of the following ways: region interpolation and anti-flicker processing.

[0154] The image frame enhancement method provided in this application can determine the target scene category of each local region through the pixel histogram of each local region, determine the target mapping curve of the local region through the target scene category, and adjust the pixels of the current image frame through the target mapping curve to obtain the target image frame. By mapping the local regions with different mapping curves to obtain the target image frame, the effectiveness of image frame enhancement is improved.

[0155] Figure 5 This is a schematic flowchart illustrating the process of determining a target mapping curve, as provided in an embodiment of this application. Please refer to... Figure 5 The method may include:

[0156] S501. Generate the initial mapping curve for each local region based on the target scene category of each local region.

[0157] For any local region, the initial starting value and initial slope of each first pixel interval corresponding to the target scene category can be obtained; based on the initial starting value and initial slope of each first pixel interval, an initial mapping curve is generated.

[0158] For any first pixel interval, a mapping function between input pixels and output pixels can be constructed based on the initial starting value and the initial slope. Pixels within the first pixel interval can be used as input pixels, and the output pixels are determined by the mapping function.

[0159] S502, Obtain multiple target scene categories in the current image frame.

[0160] The current image frame can include multiple local regions, and the target scene categories in the current image frame can be determined based on the target scene category corresponding to each local region.

[0161] Figure 6 This is a schematic diagram of a current image frame provided in an embodiment of this application. Please refer to [link / reference]. Figure 6 The current image frame comprises 5x5 local regions. Assuming there are 32 scene categories with scene numbers ranging from 1 to 32, the scene numbers for the 5x5 local regions can be as follows: Figure 6 As shown, two target scene categories can be identified, namely scene category 12 and scene category 30.

[0162] S503. Determine the number of regions corresponding to each target scene category in the local area.

[0163] For example, suppose the current image frame can be like Figure 6 As shown, the current image frame includes 5*5 local regions, and the target scene types are scene category 12 and scene category 30, respectively. The number of local regions corresponding to scene category 12 is 22, and the number of local regions corresponding to scene category 30 is 3.

[0164] S504. The target scene category with the most regions is determined as the first scene category.

[0165] For example, suppose there are two target scene categories, scene category 12 and scene category 30. Suppose the number of local regions corresponding to scene category 12 is 22 and the number of local regions corresponding to scene category 30 is 3. Then the first scene category can be determined to be scene category 12.

[0166] S505. Based on the number of regions in the first scene category, determine whether the current image frame is of OSD type.

[0167] The current image frame of OSD type can have the following characteristics: the main image is a non-natural image; the background color is uniform, usually black / white / gray; the image content is accompanied by large areas of white space.

[0168] Figure 7 This is a schematic diagram of the architecture for determining the OSD type provided in an embodiment of this application. Please refer to [link / reference]. Figure 7If the number of regions of the first scene category is greater than the first threshold and multiple preset conditions are met, then the current image frame is of type OSD.

[0169] The preset conditions may include: the difference between the scene number of the first scene category and the scene number of the second scene category is less than a second threshold; the first scene category is an OSD scene number; a scene switch occurs between the current image frame and a historical image frame, and the historical image frame is a non-OSD type; and the saturation of the current image frame is less than a third threshold.

[0170] If any of the preset conditions are not met, the current image frame is a non-OSD type.

[0171] The second scene category is the scene category with the fewest regions in the current image frame. For example, assuming there are three target scene categories in a local region, namely scene category 1, scene category 2 and scene category 3, and assuming that scene category 1 has the most regions and scene category 3 has the fewest regions, then the first scene category can be determined as scene category 1 and the second scene category as scene category 3.

[0172] The difference value is the absolute value of the difference between the scene number of the first scene category and the scene number of the second scene category. For example, if the scene number of the first scene category is 12 and the scene number of the second scene category is 30, then the difference value can be determined to be 18.

[0173] Multiple scene categories may include OSD scene numbers and non-OSD scene numbers.

[0174] A scene switch occurs between the current image frame and a historical image frame when any of the following conditions are met: Condition 1: The average brightness of the current image frame has changed significantly compared to the previous historical image frame; Condition 2: The partition statistics of the whole frame histogram of the current image frame have changed significantly; Condition 3: The scene sequence number of the local areas in the same position in most local areas has changed significantly compared to the previous historical image frame in the time domain.

[0175] It should be noted that the judgment condition 2 is the whole frame histogram of the current image frame. That is, in the pixel histogram, the first quantity corresponding to each first pixel interval is the number of pixels in that pixel interval in the entire current image frame.

[0176] Regarding condition 3, the absolute value of the difference between the current scene index of each local region in the current image frame and the historical scene index of each local region in the historical image frame can be determined as the difference value; a second number of differences greater than the fourth threshold can be determined. If the second number is greater than the fifth threshold, then a significant change has occurred.

[0177] S506. If so, then the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region.

[0178] The initial mapping curve may include a first mapping curve and a second mapping curve. The first mapping curve may be the mapping curve corresponding to the first scene category, and the second mapping curve may be the mapping curve corresponding to the second scene category.

[0179] For any local region, determine the first and second mapping curves in the initial mapping curves; determine the adjusted mapping curve of the second mapping curve pair based on the first and second mapping curves and the filtering parameters; determine the third mapping curve of the second scene category by summing the second mapping curve and the adjusted mapping curve; replace the second mapping curve in the initial mapping curve with the third mapping curve to obtain the target mapping curve.

[0180] The formula for determining the third mapping curve can be shown below:

[0181] Ts=Cs+[(OSDs-Cs)*δ] / 255

[0182] Where Ts is the third mapping curve, Cs is the second mapping curve, OSDs is the first mapping curve, and δ is the filtering parameter.

[0183] The third mapping curve is obtained by adjusting the second mapping curve using the first mapping curve. The third mapping curve is then used to process the current image frame, which can alleviate problems such as color banding and halo, and improve the effectiveness of image frame enhancement processing.

[0184] S507. If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region.

[0185] The image frame enhancement method provided in this application embodiment can determine whether the current image frame is of OSD type based on the number of regions of the first scene category in the current image frame. If it is of OSD type, it can be adjusted by the first mapping curve of the first scene category and the second mapping curve of the second scene category to obtain the target mapping curve. The pixels of the current image frame can be enhanced by the adjusted target mapping curve, which can improve the effectiveness of image frame enhancement.

[0186] Figure 8 This is a schematic diagram illustrating a process for determining a target image frame, as provided in an embodiment of this application. Please refer to... Figure 8 The method includes:

[0187] S801. Based on the target mapping curve of each local region, perform mapping processing on the pixels of each local region to obtain a mapped image frame.

[0188] For example, suppose the current image frame has 5*5 local regions, and the target mapping curve 1 corresponds to local region 1. Suppose the pixel value of pixel 11 in local region 1 is the initial pixel 11. The target pixel 11 corresponding to the initial pixel 11 can be determined in the target mapping curve. In this way, the target pixel values ​​of all pixels in the current image frame can be obtained, and the mapped image frame can be obtained.

[0189] For any given local region, the target pixel matrix corresponding to that local region can be determined using the following formula:

[0190]

[0191] Where F is the target pixel matrix, Y in This is the initial pixel matrix for the local region. This is the function expression for the target mapping curve.

[0192] S802. Based on the region weight of each local region, perform region interpolation on the mapped image frame to obtain the difference image frame.

[0193] By using region interpolation, block artifacts between local regions can be avoided. Block artifacts refer to the visual discontinuity or unnaturalness caused by large variations in pixel values ​​between adjacent regions in image or video processing.

[0194] For any given local region, the region weight is the weight matrix of the neighboring regions of that local region.

[0195] For regional interpolation processing, please refer to the following formula:

[0196] Y new =W*F / sum

[0197] Where W is the region weight matrix of the current processing region, F is the target pixel matrix of the current processing region, and sum is the sum of the weights of multiple regions in the region weight matrix.

[0198] Figure 9 This is a schematic diagram illustrating a region interpolation process provided in an embodiment of this application. Please refer to [link / reference]. Figure 9 The current image frame has 5*5 local regions. The current processing region of the current image frame is local region 22. Interpolation is performed through the 3*3 neighborhood of local region 22, and the region weight matrix is ​​as follows: Among them, w ij Let be the weight of local region ij relative to local region 22. i can take values ​​from 1, 2, and 3, and j can take values ​​from 1, 2, and 3.

[0199] S803. Based on the current image frame and historical image frames, perform anti-flicker processing on the difference image frame to obtain the target image frame.

[0200] Obtain the first adjustment weight, the second adjustment weight, and the third adjustment weight; through the first adjustment weight, the second adjustment weight, the third adjustment weight, and the target correspondence, adjust and integrate the current image frame, the historical image frame, and the difference image frame to obtain the target image frame.

[0201] The historical image frame can be the previous frame of the current image frame, and the historical image frame is the target image frame after pixel adjustment at a historical moment.

[0202] The difference image frames can be adjusted and integrated according to the following formula:

[0203] YoutTmp=[α*Yin+β*(Ynew-Yin)+γ*(YnewPre-Ynew)] / α

[0204] Where YoutTmp is the target image frame, Yin is the difference image frame, YnewPre is the historical image frame, Ynew is the current image frame, α is the first adjustment weight, β is the second adjustment weight, and γ is the third adjustment weight.

[0205] Furthermore, the deceleration rate can be determined based on the target correspondence between the current image frame and historical image frames, and the initial adjustment weight can be subtracted from the deceleration rate to obtain the third adjustment weight.

[0206] γ = γ0 - k, if γ > 0

[0207] Where γ is the third adjustment weight after processing, γ0 is the third adjustment weight, and k is the deceleration rate.

[0208] Based on the relationship between the current image frame and historical image frames, four correspondences can be determined, which are discussed below. Figure 10 The four correspondences provided in the embodiments of this application will be further explained.

[0209] Figure 10 This is a classification diagram illustrating the correspondence provided in the embodiments of this application. Please refer to... Figure 10 The first correspondence is that the historical image frame is of non-OSD type and the current image frame is of OSD type; the second correspondence is that the historical image frame is of non-OSD type and the current image frame is of non-OSD type; the third correspondence is that the historical image frame is of OSD type and the current image frame is of non-OSD type; the fourth correspondence is that the historical image frame is of OSD type and the current image frame is of OSD type.

[0210] The deceleration rate for each corresponding relationship can be preset.

[0211] The image frame enhancement method provided in this application embodiment can perform process mapping, region interpolation, and anti-flickering processing on the current image frame to obtain the target image frame, thereby improving the effectiveness of image frame enhancement.

[0212] Figure 11 This is a schematic diagram of an image frame enhancement device provided in an embodiment of the application. Please refer to [link / reference]. Figure 11 The image frame enhancement device may include an analysis and processing module 11, a first determination module 12, a second determination module 13, and a data processing module 14.

[0213] The analysis and processing module 11 is used to analyze and process multiple local regions of the current image frame to obtain a pixel histogram of each local region. The pixel histogram includes multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval.

[0214] The first determining module 12 is used to determine the target scene category of each local region based on the pixel histogram of each local region.

[0215] The second determining module 13 is used to determine the target mapping curve for each local region based on the target scene category of each local region.

[0216] The data processing module 14 is used to process the data of each local region's pixels according to the target mapping curve of each local region, so as to obtain the target image frame corresponding to the current image frame.

[0217] The image frame enhancement device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0218] In one possible implementation, the second determining module 13 is specifically used for:

[0219] Based on the target scene category of each local region, an initial mapping curve is generated for each local region;

[0220] Determine whether the current image frame is an OSD type for screen display;

[0221] If so, the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region;

[0222] If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region.

[0223] In one possible implementation, the second determining module 13 is specifically used for:

[0224] Retrieve multiple target scene categories in the current image frame;

[0225] Determine the number of regions corresponding to each target scene category in the local area;

[0226] The target scene category with the most regions is identified as the first scene category;

[0227] Based on the number of regions in the first scene category, determine whether the current image frame is of OSD type.

[0228] In one possible implementation, the second determining module 13 is specifically used for:

[0229] If the number of regions in the first scene category is greater than the first threshold and multiple preset conditions are met, then the current image frame is of type OSD.

[0230] Several preset conditions include:

[0231] The difference between the scene number of the first scene category and the scene number of the second scene category is less than the second threshold, and the second scene category is the scene category with the fewest regions.

[0232] The first scene category is the OSD scene category;

[0233] The current image frame and the historical image frame have switched scenes, and the historical image frame is a non-OSD type;

[0234] The saturation of the current image frame is less than the third threshold.

[0235] In one possible implementation, for any given local region, the second determining module 13 is specifically used for:

[0236] In the initial mapping curve, a first mapping curve and a second mapping curve are determined. The first mapping curve is the mapping curve corresponding to the first scene category, and the second mapping curve is the mapping curve corresponding to the second scene category.

[0237] Based on the first mapping curve, the second mapping curve, and the filtering parameters, determine the adjustment mapping curve corresponding to the second mapping curve;

[0238] The sum of the second mapping curve and the adjusted mapping curve is determined as the third mapping curve for the second scene category;

[0239] Replace the second mapping curve in the initial mapping curve with the third mapping curve to obtain the target mapping curve.

[0240] In one possible implementation, for any given local region, the first determining module 12 is specifically used for:

[0241] Based on the first number of pixels corresponding to each first pixel interval, the second interval weights of multiple second pixel intervals are determined, and the interval range of the second pixel interval is greater than the interval range of the first pixel interval.

[0242] By comparing and analyzing the weights of the second intervals for each second pixel interval, the target scene category of the local region can be determined.

[0243] In one possible implementation, for any second pixel interval, the first determining module 12 is specifically used for:

[0244] Obtain the first interval weight of each first pixel interval corresponding to the second pixel interval, as well as the first number and neighbor weight of the neighboring pixel intervals of the second pixel interval;

[0245] The first weight is determined by the sum of the products of the first quantity and the region weight corresponding to each first pixel interval corresponding to the second pixel interval.

[0246] The sum of the products of the first number of neighboring pixel regions and the neighbor weight is determined as the second weight;

[0247] The second interval weight is obtained by dividing the sum of the first and second weights by the sum of the first interval weights and the adjacent weights.

[0248] In one possible implementation, the data processing module 14 is specifically used for:

[0249] Based on the target mapping curve of each local region, the pixels of each local region are mapped to obtain a mapped image frame;

[0250] Based on the region weight of each local region, region interpolation is performed on the mapped image frame to obtain the difference image frame;

[0251] Based on the current image frame and historical image frames, anti-flicker processing is performed on the difference image frame to obtain the target image frame.

[0252] In one possible implementation, the data processing module 14 is specifically used for:

[0253] Obtain the first adjustment weight, the second adjustment weight, and the third adjustment weight;

[0254] The target image frame is obtained by adjusting and integrating the current image frame, historical image frames, and difference image frames through the first adjustment weight, the second adjustment weight, and the third adjustment weight.

[0255] The image frame enhancement device provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0256] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Please refer to... Figure 12 The electronic device 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0257] Memory 22 stores instructions executed by the computer;

[0258] The processor 21 executes computer execution instructions stored in the memory 22, causing the processor 21 to execute the image frame enhancement method as described in the above method embodiment.

[0259] Electronic devices can include chips, modules, integrated development environments (IDEs), etc.

[0260] Accordingly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the image frame enhancement method of the above-described method embodiments.

[0261] Accordingly, embodiments of this application may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the image frame enhancement method shown in the above method embodiments.

[0262] The computer-readable storage medium and computer program product of this application embodiment can execute the above-described information reporting method. The specific implementation process and beneficial effects are described above and will not be repeated here.

[0263] All or part of the steps in the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof.

[0264] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0265] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0266] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0267] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A method for enhancing an image frame, characterized in that, include: The current image frame is analyzed and processed to obtain a pixel histogram for each local region. The pixel histogram includes multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval. Based on the pixel histogram of each local region, the target scene category of each local region is determined; Based on the target scene category of each local region, determine the target mapping curve for each local region; Based on the target mapping curve of each local region, data processing is performed on the pixels of each local region to obtain the target image frame corresponding to the current image frame; Specifically, determining the target mapping curve for each local region based on the target scene category of each local region includes: Based on the target scene category of each local region, an initial mapping curve is generated for each local region; Obtain multiple target scene categories in the current image frame; Determine the number of regions corresponding to each target scene category in the local area; The target scene category with the largest number of regions is determined as the first scene category; if the number of regions in the first scene category is greater than a first threshold and meets multiple preset conditions, then the current image frame is of OSD type; the multiple preset conditions include: the difference between the scene number of the first scene category and the scene number of the second scene category is less than a second threshold, the second scene category is the scene category with the fewest regions; the first scene category is an OSD scene category; the current image frame and a historical image frame undergo scene switching, and the historical image frame is not of OSD type; the saturation of the current image frame is less than a third threshold; If so, the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region; If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region; For any given local region; the initial mapping curve of the local region is filtered to obtain the target mapping curve of the local region, including: A first mapping curve and a second mapping curve are determined from the initial mapping curve. The first mapping curve is the mapping curve corresponding to the first scene category, and the second mapping curve is the mapping curve corresponding to the second scene category. Based on the first mapping curve, the second mapping curve, and the filtering parameters, determine the adjustment mapping curve corresponding to the second mapping curve; The sum of the second mapping curve and the adjusted mapping curve is determined as the third mapping curve for the second scene category; The second mapping curve in the initial mapping curve is replaced with the third mapping curve to obtain the target mapping curve.

2. The method according to claim 1, characterized in that, For any given local region; based on the pixel histogram of the local region, determine the target scene category of the local region, including: Based on the first number of pixels corresponding to each first pixel interval, a second interval weight of multiple second pixel intervals is determined, wherein the interval range of the second pixel interval is greater than the interval range of the first pixel interval. By comparing and analyzing the weights of the second intervals in each second pixel interval, the target scene category of the local region is determined.

3. The method according to claim 2, characterized in that, For any given second pixel interval; based on the first number of pixels corresponding to each first pixel interval, determine the second interval weight of the second pixel interval, including: Obtain the first interval weight of each first pixel interval corresponding to the second pixel interval, and the first number and adjacent weight of the adjacent pixel intervals of the second pixel interval; The first weight is determined by the sum of the products of the first quantity corresponding to each first pixel interval corresponding to the second pixel interval and the region weight. The sum of the products of the first number of neighboring pixel intervals and the neighbor weight is determined as the second weight; The second interval weight is obtained by dividing the sum of the first weight and the second weight by the sum of the first interval weight and the adjacent weights.

4. The method according to claim 3, characterized in that, Based on the target mapping curve of each local region, data processing is performed on the pixels of each local region to obtain the target image frame corresponding to the current image frame, including: Based on the target mapping curve of each local region, the pixels of each local region are mapped to obtain a mapped image frame; Based on the region weight of each local region, the mapped image frame is subjected to region interpolation to obtain the difference image frame; Based on the current image frame and the historical image frames, anti-flicker processing is performed on the difference image frame to obtain the target image frame.

5. The method according to claim 4, characterized in that, Based on the current image frame and historical image frames, anti-flicker processing is performed on the difference image frame to obtain the target image frame, including: Obtain the first adjustment weight, the second adjustment weight, and the third adjustment weight; The target image frame is obtained by adjusting and integrating the current image frame, the historical image frame, and the difference image frame using the first adjustment weight, the second adjustment weight, and the third adjustment weight.

6. An image frame enhancement device, characterized in that, It includes an analysis and processing module, a first determination module, a second determination module, and a data processing module: The analysis and processing module is used to analyze and process multiple local regions of the current image frame to obtain a pixel histogram of each local region. The pixel histogram includes multiple first pixel intervals and a first number of pixels corresponding to each first pixel interval. The first determining module is used to determine the target scene category of each local region based on the pixel histogram of each local region; The second determining module is used to determine the target mapping curve of each local region according to the target scene category of each local region; The data processing module is used to process the pixels of each local region according to the target mapping curve of each local region to obtain the target image frame corresponding to the current image frame. The second determining module is specifically used to generate an initial mapping curve for each local region based on the target scene category of each local region; Obtain multiple target scene categories in the current image frame; Determine the number of regions corresponding to each target scene category in the local area; The target scene category with the largest number of regions is determined as the first scene category; If the number of regions in the first scene category is greater than the first threshold and multiple preset conditions are met, then the current image frame is of OSD type. The multiple preset conditions include: the difference between the scene number of the first scene category and the scene number of the second scene category is less than a second threshold, the second scene category is the scene category with the fewest regions; the first scene category is an OSD scene category; the current image frame and the historical image frame switch scenes, and the historical image frame is a non-OSD type; the saturation of the current image frame is less than a third threshold. If so, the initial mapping curve of each local region is filtered to obtain the target mapping curve of each local region; If not, then the initial mapping curve of each local region is determined as the target mapping curve of each local region; For any local region; when filtering the initial mapping curve of the local region to obtain the target mapping curve of the local region, the second determining module is specifically used to determine a first mapping curve and a second mapping curve from the initial mapping curve, wherein the first mapping curve is the mapping curve corresponding to the first scene category, and the second mapping curve is the mapping curve corresponding to the second scene category; Based on the first mapping curve, the second mapping curve, and the filtering parameters, determine the adjustment mapping curve corresponding to the second mapping curve; The sum of the second mapping curve and the adjusted mapping curve is determined as the third mapping curve for the second scene category; The second mapping curve in the initial mapping curve is replaced with the third mapping curve to obtain the target mapping curve.

7. An electronic device, characterized in that, include: Processor, memory; The memory stores computer programs; The processor invokes the computer program stored in the memory, causing the electronic device to execute the image frame enhancement method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image frame enhancement method according to any one of claims 1 to 5.

9. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the image frame enhancement method according to any one of claims 1 to 5.

Images