AWB calibration method, device, electronic device and storage medium based on semantic segmentation

Abstract

The present application relates to the field of digital imaging, and in particular to an AWB calibration method, device, electronic device and storage medium based on semantic segmentation. The present application obtains multiple scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks, and determines the block images corresponding to the scene image; based on the block images, gray areas and non-gray areas of each scene image, determines the gray area blocks, non-gray areas and gray area binary images and pixel ratio coordinates; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variables through a preset cost function; based on the AWB calibration variables, determines the AWB calibration frame, and the AWB calibration frame is used for AWB calibration. The present application uses a method combining semantic segmentation with a cost function, and uses the AWB calibration frame as the final calibration frame or the adjusted initialization frame. By automating the AWB calibration and initialization process, the debugging time and process are reduced.

Description

Technical Field

[0001] The present application relates to the field of digital imaging, and in particular to an AWB calibration method, device, electronic device, and storage medium based on semantic segmentation. Background Art

[0002] The Auto White Balance (AWB) function of the Image Signal Processor (ISP) can automatically adjust the color balance of the image according to the color temperature of the light source, so that white objects appear white under different light sources, thereby maintaining the natural color of the image.

[0003] AWB is typically implemented using the Planck curve, which characterizes the sensor's perception of different wavelengths of light. Combined with the pixel values in grayscale areas of the scene, this curve can be used to restore realistic colors. The typical AWB debugging method involves inputting a color temperature chart and adjusting the frame. Once a color temperature is adjusted, it's necessary to review the adjustment to ensure that it meets the requirements for other color temperatures, which is quite time-consuming.

[0004] After searching the prior art, it was found that Chinese invention patent CN106358031B discloses an adaptive adjustment algorithm for the Planck curve. The algorithm uses an adjusted Planck curve table to achieve adaptive adjustment of the high color temperature part and the low color temperature part. However, it does not disclose how to quickly and accurately obtain the Planck curve.

[0005] In addition, Chinese invention patent CN117835076A discloses a method for finding a white point area. This patent obtains multiple color temperature curves by fitting the gray points of each color temperature R / G and B / G on the XoY plane. This method can more accurately reflect the sensor's response to the light source's hue. Based on the multiple color temperature curves, R / G value sets, and B / G value sets, the initial white point area can be scaled to obtain an accurate white point area, allowing the white area to be applied to the AWB algorithm, thereby improving the AWB algorithm's effectiveness. However, this method is accomplished through multi-color temperature curve fitting, resulting in a relatively complex calculation process and a non-mainstream approach.

[0006] Therefore, how to simplify the AWB calibration process and debugging time and improve the AWB calibration efficiency has become an urgent problem to be solved. Summary of the Invention

[0007] The present application provides an AWB calibration method, device, electronic device, and storage medium based on semantic segmentation, which simplifies the AWB calibration process and debugging time and improves the AWB calibration efficiency.

[0008] In a first aspect, an embodiment of the present application provides an AWB calibration method based on semantic segmentation, the method comprising: acquiring a plurality of scene images, each of the scene images corresponding to a different color temperature; determining the gray area and non-gray area corresponding to each of the scene images by semantic segmentation for any scene image in the scene images; dividing any scene image in the scene images into blocks to determine the block images corresponding to each of the scene images; determining the gray block, non-gray block and gray area binary image corresponding to each of the scene images based on the block images, gray areas and non-gray areas corresponding to each of the scene images, wherein the gray block is any block of the block image The gray area of each scene image is greater than a block with a preset ratio, and the gray area binary image is a binary image composed of all the gray blocks; based on the gray blocks, non-gray blocks and gray area binary images corresponding to each scene image, the pixel ratio coordinates of each scene image in the same coordinate system are determined; based on the pixel ratio coordinates of each scene image in the same coordinate system and the color temperature corresponding to each scene image, an AWB calibration variable is determined through a preset cost function; based on the AWB calibration variable, an AWB calibration frame is determined, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each scene image.

[0009] In a possible implementation, for any scene image among the scene images, before determining the gray area and non-gray area corresponding to each of the scene images through semantic segmentation, a semantic segmentation model is constructed, and the method for constructing the semantic segmentation model is: obtaining a color temperature scene image set, wherein the color temperature scene image set includes fixed scene images of multiple color temperatures; labeling the color temperature scene image set to determine the gray area and non-gray area of each of the fixed scene images; and determining the semantic segmentation model based on the gray area and non-gray area of each of the fixed scene images and a preset model.

[0010] In a possible implementation, the AWB calibration variables include a spacing coefficient between a first AWB calibration subframe and a second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe, and a vertical coordinate of the second AWB calibration subframe, where the first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame; when the spacing coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0, 1], the right portion of the first AWB calibration subframe overlaps with the second AWB calibration subframe; and determining the AWB calibration frame based on the AWB calibration variables. The method includes: determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; determining the horizontal coordinate of the second AWB calibration subframe based on the horizontal coordinate and frame width of the first AWB calibration subframe; determining the first AWB calibration subframe based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe; determining the second AWB calibration subframe based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe; and determining the AWB calibration frame based on the first AWB calibration subframe and the second AWB calibration subframe.

[0011] In a possible implementation, when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [1, 2], the bottom of the first AWB calibration subframe coincides with the bottom of the second AWB calibration subframe, and determining the AWB calibration frame based on the AWB calibration variable includes: determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate and frame height of the second AWB calibration subframe; determining the horizontal coordinate of the first AWB calibration subframe based on the horizontal coordinate, the interval difference, and the frame width of the first AWB calibration subframe. The horizontal coordinate of the frame, the interval difference is the difference between a preset constant and an interval coefficient, the interval coefficient is the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the value of the preset constant is 2; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe, the first AWB calibration subframe is determined; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe, the second AWB calibration subframe is determined; based on the first AWB calibration subframe and the second AWB calibration subframe, the AWB calibration frame is determined.

[0012] In a possible implementation, the preset cost function is:

[0013] cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou

[0014] Wherein, sum_ct_diff represents the sum of the distances between the color temperature corresponding to each of the AWB calibration subframes and the target color temperature, valid_iou represents the sum of the intersection-and-union ratios of the gray blocks in each of the AWB calibration subframes, invalid_iou represents the sum of the intersection-and-union ratios of the non-gray blocks in each of the AWB calibration subframes, and k0, k1, and k2 represent weights.

[0015] In a possible implementation, the sum of the intersection-over-union ratios of the gray block in each of the AWB calibration subframes is:

[0016]

[0017] Wherein, sum_GrayRect represents the number of gray blocks in each of the AWB calibration subframes, and sum_Gray represents the number of gray blocks in each of the block images.

[0018] In a possible implementation, the sum of the intersection-and-union ratios of the non-gray block in each of the AWB calibration subframes is:

[0019]

[0020] Wherein, sum_NoGrayRect represents the number of non-gray blocks in each of the AWB calibration subframes, and sum_NoGray represents the number of non-gray blocks in each of the block images.

[0021] In a second aspect, an embodiment of the present application provides an AWB calibration device based on semantic segmentation, the device comprising: an acquisition module for acquiring a plurality of scene images, each of the scene images corresponding to a different color temperature; a determination module for determining, for any scene image in the scene images, the gray area and non-gray area corresponding to each of the scene images through semantic segmentation; the determination module is further used to block any scene image in the scene images and determine the block images corresponding to each of the scene images; the determination module determines the gray block, non-gray block and gray area binary image corresponding to each of the scene images based on the block images, gray areas and non-gray areas corresponding to each of the scene images, the gray block being any block within the block image. , a block with a gray area greater than a preset ratio, the gray area binary image is a binary image composed of all the gray areas; an extraction module is used to determine the pixel ratio coordinates of each scene image in the same coordinate system based on the gray areas, non-gray areas and gray area binary images corresponding to each scene image; the determination module is further used to determine the AWB calibration variable based on the pixel ratio coordinates of each scene image in the same coordinate system and the color temperature corresponding to each scene image through a preset cost function; the determination module is further used to determine the AWB calibration frame based on the AWB calibration variable, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each scene image.

[0022] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements the method described in the first aspect or any one of the implementation methods thereof.

[0024] The present application obtains multiple scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks to determine the block image corresponding to each scene image; based on the block image, gray area and non-gray area of each scene image, determines the gray block, non-gray block and gray area binary image, the gray block is a block in any block of the block image with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all gray blocks; based on the gray block, non-gray block and gray area binary image corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variable through a preset cost function; based on the AWB calibration variable, determines the AWB calibration frame, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes an AWB calibration sub-frame corresponding to each of the scene images. Compared to the prior art method of inputting a color temperature map and adjusting the calibration frame once, once a color temperature is adjusted, it is necessary to check whether it meets the requirements of other color temperatures, which is very time-consuming. This application uses a method that combines semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final calibration frame or the initialization frame for adjustment. The more color temperatures input, the more accurate the AWB calibration frame obtained. In short, by automating the AWB calibration and initialization process, the debugging time and process are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0026] Figure 1 A flowchart of an AWB calibration method based on semantic segmentation provided in one embodiment of the present application;

[0027] Figure 2 The scene image collected at a color temperature of 5600K provided by this application;

[0028] Figure 3 The non-gray area image of the scene image collected at a color temperature of 5600K provided in this application after being differentiated by the semantic segmentation model;

[0029] Figure 4 The gray area binary image after the scene image provided in this application is divided into blocks;

[0030] Figure 5 Overlay of the gray area binary image provided for this application;

[0031] Figure 6A schematic diagram of any two adjacent AWB calibration subframes provided in this application;

[0032] Figure 7 Another schematic diagram of any two adjacent AWB calibration subframes provided by this application;

[0033] Figure 8 Schematic diagram of the AWB calibration frame corresponding to the 10 scene images provided in this application;

[0034] Figure 9 This is a structural block diagram of an AWB calibration device based on semantic segmentation provided in one embodiment of the present application;

[0035] Figure 10 This is a schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0036] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0037] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0038] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0039] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0040] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0041] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0042] The ISP pipeline is used to convert the digital quantity output by the imaging chip into a digital quantity that can be displayed on the screen. A good ISP pipeline can make the final displayed image have a sense of reality comparable to that of the human eye. The role of the AWB automatic white balance is to ensure that the true and objective colors are restored under different color temperatures. However, in order to simulate the human eye, the photosensitive chip is generally more sensitive to green than red and blue. If the white scene is displayed under normal fluorescent light, it will be green when displayed by the camera. If it is displayed under other color temperatures, such as sunset (low color temperature), the white will turn red. If it is displayed by a camera, the color cast will be more serious. Therefore, the goal of AWB is to ensure that white still appears white through the camera in this scene.

[0043] The method for implementing AWB is generally to use the Planck curve, which characterizes the sensor's perception of light of different wavelengths. Combined with the imaging pixel values in the gray area of the scene, it can restore more realistic colors. This is called AWB calibration. The general implementation process is:

[0044] 1. Obtain the (R / G, B / G) values in the gray area at different color temperatures (generally 4 to 5 color temperatures are used);

[0045] 2. Use R / G as the value of the horizontal coordinate x and B / G as the value of the vertical coordinate y, and draw all the points on the coordinate plane XoY (where o is the coordinate origin);

[0046] 3. Use multiple straight lines / curves or regions to enclose all the points as closely as possible;

[0047] 4. Mark the range of different color temperatures (an area on the XoY).

[0048] The most difficult part of the above implementation process is to obtain the gray area in the image and an area that can better enclose the points. The method proposed in this application can automate this process and achieve better results.

[0049] Figure 1A flowchart of an AWB calibration method based on semantic segmentation is provided in one embodiment of the present application. The method includes:

[0050] S110 , acquiring a plurality of scene images, each scene image corresponding to a different color temperature.

[0051] As an example, multiple scene images are collected at multiple color temperatures. Specifically, 10 scene images can be collected in the color temperature range of 1000K and 15000K, where the first scene image is marked as 0 and the color temperature is 1000K, and the tenth scene image is marked as 9 and the color temperature is 15000K.

[0052] S120 , determining the gray area and the non-gray area corresponding to each scene image by semantic segmentation for any scene image in the scene images.

[0053] For example, for the 10 collected scene images, the gray area and non-gray area of each scene image are determined respectively by a gray area binary classification semantic segmentation model.

[0054] In one possible implementation, for any scene image in the scene images, before determining the gray area and non-gray area corresponding to each scene image through semantic segmentation, a semantic segmentation model needs to be constructed. The method for constructing the semantic segmentation model is: obtaining a color temperature scene image set, the color temperature scene image set including fixed scene images of multiple color temperatures; labeling the color temperature scene image set to determine the gray area and non-gray area of each fixed scene image; and determining the semantic segmentation model based on the gray area and non-gray area of each fixed scene image and a preset model.

[0055] For example, the color representation of the scene image in S110 obtained by different color temperatures and sensors may be significantly different. Therefore, when building a semantic segmentation model, it is necessary to scramble the color data to improve the model's robustness to different color changes.

[0056] To achieve this goal, the present application obtains fixed scene images containing multiple color temperatures so that the model can learn the impact of these color temperature changes on the image color. In order to reduce the network inference time (that is, the time required for the model to process the image and output the result), the present application uses a smaller network architecture and simply identifies the scene so that the model can quickly and accurately identify it. Furthermore, the color temperature scene image set is annotated to determine the gray area and non-gray area of each fixed scene image. Annotating the color temperature scene image set can provide accurate supervision information for the preset model to guide the training of the model. Furthermore, based on the gray area and non-gray area of each fixed scene image and the preset model, a semantic segmentation model is determined. Specifically, the preset model can be a fully convolutional network model (FCN) or a DeepLabv3+ model. Among them, the semantic segmentation model is specifically used for gray area binary classification. The semantic segmentation model will learn how to distinguish gray areas from non-gray areas based on image color, texture and other features. For example, the color temperature scene image such as Figure 2 As shown, the non-gray area image after being distinguished by the semantic segmentation model is as follows Figure 3 shown.

[0057] S130 , dividing any scene image in the scene images into blocks, and determining block images corresponding to each scene image.

[0058] For example, before performing AWB calibration, the input scene image needs to be segmented. For example, a 1280*720 image can be evenly divided into 64*36 blocks, each 20*20 in size. This segmentation process is intended to break the image into smaller, more manageable regions, allowing for separate analysis of the color characteristics of each region. The block size is manually predetermined and can be adjusted based on the specific needs and effects of the algorithm. This application does not impose any restrictions on this.

[0059] S140, based on the block images, gray areas and non-gray areas corresponding to each scene image, determine the gray blocks, non-gray blocks and gray area binary images corresponding to each scene image, where a gray block is a block in any block of the block image with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all gray blocks.

[0060] Exemplarily, based on each small block of the divided image, the proportion of gray area pixels therein is calculated to determine whether the small block is a gray area block. This proportion is obtained by dividing the number of all gray area pixels in the small block by the total number of pixels in the small block. For example: set a preset ratio of 70%. If the proportion of gray area in a small block is greater than or equal to this threshold, the small block is considered to be a gray area. Extract the binary image of all gray area blocks and determine a mask containing only gray area blocks, that is, a gray area binary image. For example: Figure 4This is the gray area binary image after block processing provided by this application. Figure 5 It is an overlay image of the gray area binary image, where green represents the gray area.

[0061] S150 , determining pixel ratio coordinates of each scene image in the same coordinate system based on the gray blocks, non-gray blocks, and gray binary images corresponding to each scene image.

[0062] For example, multiple scene images at different color temperatures are obtained to obtain gray blocks, non-gray blocks, and gray binary images corresponding to the scene images at different color temperatures. The (R / G, B / G) coordinates of the gray blocks and non-gray blocks on the gray binary images are extracted, i.e., the pixel ratios of red to green, and blue to green in the gray blocks and non-gray blocks, respectively. These (R / G, B / G) values are then plotted on a two-dimensional XoY space, where the X-axis represents R / G and the Y-axis represents B / G.

[0063] S160 , determining an AWB calibration variable based on the pixel ratio coordinates of each scene image in the same coordinate system and the color temperature corresponding to each scene image using a preset cost function.

[0064] Exemplarily, step S110 collects 10 scene images as M (m0, m2, ..., m9), whose corresponding target color temperature set is CT (ct0, ct2, ..., ct9), and the function for calculating the color temperature of the scene image is g, then the estimated color temperature is est_ctN = g(mN). In this embodiment, N∈[0, 9].

[0065] Furthermore, in the following calculation process of the present application, the calculation of the intersection-over-union ratio requires the pixel ratio coordinates of each scene image in the same coordinate system.

[0066] Specifically, at a certain color temperature, the estimated color temperature difference and the target color temperature difference are:

[0067] ct_diffN=wN×|est_ctN-ctN|

[0068] Then, the sum of the distances between the color temperature corresponding to each AWB calibration subframe and the target color temperature is:

[0069]

[0070] Wherein, in the embodiment listed, N ranges from [0,9], represents the identification of the scene image, ct_diffN represents the distance between the color temperature corresponding to each AWB calibration subframe and the target color temperature, and w N Represents weight.

[0071] In one possible implementation, the sum of the intersection-over-union ratios of the gray blocks in each AWB calibration subframe is:

[0072]

[0073] Wherein, sum_GrayRect represents the number of gray blocks in each AWB calibration subframe, and sum_Gray represents the number of gray blocks in each block image.

[0074] As another example, the traditional IoU calculation formula may be used to determine the sum of the IoUs of the gray block in each AWB calibration subframe. The traditional IoU calculation formula is:

[0075]

[0076] The calculation formula for calculating the sum of the intersection-over-union ratios of gray blocks in each AWB calibration subframe proposed in this application is not restricted by the shape and size of the frame set itself, has fewer constraints, and is conducive to the stability of optimization.

[0077] In a possible implementation, the sum of the intersection-over-union ratios of the non-gray blocks in each AWB calibration subframe is:

[0078]

[0079] Wherein, sum_NoGrayRect represents the number of non-gray blocks in each AWB calibration subframe, and sum_NoGray represents the number of non-gray blocks in each block image.

[0080] As another example, the traditional IoU calculation formula may be used to determine the sum of the IoUs of the non-gray blocks in each AWB calibration subframe. The traditional IoU calculation formula is:

[0081]

[0082] The calculation formula for calculating the sum of the intersection-over-union ratios of the non-gray blocks in each AWB calibration sub-frame proposed in this application is not restricted by the shape and size of the frame set itself, has fewer constraints, and is conducive to the stability of optimization.

[0083] In a possible implementation, the final preset cost function is determined according to the above formula:

[0084] cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou

[0085] Where sum_ct_diff represents the sum of the distances between the color temperature corresponding to each AWB calibration subframe and the target color temperature, valid_iou represents the sum of the intersection-and-union ratios of the gray blocks in each AWB calibration subframe, invalid_iou represents the sum of the intersection-and-union ratios of the non-gray blocks in each AWB calibration subframe, k0, k1, and k2 represent weights, and k2 can generally be set to 2×k1=100×k0.

[0086] Furthermore, the optimized AWB calibration variables are determined by minimizing the cost function cost_func. When 10 scene images are collected, 31 variables need to be optimized, namely [x0, y9, w0, h0] + [a1, w1, h1] + … + [a9, w9, h9], where x0 represents the horizontal coordinate of the first scene image, y9 represents the vertical coordinate of the tenth scene image, w0-w9 represent the width of the AWB calibration sub-frame corresponding to each scene image, h0-h9 represent the height of the AWB calibration sub-frame corresponding to each scene image, and a1-a9 represent the interval coefficients corresponding to the scene image and the previous scene image.

[0087] Specifically, the Nelder-Mead optimization algorithm can be used to optimize the preset cost function, or other optimization algorithms with range constraints, such as gradient descent, genetic algorithm, etc., can be used, which is not limited in this application.

[0088] S170 , determining an AWB calibration frame based on the AWB calibration variable, where the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each scene image.

[0089] In a possible implementation, the AWB calibration variables include a spacing coefficient between the first AWB calibration subframe and the second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe, and a vertical coordinate of the second AWB calibration subframe. The first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame. When the spacing coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0,1], the right part of the first AWB calibration subframe coincides with the second AWB calibration subframe. Based on the AWB calibration variables, determine A The WB calibration frame includes: determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; determining the horizontal coordinate of the second AWB calibration subframe based on the horizontal coordinate and frame width of the first AWB calibration subframe; determining the first AWB calibration subframe based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe; determining the second AWB calibration subframe based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe; and determining the AWB calibration frame based on the first AWB calibration subframe and the second AWB calibration subframe.

[0090] For example, for 10 scene images, after 31 AWB calibration variables are obtained through a preset cost function, it is necessary to obtain the AWB calibration sub-frame corresponding to each scene image based on the 31 calibration variables.

[0091] Specifically, when the interval coefficient of two adjacent calibration subframes is in [0,1], such as Figure 6 As shown, the right side of the first AWB calibration subframe overlaps with the second AWB calibration subframe. Assuming that calibration subframe 0 corresponds to the first scene image and calibration subframe 1 corresponds to the second scene image, the AWB calibration variables x0, y1, w0, h0, a1, w1, and h1 are now known, where x0 is the horizontal coordinate of calibration subframe 0, y1 is the vertical coordinate of calibration subframe 1, w0 and w1 are the widths of calibration subframe 0 and calibration subframe 1, respectively, h0 and h1 are the heights of calibration subframe 0 and calibration subframe 1, respectively, and a1 is the spacing coefficient between calibration subframe 0 and calibration subframe 1. Then y0 = y1 + a1 × h1 and x1 = x0 + w0. From this, the vertical coordinate of calibration subframe 0 and the horizontal coordinate of calibration subframe 1 can be calculated.

[0092] In another possible implementation, when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [1, 2], the bottom of the first AWB calibration subframe coincides with the second AWB calibration subframe, and the AWB calibration frame is determined based on the AWB calibration variable, including: determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate and frame height of the second AWB calibration subframe; determining the horizontal coordinate of the second AWB calibration subframe based on the horizontal coordinate, the interval difference and the frame width of the first AWB calibration subframe. The horizontal coordinate of the subframe, the interval difference is the difference between the preset constant and the interval coefficient, the interval coefficient is the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the preset constant is 2; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe, the first AWB calibration subframe is determined; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe, the second AWB calibration subframe is determined; based on the first AWB calibration subframe and the second AWB calibration subframe, the AWB calibration frame is determined.

[0093] Specifically, when the interval coefficient of two adjacent calibration subframes is [1, 2], such as Figure 7 As shown, the bottom of the first AWB calibration subframe coincides with the bottom of the second AWB calibration subframe. Assume that calibration subframe No. 0 corresponds to the first scene image and calibration subframe No. 1 corresponds to the second scene image. Now the AWB calibration variables x0, y1, w0, h0, a1, w1, h1 are known, where x0 is the horizontal coordinate of calibration subframe No. 0, y1 is the vertical coordinate of calibration subframe No. 1, w0 and w1 are the widths of calibration subframe No. 0 and calibration subframe No. 1 respectively, h0 and h1 are the heights of calibration subframe No. 0 and calibration subframe No. 1 respectively, and a1 is the interval coefficient between calibration subframe No. 0 and calibration subframe No. 1. Then y0 = y1 + h1, x1 = x0 + (2-a1) × w0. From this, the vertical coordinate of calibration subframe No. 0 and the horizontal coordinate of calibration subframe No. 1 can be calculated. The reason why the interval coefficient is divided into [0,1] and [1,2] is to ensure that the interval coefficient (in Figure 4 and Figure 5 When a1 is equal to 1, the results obtained by overlapping the right and bottom axes are consistent, ensuring the continuity of the change of variable a1. Furthermore, the initial setting of x0 and y1 is because they are both variables greater than 0. Subsequent relationships are derived through addition, thus avoiding the problem of negative numbers resulting from subtraction, which can hinder optimization convergence.

[0094] It should be noted that, in the above process of finding y0 and x1 through x0, y1, w0, h0, a1, w1, h1, for ease of understanding, only two AWB calibration sub-frames are used as examples. In the example provided in this application, in the process of finding the corresponding AWB calibration sub-frames through 10 scene images, the 31 known calibration variables required are [x0, y9, w0, h0] + [a1, w1, h1] + … + [a9, w9, h9], and according to the range of a1-a9 ([0,1] or [1,2]), x1 is found through x0, x2 is found through x1, …, x9 is found through x8, and y8 is found through y9, y7 is found through y8, …, y0 is found through y1. Then, the AWB calibration frame is determined based on the results and the 31 AWB calibration variables, as shown in the following example. Figure 8 shown.

[0095] The present application obtains multiple scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks to determine the block image corresponding to each scene image; based on the block image, gray area and non-gray area of each scene image, determines the gray block, non-gray block and gray area binary image, the gray block is a block in any block of the block image with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all gray blocks; based on the gray block, non-gray block and gray area binary image corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variable through a preset cost function; based on the AWB calibration variable, determines the AWB calibration frame, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes an AWB calibration sub-frame corresponding to each of the scene images. This application uses a method that combines semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final result frame or the adjusted initialization frame. By automating the AWB calibration and initialization process, the debugging time and process are reduced. In addition, this application can input scene images of multiple color temperatures according to user needs. The more color temperatures input, the more accurate the AWB calibration frame obtained.

[0096] Figure 9 This is a structural block diagram of an AWB calibration device based on semantic segmentation provided by an embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown. Figure 9 The AWB calibration device 900 based on semantic segmentation includes an acquisition module 901 , a determination module 902 , and an extraction module 903 .

[0097] In one implementation, the apparatus 900 may be used to implement the above Figure 1For example, the acquisition module 901 is used to implement S110, the determination module 902 is used to implement S120 to S140, S160 to S170, and the extraction module 903 is used to implement S150.

[0098] The present application obtains multiple scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks to determine the block image corresponding to each scene image; based on the block image, gray area and non-gray area of each scene image, determines the gray block, non-gray block and gray area binary image, the gray block is a block in any block of the block image with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all gray blocks; based on the gray block, non-gray block and gray area binary image corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variable through a preset cost function; based on the AWB calibration variable, determines the AWB calibration frame, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes an AWB calibration sub-frame corresponding to each of the scene images. This application uses a method that combines semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final result frame or the adjusted initialization frame. By automating the AWB calibration and initialization process, the debugging time and process are reduced. In addition, this application can input scene images of multiple color temperatures according to user needs. The more color temperatures input, the more accurate the AWB calibration frame obtained.

[0099] Figure 10 This is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. Figure 10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Figure 10 Only one is shown in the figure) a processor, a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100, wherein the processor 100 implements the steps of any of the above method embodiments when executing the computer program 102.

[0100] The electronic device 10 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will understand that Figure 10 This is merely an example of the electronic device 10 and does not constitute a limitation on the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0101] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0102] In some embodiments, the memory 101 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. In other embodiments, the memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 10. Furthermore, the memory 101 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0104] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0105] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0106] An embodiment of the present application provides a computer program product. When the computer program product is run on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned various method embodiments when executing the computer program product.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk.

[0108] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0109] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0110] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0112] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

[0113] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An AWB calibration method based on semantic segmentation, characterized in that: The method comprises: Acquire multiple scene images, each of the scene images corresponding to a different color temperature; For any scene image among the scene images, determining a gray area and a non-gray area corresponding to each scene image through semantic segmentation; Divide any scene image among the scene images into blocks, and determine the block images corresponding to each scene image; Based on the block images, gray areas, and non-gray areas corresponding to each of the scene images, determining the gray blocks, non-gray blocks, and gray area binary images corresponding to each of the scene images, wherein the gray blocks are blocks within any block of the block images with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all the gray blocks; Determining pixel ratio coordinates of each scene image in the same coordinate system based on the gray blocks, non-gray blocks, and gray area binary images corresponding to each scene image; determining an AWB calibration variable based on a preset cost function based on pixel ratio coordinates of each of the scene images in the same coordinate system and a color temperature corresponding to each of the scene images; Determining an AWB calibration frame based on the AWB calibration variables, where the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration subframes corresponding to each of the scene images; The preset cost function is: cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou Wherein, sum_ct_diff represents the sum of the distances between the color temperature corresponding to each of the AWB calibration subframes and the target color temperature, valid_iou represents the sum of the intersection-and-union ratios of the gray blocks in each of the AWB calibration subframes, invalid_iou represents the sum of the intersection-and-union ratios of the non-gray blocks in each of the AWB calibration subframes, and k0, k1, and k2 represent weights. The sum of the intersection-over-combination ratios of the gray block in each of the AWB calibration subframes is: ； Wherein, sum_GrayRect represents the number of gray blocks in each of the AWB calibration subframes, sum_Gray represents the number of gray blocks in each of the block images, and N represents the identifier of the scene image; The sum of the intersection-over-combination ratios of the non-gray blocks in each of the AWB calibration subframes is: ； Wherein, sum_NoGrayRect represents the number of non-gray blocks in each of the AWB calibration subframes, and sum_NoGray represents the number of non-gray blocks in each of the block images.

2. The method according to claim 1, characterized in that For any scene image in the scene images, before determining the gray area and the non-gray area corresponding to each scene image through semantic segmentation, a semantic segmentation model is constructed. The method for constructing the semantic segmentation model is: Acquire a color temperature scene image set, wherein the color temperature scene image set includes fixed scene images of multiple color temperatures; Annotating the color temperature scene image set to determine the gray area and non-gray area of each fixed scene image; The semantic segmentation model is determined based on the gray area and the non-gray area of each of the fixed scene images and a preset model.

3. The method according to claim 1, characterized in that The AWB calibration variables include a spacing coefficient between the first AWB calibration subframe and the second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe, and a vertical coordinate of the second AWB calibration subframe, where the first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame; When the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0, 1], the right portion of the first AWB calibration subframe overlaps with the second AWB calibration subframe; The determining of the AWB calibration frame based on the AWB calibration variable includes: Determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; Determining a horizontal coordinate of the second AWB calibration subframe based on the horizontal coordinate and the frame width of the first AWB calibration subframe; Determine the first AWB calibration subframe based on the abscissa, ordinate, width, and height of the first AWB calibration subframe; Determine the second AWB calibration subframe based on the abscissa, ordinate, width, and height of the second AWB calibration subframe; The AWB calibration frame is determined based on the first AWB calibration subframe and the second AWB calibration subframe.

4. The method according to claim 3, characterized in that When the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is [1, 2], the bottom of the first AWB calibration subframe overlaps with the second AWB calibration subframe, and determining the AWB calibration frame based on the AWB calibration variable includes: Determining the vertical coordinate of the first AWB calibration subframe based on the vertical coordinate and the frame height of the second AWB calibration subframe; Determining the abscissa of the second AWB calibration subframe based on the abscissa of the first AWB calibration subframe, the interval difference, and the width of the first AWB calibration subframe, where the interval difference is a difference between a preset constant and an interval coefficient, the interval coefficient is the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the preset constant is 2; Determine the first AWB calibration subframe based on the abscissa, ordinate, width, and height of the first AWB calibration subframe; Determine the second AWB calibration subframe based on the abscissa, ordinate, width, and height of the second AWB calibration subframe; The AWB calibration frame is determined based on the first AWB calibration subframe and the second AWB calibration subframe.

5. An AWB calibration device based on semantic segmentation, characterized in that: The device comprises: An acquisition module is configured to acquire a plurality of scene images, each of the scene images corresponding to a different color temperature; A determination module, for determining, by semantic segmentation, a gray area and a non-gray area corresponding to each scene image in the scene images; The determining module is further configured to divide any scene image in the scene images into blocks and determine the block images corresponding to each scene image; The determining module determines the gray block, non-gray block, and gray area binary image corresponding to each scene image based on the block images, gray areas, and non-gray areas corresponding to each scene image, wherein the gray block is a block with a gray area greater than a preset ratio within any block of the block image, and the gray area binary image is a binary image composed of all the gray blocks; an extraction module, configured to determine pixel ratio coordinates of each of the scene images in the same coordinate system based on the gray blocks, non-gray blocks, and gray binary images corresponding to each of the scene images; The determination module is further configured to determine an AWB calibration variable using a preset cost function based on the pixel ratio coordinates of each of the scene images in the same coordinate system and the color temperature corresponding to each of the scene images; The determining module is further configured to determine an AWB calibration frame based on the AWB calibration variable, wherein the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration subframes corresponding to each of the scene images; The preset cost function is: cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou Wherein, sum_ct_diff represents the sum of the distances between the color temperature corresponding to each of the AWB calibration subframes and the target color temperature, valid_iou represents the sum of the intersection-and-union ratios of the gray blocks in each of the AWB calibration subframes, invalid_iou represents the sum of the intersection-and-union ratios of the non-gray blocks in each of the AWB calibration subframes, and k0, k1, and k2 represent weights. The sum of the intersection-over-combination ratios of the gray block in each of the AWB calibration subframes is: ； Wherein, sum_GrayRect represents the number of gray blocks in each of the AWB calibration subframes, sum_Gray represents the number of gray blocks in each of the block images, and N represents the identifier of the scene image; The sum of the intersection-over-combination ratios of the non-gray blocks in each of the AWB calibration subframes is: ； Wherein, sum_NoGrayRect represents the number of non-gray blocks in each of the AWB calibration subframes, and sum_NoGray represents the number of non-gray blocks in each of the block images.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images