Semantic segmentation-based AWB calibration method and device, electronic equipment and storage medium

Abstract

The invention relates to the field of digital imaging, in particular to an AWB calibration method and device based on semantic segmentation, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of scene images under different color temperatures; based on each scene image, determining a gray area and a non-gray area corresponding to each scene image through semantic segmentation; blocking each scene image, and determining block images corresponding to the scene images; determining binary images of gray blocks, non-gray blocks and gray areas and pixel ratio coordinates based on the block images, the gray areas and the non-gray areas of the scene images; determining an AWB calibration variable through a preset cost function based on the pixel ratio coordinate and the color temperature corresponding to each scene image; based on the AWB calibration variable, an AWB calibration frame is determined, and the AWB calibration frame is used for AWB calibration. According to the method, a semantic segmentation and cost function combined method is used, the AWB calibration frame is used as a final calibration frame or an adjusted initialization frame, and the debugging time and process are reduced through the automatic AWB calibration and initialization process.

Description

Technical Field

[0001] The present application relates to the field of digital imaging, and in particular to an AWB calibration method, device, electronic device and storage medium based on semantic segmentation. Background Art

[0002] The Auto White Balance (AWB) function of the Image Signal Processor (ISP) can automatically adjust the color balance of the image according to the color temperature of the light source, so that white objects can appear white under different light sources, thereby maintaining the natural color of the image.

[0003] The method to implement AWB is generally to use the Planck curve, which characterizes the sensor's perception of light of different wavelengths. Combined with the imaging pixel value of the gray area in the scene, it can restore more realistic colors. The general AWB debugging method is to input a color temperature chart and adjust the frame once. If a certain color temperature is adjusted, it is necessary to check whether it meets other color temperatures, which is quite time-consuming.

[0004] After searching the prior art, it was found that Chinese invention patent CN106358031B discloses an adaptive adjustment algorithm of the Planck curve, which uses the adjusted Planck curve table to achieve adaptive adjustment of the high color temperature part and the low color temperature part, but does not disclose how to quickly and accurately obtain the Planck curve.

[0005] In addition, Chinese invention patent CN117835076A discloses a method for finding a white point area. In this patent, multiple color temperature curves are obtained by fitting the gray points of each color temperature R / G, B / G on the XoY plane, which can more accurately reflect the sensor's response to the light source hue. According to multiple color temperature curves, R / G value sets and B / G value sets, the initial white point area can be scaled to obtain an accurate white point area, so that the white area can be applied to the AWB algorithm, thereby improving the effect of the AWB algorithm. However, this method is completed by fitting multiple color temperature curves, and the calculation process is relatively complex, which is a non-mainstream method.

[0006] Therefore, how to simplify the calibration process and debugging time of AWB and improve the calibration efficiency of AWB has become an urgent problem to be solved. Summary of the invention

[0007] The present application provides an AWB calibration method, device, electronic device and storage medium based on semantic segmentation, which simplifies the AWB calibration process and debugging time and improves the AWB calibration efficiency.

[0008] In a first aspect, an embodiment of the present application provides an AWB calibration method based on semantic segmentation, the method comprising: acquiring multiple scene images, each of the scene images corresponding to a different color temperature; for any scene image in the scene images, determining the gray area and non-gray area corresponding to each of the scene images through semantic segmentation; for any scene image in the scene images, dividing the scene images into blocks to determine the block images corresponding to each of the scene images; based on the block images, gray areas and non-gray areas corresponding to each of the scene images, determining the gray area blocks, non-gray area blocks and gray area binary images corresponding to each of the scene images, the gray area blocks being any block of the block images The gray area is greater than the blocks of the preset ratio, and the gray area binary image is a binary image composed of all the gray area blocks; based on the gray area blocks, non-gray area blocks and gray area binary images corresponding to each of the scene images, the pixel ratio coordinates of each of the scene images in the same coordinate system are determined; based on the pixel ratio coordinates of each of the scene images in the same coordinate system and the color temperature corresponding to each of the scene images, the AWB calibration variables are determined through a preset cost function; based on the AWB calibration variables, an AWB calibration frame is determined, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each of the scene images.

[0009] In a possible implementation, for any scene image among the scene images, before determining the gray area and non-gray area corresponding to each of the scene images through semantic segmentation, a semantic segmentation model is constructed, and the method for constructing the semantic segmentation model is: obtaining a color temperature scene image set, the color temperature scene image set including fixed scene images of multiple color temperatures; annotating the color temperature scene image set to determine the gray area and non-gray area of ​​each of the fixed scene images; and determining the semantic segmentation model based on the gray area and non-gray area of ​​each of the fixed scene images and a preset model.

[0010] In a possible implementation, the AWB calibration variable includes an interval coefficient between a first AWB calibration subframe and a second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe, and a vertical coordinate of the second AWB calibration subframe, wherein the first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame; when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0,1], the right part of the first AWB calibration subframe overlaps with the second AWB calibration subframe; and based on the AWB calibration variable, determining the AWB calibration frame, The method comprises: determining the ordinate of the first AWB calibration subframe based on the ordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; determining the abscissa of the second AWB calibration subframe based on the abscissa and frame width of the first AWB calibration subframe; determining the first AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the first AWB calibration subframe; determining the second AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the second AWB calibration subframe; and determining the AWB calibration frame based on the first AWB calibration subframe and the second AWB calibration subframe.

[0011] In a possible implementation, when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is [1, 2], the bottom of the first AWB calibration subframe overlaps with the second AWB calibration subframe, and the determining of the AWB calibration frame based on the AWB calibration variable includes: determining the ordinate of the first AWB calibration subframe based on the ordinate and frame height of the second AWB calibration subframe; determining the abscissa of the first AWB calibration subframe, the interval difference and the frame width of the first AWB calibration subframe based on the abscissa of the first AWB calibration subframe, the interval difference and the frame width of the first AWB calibration subframe. The horizontal coordinate of the frame, the interval difference is the difference between a preset constant and an interval coefficient, the interval coefficient is the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the preset constant is 2; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe, the first AWB calibration subframe is determined; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe, the second AWB calibration subframe is determined; based on the first AWB calibration subframe and the second AWB calibration subframe, the AWB calibration frame is determined.

[0012] In a possible implementation, the preset cost function is:

[0013] cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou

[0014] Wherein, sum_ct_diff represents the distance sum between the color temperature corresponding to each of the AWB calibration subframes and the target color temperature, valid_iou represents the sum of the intersection and union ratios of the gray block in each of the AWB calibration subframes, invalid_iou represents the sum of the intersection and union ratios of the non-gray block in each of the AWB calibration subframes, and k0, k1, k2 represent weights.

[0015] In a possible implementation, the sum of the intersection-and-combination ratios of the gray block in each of the AWB calibration subframes is:

[0016]

[0017] Wherein, sum_GrayRect represents the number of gray blocks in each of the AWB calibration subframes, and sum_Gray represents the number of gray blocks in each of the block images.

[0018] In a possible implementation, the sum of the intersection-and-joint ratios of the non-gray block in each of the AWB calibration subframes is:

[0019]

[0020] Wherein, sum_NoGrayRect represents the number of non-gray blocks in each of the AWB calibration subframes, and sum_NoGray represents the number of non-gray blocks in each of the block images.

[0021] In a second aspect, an embodiment of the present application provides an AWB calibration device based on semantic segmentation, the device comprising: an acquisition module, which acquires multiple scene images, each of the scene images corresponds to a different color temperature; a determination module, which determines, for any scene image in the scene images, a gray area and a non-gray area corresponding to each of the scene images through semantic segmentation; the determination module is also used to block any scene image in the scene images and determine the block images corresponding to each of the scene images; the determination module determines the gray area blocks, non-gray area blocks and gray area binary images corresponding to each of the scene images based on the block images, gray areas and non-gray areas corresponding to each of the scene images, the gray area blocks being any block within the block images , a block with a gray area greater than a preset ratio, the gray area binary image is a binary image composed of all the gray area blocks; an extraction module is used to determine the pixel ratio coordinates of each of the scene images in the same coordinate system based on the gray area blocks, non-gray area blocks and gray area binary images corresponding to each of the scene images; the determination module is also used to determine the AWB calibration variable based on the pixel ratio coordinates of each of the scene images in the same coordinate system and the color temperature corresponding to each of the scene images through a preset cost function; the determination module is also used to determine the AWB calibration frame based on the AWB calibration variable, the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each of the scene images.

[0022] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or any one of the implementation methods thereof is implemented.

[0024] The present application acquires a plurality of scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks, and determines the block images corresponding to each scene image; based on the block images, gray areas and non-gray areas of each scene image, determines the gray area blocks, non-gray area blocks and gray area binary images, wherein the gray area blocks are blocks with a gray area greater than a preset ratio in any block of the block images, and the gray area binary images are binary images composed of all gray area blocks; based on the gray area blocks, non-gray blocks and gray area binary images corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variables through a preset cost function; based on the AWB calibration variables, determines the AWB calibration frame, which is used for AWB calibration, and includes AWB calibration subframes corresponding to each of the scene images. Compared with the prior art, when a color temperature map is input, the calibration frame is adjusted once. If a color temperature is adjusted, it is necessary to check whether it meets other color temperatures, which takes a lot of time. This application uses a method combining semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final calibration frame or the initialization frame for adjustment. The more color temperatures are input, the more accurate the AWB calibration frame is. In short, by automating the AWB calibration and initialization process, the debugging time and process are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A flowchart of an AWB calibration method based on semantic segmentation provided in one embodiment of the present application;

[0027] Figure 2 The scene image collected at a color temperature of 5600k provided for this application;

[0028] Figure 3 The non-gray area image of the scene image collected at a color temperature of 5600k provided in this application after being differentiated by the semantic segmentation model;

[0029] Figure 4 The gray area binary image after the scene image provided in this application is divided into blocks;

[0030] Figure 5 Overlay of gray area binary image provided for this application;

[0031] Figure 6A schematic diagram of any two adjacent AWB calibration subframes provided in this application;

[0032] Figure 7 Another schematic diagram of any two adjacent AWB calibration subframes provided by this application;

[0033] Figure 8 Schematic diagram of the AWB calibration frame corresponding to the 10 scene images provided in this application;

[0034] Fig. 9 A structural block diagram of an AWB calibration device based on semantic segmentation provided in one embodiment of the present application;

[0035] Fig.10 A schematic diagram of the structure of the electronic device provided in this application. DETAILED DESCRIPTION

[0036] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0037] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0038] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0039] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0040] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0041] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] The ISP pipeline is used to convert the digital quantity output by the imaging chip into a digital quantity that can be displayed on the screen. A good ISP pipeline can make the final displayed image obtain a sense of reality comparable to that of the human eye. The role of the AWB automatic white balance is to ensure that the real and objective colors are restored under different color temperatures. However, in order to simulate the human eye, the green sensitivity of the photosensitive chip is generally higher than that of red and blue. If the white scene is presented by the camera under normal fluorescent light, it will be greenish. If it is presented under other color temperatures, such as sunset (low color temperature), the white will turn red. If it is presented by a camera, the color cast will be more serious. Therefore, the goal of AWB is to ensure that the white still appears white through the camera in this scene.

[0043] The method to implement AWB is generally to use the Planck curve, which characterizes the sensor's perception of light of different wavelengths. Combined with the imaging pixel values ​​of the gray area in the scene, a more realistic color can be restored. This is called AWB calibration. The general implementation process is:

[0044] 1. Get the values ​​of (R / G, B / G) in the gray area of ​​different color temperatures (generally 4 to 5 color temperatures are used);

[0045] 2. Use R / G as the value of the horizontal coordinate x, B / G as the value of the vertical coordinate y, and draw all the points on the coordinate plane XoY (o is the coordinate origin);

[0046] 3. Use multiple straight lines / curves or areas to enclose all the points as exactly as possible;

[0047] 4. Mark the range of different color temperatures (an area on XoY).

[0048] The most difficult part of the above implementation process is to obtain the gray area in the image and an area that can circle the points well. The method proposed in this application can automate the process and obtain better results.

[0049] Figure 1A flowchart of an AWB calibration method based on semantic segmentation is provided in an embodiment of the present application. The method includes:

[0050] S110, acquiring a plurality of scene images, each scene image corresponding to a different color temperature.

[0051] As an example, multiple scene images are collected at multiple color temperatures. Specifically, 10 scene images can be collected in the color temperature range of 1000K and 15000K, where the first scene image is marked as 0 and the color temperature is 1000K, and the tenth scene image is marked as 9 and the color temperature is 15000K.

[0052] S120 , determining the gray area and the non-gray area corresponding to each scene image through semantic segmentation for any scene image in the scene images.

[0053] Exemplarily, for the 10 collected scene images, the gray area and non-gray area of ​​each scene image are determined respectively through a gray area binary classification semantic segmentation model.

[0054] In a possible implementation, for any scene image in the scene images, before determining the gray area and non-gray area corresponding to each scene image through semantic segmentation, a semantic segmentation model needs to be constructed, and the method for constructing the semantic segmentation model is: obtaining a color temperature scene image set, the color temperature scene image set includes fixed scene images of multiple color temperatures; annotating the color temperature scene image set to determine the gray area and non-gray area of ​​each fixed scene image; and determining the semantic segmentation model based on the gray area and non-gray area of ​​each fixed scene image and a preset model.

[0055] For example, the color representation of the scene image in S110 obtained by different color temperatures and sensors may be significantly different. Therefore, when building a semantic segmentation model, it is necessary to perform scrambling processing on the color data to improve the robustness of the model to different color changes.

[0056] To achieve this goal, the present application obtains fixed scene images containing multiple color temperatures so that the model can learn the impact of these color temperature changes on the image color. In order to reduce the network inference time (i.e., the time required for the model to process the image and output the result), the present application uses a smaller network architecture to simply identify the scene so that the model can quickly and accurately identify it. Furthermore, the color temperature scene image set is annotated to determine the gray area and non-gray area of ​​each fixed scene image. Annotating the color temperature scene image set can provide accurate supervision information for the preset model to guide the training of the model. Furthermore, based on the gray area and non-gray area of ​​each fixed scene image and the preset model, a semantic segmentation model is determined. Specifically, the preset model can be a fully convolutional network model (Fully Convolutional Networks, FCN) or a DeepLabv3+ model. Among them, the semantic segmentation model is specifically used for gray area binary classification. The semantic segmentation model will learn how to distinguish gray areas from non-gray areas based on image color, texture and other features. Exemplarily, the color temperature scene image such as Figure 2 As shown in the figure, the non-gray area image after being distinguished by the semantic segmentation model is as follows Figure 3 shown.

[0057] S130, dividing any scene image in the scene images into blocks, and determining the block images corresponding to each scene image.

[0058] Exemplarily, before AWB calibration, the input scene image needs to be processed in blocks, for example, a 1280*720 image is evenly divided into 64*36 blocks, each block size is 20*20. This block processing is to decompose the image into smaller, more manageable areas so that the color characteristics of each area can be analyzed separately. The size of the block is artificially given in advance, and this size can be adjusted according to the specific needs and effects of the algorithm, and this application does not limit it.

[0059] S140, based on the block images, gray areas and non-gray areas corresponding to each scene image, determine the gray blocks, non-gray blocks and gray area binary images corresponding to each scene image, the gray block is a block in any block of the block image with a gray area greater than a preset ratio, and the gray area binary image is a binary image composed of all gray blocks.

[0060] Exemplarily, according to each small block of the divided image, the proportion of gray area pixels therein is calculated to determine whether the small block is a gray area block. This proportion is obtained by dividing the number of all gray area pixels in the small block by the total number of pixels in the small block. For example: a preset proportion of 70% is set. If the proportion of gray area in a small block is greater than or equal to this threshold, the small block is considered to be a gray area. Extract the binary image of all gray area blocks and determine the mask containing only gray area blocks, that is, the gray area binary image. For example: Figure 4The gray area binary image after block processing provided by this application, Figure 5 It is an overlay image of the gray area binary image, where green represents the gray area.

[0061] S150 , determining pixel ratio coordinates of each scene image in the same coordinate system based on the gray block, non-gray block and gray binary image corresponding to each scene image.

[0062] Exemplarily, through multiple scene images at different color temperatures, gray blocks, non-gray blocks and gray binary images corresponding to multiple scene images at different color temperatures are obtained, and the (R / G, B / G) coordinates of the gray blocks and non-gray blocks on the gray binary images are extracted, that is, the pixel ratios of red to green, blue to green in the gray blocks and non-gray blocks. Then, these (R / G, B / G) values ​​are plotted on a two-dimensional XoY space, where the X axis represents R / G and the Y axis represents B / G.

[0063] S160 , determining an AWB calibration variable based on the pixel ratio coordinates of each scene image in the same coordinate system and the color temperature corresponding to each scene image through a preset cost function.

[0064] Exemplarily, step S110 collects 10 scene images M (m0, m2, ..., m9), whose corresponding target color temperature set is CT (ct0, ct2, ..., ct9), and the function for calculating the color temperature of the scene image is g, then the estimated color temperature is est_ctN = g (mN), in this embodiment, N∈[0,9].

[0065] Furthermore, in the following calculation process of the present application, the calculation of the intersection-over-union ratio requires the pixel ratio coordinates of each scene image in the same coordinate system.

[0066] Specifically, at a certain color temperature, the estimated color temperature difference and the target color temperature difference are:

[0067] ct_diffN=wN×|est_ctN-ctN|

[0068] Then, the sum of the distances between the color temperature corresponding to each AWB calibration subframe and the target color temperature is:

[0069]

[0070] Wherein, in the embodiment listed, the range of N is [0,9], represents the identification of the scene image, ct_diffN represents the distance between the color temperature corresponding to each AWB calibration subframe and the target color temperature, and w N Represents weight.

[0071] In a possible implementation, the sum of the intersection-and-union ratios of the gray blocks in each AWB calibration subframe is:

[0072]

[0073] Wherein, sum_GrayRect represents the number of gray blocks in each AWB calibration subframe, and sum_Gray represents the number of gray blocks in each block image.

[0074] As another example, the traditional IoU calculation formula may be used to determine the sum of the IoUs of the gray block in each AWB calibration subframe. The traditional IoU calculation formula is:

[0075]

[0076] Compared with the traditional IoU calculation formula, the calculation formula for calculating the sum of the IoU of the gray block in each AWB calibration subframe proposed in the present application does not limit the shape and size of the frame set itself, has fewer constraints, and is conducive to the stability of optimization.

[0077] In a possible implementation, the sum of the intersection-and-union ratios of the non-gray blocks in each AWB calibration subframe is:

[0078]

[0079] Wherein, sum_NoGrayRect represents the number of non-gray blocks in each AWB calibration subframe, and sum_NoGray represents the number of non-gray blocks in each block image.

[0080] As another example, the traditional I / O ratio calculation formula may be used to determine the sum of the I / O ratios of the non-gray blocks in each AWB calibration subframe. The traditional I / O ratio calculation formula is:

[0081]

[0082] Compared with the traditional IoU calculation formula, the calculation formula for calculating the sum of the IoU of the non-gray block in each AWB calibration subframe proposed in the present application does not limit the shape and size of the frame set itself, has fewer constraints, and is conducive to the stability of optimization.

[0083] In a possible implementation, the final preset cost function is determined according to the above formula as:

[0084] cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou

[0085] Where sum_ct_diff represents the sum of the distances between the color temperature corresponding to each AWB calibration subframe and the target color temperature, valid_iou represents the sum of the intersection and union ratios of the gray block in each AWB calibration subframe, invalid_iou represents the sum of the intersection and union ratios of the non-gray block in each AWB calibration subframe, k0, k1, k2 represent weights, and k2 can generally be set to 2×k1=100×k0.

[0086] Furthermore, the optimized AWB calibration variables are determined by minimizing the cost function cost_func. When 10 scene images are collected, 31 variables need to be optimized, namely [x0, y9, w0, h0] + [a1, w1, h1] + … + [a9, w9, h9], where x0 represents the horizontal coordinate of the first scene image, y9 represents the vertical coordinate of the tenth scene image, w0-w9 represent the frame width of the AWB calibration sub-frame corresponding to each scene image, h0-h9 represent the frame height of the AWB calibration sub-frame corresponding to each scene image, and a1-a9 represent the interval coefficients corresponding to the scene image and the previous scene image.

[0087] Specifically, the Nelder-Mead optimization algorithm can be used to optimize the preset cost function, or other optimization algorithms with range constraints, such as gradient descent, genetic algorithm, etc., can be used, which is not limited in this application.

[0088] S170 , determining an AWB calibration frame based on the AWB calibration variable, where the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration subframes corresponding to each scene image.

[0089] In a possible implementation, the AWB calibration variable includes an interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe, and a vertical coordinate of the second AWB calibration subframe, and the first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame; when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0,1], the right part of the first AWB calibration subframe overlaps with the second AWB calibration subframe; based on the AWB calibration variable, determine A The WB calibration frame includes: determining the ordinate of the first AWB calibration subframe based on the ordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; determining the abscissa of the second AWB calibration subframe based on the abscissa and frame width of the first AWB calibration subframe; determining the first AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the first AWB calibration subframe; determining the second AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the second AWB calibration subframe; and determining the AWB calibration frame based on the first AWB calibration subframe and the second AWB calibration subframe.

[0090] For example, for 10 scene images, after 31 AWB calibration variables are obtained by using a preset cost function, it is necessary to obtain an AWB calibration sub-frame corresponding to each scene image based on the 31 calibration variables.

[0091] Specifically, when the interval coefficient between two adjacent calibration subframes is in [0,1], such as Figure 6 As shown, the right part of the first AWB calibration subframe overlaps with the second AWB calibration subframe. Assuming that the calibration subframe No. 0 corresponds to the first scene image and the calibration subframe No. 1 corresponds to the second scene image, the AWB calibration variables x0, y1, w0, h0, a1, w1, h1 are now known, where x0 is the horizontal coordinate of the calibration subframe No. 0, y1 is the vertical coordinate of the calibration subframe No. 1, w0 and w1 are the widths of the calibration subframe No. 0 and the calibration subframe No. 1, respectively, h0 and h1 are the heights of the calibration subframe No. 0 and the calibration subframe No. 1, respectively, and a1 is the spacing coefficient of the calibration subframe No. 0 and the calibration subframe No. 1. Then y0 = y1 + a1 × h1, x1 = x0 + w0, and the vertical coordinate of the calibration subframe No. 0 and the horizontal coordinate of the calibration subframe No. 1 can be obtained.

[0092] In another possible implementation, when the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [1,2], the bottom of the first AWB calibration subframe coincides with the second AWB calibration subframe, and the AWB calibration frame is determined based on the AWB calibration variable, including: determining the ordinate of the first AWB calibration subframe based on the ordinate and frame height of the second AWB calibration subframe; determining the second AWB calibration subframe based on the abscissa, the interval difference and the frame width of the first AWB calibration subframe. The horizontal coordinate of the subframe, the interval difference is the difference between a preset constant and an interval coefficient, the interval coefficient is the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the preset constant is 2; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the first AWB calibration subframe, the first AWB calibration subframe is determined; based on the horizontal coordinate, vertical coordinate, frame width and frame height of the second AWB calibration subframe, the second AWB calibration subframe is determined; based on the first AWB calibration subframe and the second AWB calibration subframe, the AWB calibration frame is determined.

[0093] Specifically, when the interval coefficient between two adjacent calibration subframes is [1,2], such as Figure 7 As shown, the bottom of the first AWB calibration subframe coincides with the bottom of the second AWB calibration subframe. Assume that calibration subframe No. 0 corresponds to the first scene image and calibration subframe No. 1 corresponds to the second scene image. Now the AWB calibration variables x0, y1, w0, h0, a1, w1, h1 are known, where x0 is the horizontal coordinate of calibration subframe No. 0, y1 is the vertical coordinate of calibration subframe No. 1, w0 and w1 are the frame widths of calibration subframe No. 0 and calibration subframe No. 1, respectively, h0 and h1 are the frame heights of calibration subframe No. 0 and calibration subframe No. 1, respectively, and a1 is the spacing coefficient between calibration subframe No. 0 and calibration subframe No. 1. Then y0 = y1 + h1, x1 = x0 + (2-a1) × w0. In this way, the vertical coordinate of calibration subframe No. 0 and the horizontal coordinate of calibration subframe No. 1 can be calculated. The reason why the spacing coefficient is divided into [0,1] and [1,2] is to ensure that the spacing coefficient (in Figure 4 and Figure 5 When a1) is equal to 1, the results obtained by coinciding on the right and the bottom are consistent, ensuring the continuity of the change of variable a1. In addition, the initial setting of x0 and y1 is because they are both variables greater than 0, and the subsequent relationship is obtained by addition, thus avoiding the problem of negative numbers obtained by subtraction, which makes it difficult for optimization to converge.

[0094] It should be noted that in the above process of finding y0 and x1 through x0, y1, w0, h0, a1, w1, h1, for ease of understanding, only two AWB calibration sub-frames are used as examples. In the example provided in the present application, in the process of finding the corresponding AWB calibration sub-frames through 10 scene images, the 31 known calibration variables that need to be known are [x0, y9, w0, h0] + [a1, w1, h1] + ... + [a9, w9, h9], and according to the range of a1-a9 ([0,1] or [1,2]), x1 is found through x0, x2 is found through x1, ..., x9 is found through x8, and y8 is found through y9, y7 is found through y8, ..., y0 is found through y1. Then, the AWB calibration frame is determined based on the results and the 31 AWB calibration variables, as shown in the following example. Figure 8 shown.

[0095] The present application acquires a plurality of scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks, and determines the block images corresponding to each scene image; based on the block images, gray areas and non-gray areas of each scene image, determines the gray area blocks, non-gray area blocks and gray area binary images, wherein the gray area blocks are blocks with a gray area greater than a preset ratio in any block of the block images, and the gray area binary images are binary images composed of all gray area blocks; based on the gray area blocks, non-gray blocks and gray area binary images corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variables through a preset cost function; based on the AWB calibration variables, determines the AWB calibration frame, which is used for AWB calibration, and includes AWB calibration subframes corresponding to each of the scene images. This application uses a method that combines semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final result frame or the adjusted initialization frame. By automating the AWB calibration and initialization process, the debugging time and process are reduced. In addition, this application can input scene images of multiple color temperatures according to user needs. The more color temperatures input, the more accurate the AWB calibration frame obtained.

[0096] Fig. 9 This is a structural block diagram of an AWB calibration device based on semantic segmentation provided in one embodiment of the present application. For the sake of convenience, only the part related to the embodiment of the present application is shown. Fig. 9 The AWB calibration device 900 based on semantic segmentation includes an acquisition module 901 , a determination module 902 , and an extraction module 903 .

[0097] In one implementation, the apparatus 900 may be used to implement the above Figure 1For example, the acquisition module 901 is used to implement S110, the determination module 902 is used to implement S120 to S140, S160 to S170, and the extraction module 903 is used to implement S150.

[0098] The present application acquires a plurality of scene images at different color temperatures; based on each scene image, determines the gray area and non-gray area corresponding to each scene image through semantic segmentation; divides each scene image into blocks, and determines the block images corresponding to each scene image; based on the block images, gray areas and non-gray areas of each scene image, determines the gray area blocks, non-gray area blocks and gray area binary images, wherein the gray area blocks are blocks with a gray area greater than a preset ratio in any block of the block images, and the gray area binary images are binary images composed of all gray area blocks; based on the gray area blocks, non-gray blocks and gray area binary images corresponding to each scene image, determines the pixel ratio coordinates of each scene image in the same coordinate system; based on the pixel ratio coordinates and the color temperature corresponding to each scene image, determines the AWB calibration variables through a preset cost function; based on the AWB calibration variables, determines the AWB calibration frame, which is used for AWB calibration, and includes AWB calibration subframes corresponding to each of the scene images. This application uses a method that combines semantic segmentation with a cost function. The calculated AWB calibration frame can be used as the final result frame or the adjusted initialization frame. By automating the AWB calibration and initialization process, the debugging time and process are reduced. In addition, this application can input scene images of multiple color temperatures according to user needs. The more color temperatures input, the more accurate the AWB calibration frame obtained.

[0099] Fig.10 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device 10 of this embodiment includes: at least one processor 100 ( Fig.10 Only one is shown in the figure) a processor, a memory 101, and a computer program 102 stored in the memory 101 and executable on the at least one processor 100, and when the processor 100 executes the computer program 102, the steps in any of the above method embodiments are implemented.

[0100] The electronic device 10 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art will appreciate that Fig.10 This is only an example of the electronic device 10 and does not constitute a limitation on the electronic device 10 . The electronic device 10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0101] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0102] In some embodiments, the memory 101 may be an internal storage unit of the electronic device 10, such as a hard disk or memory of the electronic device 10. In other embodiments, the memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 10. Further, the memory 101 may also include both an internal storage unit of the electronic device 10 and an external storage device. The memory 101 is used to store an operating system, an application program, a boot loader (BootLoader), data and other programs, such as the program code of the computer program, etc. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0103] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0105] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0106] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0107] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0108] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0109] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0110] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0111] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0112] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0113] The above description is only a preferred implementation mode of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An AWB calibration method based on semantic segmentation, characterized in that: The method comprises: Acquire multiple scene images, each of the scene images corresponds to a different color temperature; For any scene image among the scene images, determining the gray area and the non-gray area corresponding to each of the scene images through semantic segmentation; Divide any scene image among the scene images into blocks, and determine the block images corresponding to each scene image; Based on the block images, gray areas and non-gray areas corresponding to each of the scene images, the gray area blocks, non-gray area blocks and gray area binary images corresponding to each of the scene images are determined, wherein the gray area blocks are blocks in any block of the block images whose gray area is greater than a preset ratio, and the gray area binary image is a binary image composed of all the gray area blocks; Determine the pixel ratio coordinates of each scene image in the same coordinate system based on the gray block, non-gray block and gray area binary image corresponding to each scene image; Determining an AWB calibration variable through a preset cost function based on the pixel ratio coordinates of each of the scene images in the same coordinate system and the color temperature corresponding to each of the scene images; Based on the AWB calibration variables, an AWB calibration frame is determined, where the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each of the scene images.

2. The method according to claim 1, characterized in that For any scene image in the scene images, before determining the gray area and the non-gray area corresponding to each scene image through semantic segmentation, a semantic segmentation model is constructed. The method for constructing the semantic segmentation model is: Acquire a color temperature scene image set, wherein the color temperature scene image set includes fixed scene images of multiple color temperatures; Annotating the color temperature scene image set to determine the gray area and non-gray area of ​​each of the fixed scene images; The semantic segmentation model is determined based on the gray area and the non-gray area of ​​each of the fixed scene images and a preset model.

3. The method according to claim 1, characterized in that The AWB calibration variables include an interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, a frame height and a frame width of the first AWB calibration subframe, a frame height and a frame width of the second AWB calibration subframe, a horizontal coordinate of the first AWB calibration subframe and a vertical coordinate of the second AWB calibration subframe, and the first AWB calibration subframe and the second AWB calibration subframe are any two adjacent AWB calibration subframes in the AWB calibration frame; When the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is in [0, 1], the right portion of the first AWB calibration subframe overlaps with the second AWB calibration subframe; The determining of the AWB calibration frame based on the AWB calibration variable comprises: Determine the ordinate of the first AWB calibration subframe based on the ordinate of the second AWB calibration subframe, the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the frame height of the second AWB calibration subframe; Determine the abscissa of the second AWB calibration subframe based on the abscissa and the frame width of the first AWB calibration subframe; Determine the first AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the first AWB calibration subframe; Determine the second AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the second AWB calibration subframe; The AWB calibration frame is determined based on the first AWB calibration subframe and the second AWB calibration subframe.

4. The method according to claim 3, characterized in that: When the interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe is [1, 2], the bottom of the first AWB calibration subframe overlaps with the second AWB calibration subframe, and determining the AWB calibration frame based on the AWB calibration variable includes: Determine the ordinate of the first AWB calibration subframe based on the ordinate and the frame height of the second AWB calibration subframe; Determine the abscissa of the second AWB calibration subframe based on the abscissa of the first AWB calibration subframe, the interval difference, and the frame width of the first AWB calibration subframe, wherein the interval difference is a difference between a preset constant and an interval coefficient, the interval coefficient is an interval coefficient between the first AWB calibration subframe and the second AWB calibration subframe, and the preset constant is 2; Determine the first AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the first AWB calibration subframe; Determine the second AWB calibration subframe based on the abscissa, ordinate, frame width and frame height of the second AWB calibration subframe; The AWB calibration frame is determined based on the first AWB calibration subframe and the second AWB calibration subframe.

5. The method according to claim 1, characterized in that The preset cost function is: cost_func=k0×sum_ct_diff+k1×valid_iou+k2×invalid_iou Wherein, sum_ct_diff represents the sum of the distances between the color temperature corresponding to each of the AWB calibration subframes and the target color temperature, valid_iou represents the sum of the intersection and union ratios of the gray blocks in each of the AWB calibration subframes, invalid_iou represents the sum of the intersection and union ratios of the non-gray blocks in each of the AWB calibration subframes, and k0, k1, and k2 represent weights.

6. The method according to claim 5, characterized in that The sum of the intersection-and-combination ratios of the gray block in each of the AWB calibration subframes is: Wherein, sum_GrayRect represents the number of gray blocks in each of the AWB calibration subframes, sum_Gray represents the number of gray blocks in each of the block images, and N represents the identifier of the scene image.

7. The method according to claim 5, characterized in that The sum of the intersection-and-combination ratios of the non-gray blocks in each of the AWB calibration subframes is: Wherein, sum_NoGrayRect represents the number of non-gray blocks in each of the AWB calibration subframes, and sum_NoGray represents the number of non-gray blocks in each of the block images.

8. An AWB calibration device based on semantic segmentation, characterized in that: The device comprises: An acquisition module is used to acquire a plurality of scene images, each of which corresponds to a different color temperature; A determination module, for any scene image in the scene images, determines the gray area and the non-gray area corresponding to each of the scene images through semantic segmentation; The determination module is further used to divide any scene image in the scene images into blocks, and determine the block images corresponding to each scene image; The determination module determines the gray block, non-gray block and gray area binary image corresponding to each of the scene images based on the block images, gray areas and non-gray areas corresponding to each of the scene images, wherein the gray block is a block with a gray area greater than a preset ratio in any block of the block images, and the gray area binary image is a binary image composed of all the gray blocks; An extraction module, used for determining the pixel ratio coordinates of each scene image in the same coordinate system based on the gray block, non-gray block and gray area binary image corresponding to each scene image; The determination module is further used to determine the AWB calibration variable through a preset cost function based on the pixel ratio coordinates of each of the scene images in the same coordinate system and the color temperature corresponding to each of the scene images; The determination module is further used to determine an AWB calibration frame based on the AWB calibration variable, wherein the AWB calibration frame is used for AWB calibration, and the AWB calibration frame includes AWB calibration sub-frames corresponding to each of the scene images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images