Image quality improvement method, electronic device, and storage medium

Abstract

Embodiments of the present application disclose an image quality improving method, an electronic device and a storage medium. The method comprises: acquiring a shooting image of a camera in a third shooting scene, denoted as a third image; acquiring a virtual focus area of an infrared image of the camera in the third shooting scene; acquiring a stored black-and-white image in a first shooting scene, denoted as a first image; determining a to-be-enhanced area corresponding to the third image according to the virtual focus area; determining an enhanced area corresponding to the first image according to the virtual focus area; determining a fusion coefficient of the enhanced area and the to-be-enhanced area according to a virtual focus degree of the to-be-enhanced area; and performing fusion processing on the enhanced area and the to-be-enhanced area according to the fusion coefficient to obtain a processed third image. According to the image quality improving scheme provided by the embodiments of the present application, the image quality of a day-and-night non-convergent camera in a night mode is effectively improved.

Description

Technical Field

[0001] This invention relates to, but is not limited to, the field of video surveillance, and particularly to a method for improving image quality, an electronic device, and a storage medium. Background Technology

[0002] With the popularization of video surveillance technology, cameras are developing towards intelligence, high performance, and diversification. However, traditional cameras, primarily focused on night vision, still occupy a large market share. Traditional night vision cameras utilize infrared illumination to produce clearer images with more detail. Fixed-focus cameras typically focus under visible light, but because visible and infrared light have different refractive indices, the point of focus shifts when the camera switches to infrared, severely reducing the night vision image quality. In recent years, with advancements in lens manufacturing technology and the widespread adoption of other solutions such as filters, most lenses can achieve day and night co-focus. However, some lenses, for various reasons, cannot achieve day and night co-focus and can only address this issue by adding filters of varying thicknesses. This solution alters the infrared light path by changing the thickness of the infrared transmission filter, ensuring clear imaging on the sensor, but it also changes the visible light path. Therefore, this solution only guarantees clear images in pure infrared scenes or scenes with a high infrared component. Once visible light is present in the scene, the visible light portion will be blurred, meaning it still cannot avoid the problem of localized defocusing when infrared and visible light coexist. Summary of the Invention

[0003] This disclosure provides an image quality improvement method, electronic device, and storage medium, which effectively improves the local defocusing problem that occurs when a day / night non-confocal lens encounters mixed light sources in infrared mode, thereby improving image quality.

[0004] On one hand, embodiments of this disclosure provide an image quality improvement method, including:

[0005] Acquire the image captured by the camera in the third shooting scene, and denote it as the third image;

[0006] Obtain the out-of-focus area of ​​the infrared image of the camera in the third shooting scene;

[0007] Acquire the existing black and white image of the first shooting scene and denote it as the first image;

[0008] Based on the defocused area, determine the corresponding area to be enhanced in the third image;

[0009] Based on the defocused area, determine the corresponding enhancement area in the first image;

[0010] Based on the degree of defocus in the region to be enhanced, the fusion coefficients of the enhanced region and the region to be enhanced are determined respectively. The enhanced region and the region to be enhanced are then fused according to the fusion coefficients to obtain the processed third image.

[0011] The third shooting scenario is a shooting scenario with infrared fill light after the camera switches to night mode dual filters, and the first shooting scenario is a shooting scenario before the camera last switched to night mode dual filters.

[0012] On the other hand, embodiments of this disclosure also provide an electronic device, including:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the image quality improvement method as described in any embodiment of this disclosure.

[0016] On the other hand, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the image quality improvement method as described in any embodiment of this disclosure.

[0017] After reading and understanding the accompanying diagrams and detailed descriptions, the other aspects can be understood. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0019] Figure 1 This is a flowchart of an image quality improvement method provided in an embodiment of the present invention;

[0020] Figure 2 This is a flowchart of another image quality improvement method provided in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart of another image quality improvement method provided in an embodiment of the present invention.

[0022] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0025] Furthermore, in this invention, descriptions involving "first," "second," etc., are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0026] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0027] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0028] In video surveillance solutions, cameras that don't co-focus during day and night use dual filter switching (ICR) when switching between day and night. This dual filter switching scheme blocks infrared light during the day and allows infrared light to pass through at night. Simultaneously, an infrared supplementary light is activated in night mode to improve nighttime image quality. However, it can be seen that in night mode, the presence of visible light in the scene causes blurring of the visible light portion of the image. Therefore, related technical solutions cannot effectively avoid localized defocusing issues when infrared and visible light coexist in night mode.

[0029] This disclosure provides an image quality improvement scheme. By using an image fusion scheme, the out-of-focus areas in the infrared image captured after the day-night switch are enhanced using a visible light image captured before the day-night switch. This solves the out-of-focus problem of infrared images in night mode and effectively improves the imaging quality in night mode.

[0030] The process of day-night switching for a camera with dual filter switching ICR described in this embodiment is as follows:

[0031] Daytime mode (dual filters switch to infrared filter) → Night mode 1 (dual filters switch to infrared sensor, no infrared light supplement) → Night mode 2 (dual filters switch to infrared sensor, infrared light supplement).

[0032] This disclosure provides an image quality improvement method, such as... Figure 1 As shown, it includes:

[0033] Step 110: Obtain the image captured by the camera in the third shooting scene, and record it as the third image;

[0034] Step 120: Obtain the out-of-focus area of ​​the infrared image of the camera in the third shooting scene;

[0035] Step 130: Obtain the stored black and white image of the first shooting scene and record it as the first image;

[0036] Step 140: Based on the out-of-focus area, determine the region to be enhanced in the third image;

[0037] Step 150: Determine the enhancement region corresponding to the defocused region in the first image;

[0038] Step 160: Based on the out-of-focus degree of the region to be enhanced, determine the fusion coefficients of the enhanced region and the region to be enhanced respectively, and perform fusion processing on the enhanced region and the region to be enhanced according to the fusion coefficients to obtain the processed third image;

[0039] The third shooting scenario is a shooting scenario with infrared fill light after the camera switches to night mode dual filters, and the first shooting scenario is a shooting scenario before the camera last switched to night mode dual filters.

[0040] It should be noted that some embodiments of this application also involve a second shooting scenario, which is a shooting scenario without infrared supplementary light after the camera switches to night mode dual filters. It can be seen that the first, second, and third shooting scenarios correspond to three states of the camera during the day-night switching process. The related first, second, and third images represent images corresponding to different shooting scenarios and do not represent a specific processing order or priority. The first and second shooting scenarios refer to the closest scenario before the current shooting scenario (third shooting scenario). Unless otherwise specified, the images captured in each scenario are images captured when the device reaches a stable state in the corresponding scenario.

[0041] As can be seen, the third image obtained in step 110 is the scene captured under infrared illumination after the night mode dual filter switching, that is, the image captured when the camera is working in the scene where the ICR is switched to night mode and there is infrared illumination.

[0042] In some exemplary embodiments, the out-of-focus area of ​​the infrared image of the camera in the third shooting scenario is determined according to the following method:

[0043] Step 120110: According to the same division rules, the existing black and white images in the first shooting scene, the existing infrared-free images in the second shooting scene, and the existing infrared images in the third shooting scene are divided into blocks to obtain multiple black and white image blocks, multiple infrared-free image blocks, and multiple infrared image blocks.

[0044] Step 120120: Calculate the sharpness deviation ratio for each block position based on the black and white image block, the non-infrared image block, and the infrared image block corresponding to each identical block position; determine the block positions with a sharpness deviation ratio greater than a set deviation threshold as out-of-focus block positions.

[0045] Step 120130: The infrared image blocks corresponding to all the out-of-focus block positions constitute the out-of-focus area.

[0046] In some exemplary embodiments, the existing black-and-white image of the first shooting scene is a black-and-white image captured and saved by the camera in daytime mode (dual filter switched to infrared filter) before determining to switch to night mode.

[0047] In some exemplary embodiments, the existing infrared-free images in the second shooting scene are images captured and saved by the camera in Night Mode 1 (dual filters switched to infrared sensing, no infrared lamp supplementary light) during the process of determining to switch to this night mode.

[0048] In some exemplary embodiments, the stored infrared image in the third shooting scene is an image captured and saved by the camera in Night Mode 2 (dual filter switched to infrared sensing, with infrared lamp supplementary lighting) after the camera has determined that the night mode switch has been completed.

[0049] It should be noted that the stored infrared image used in step 120 to determine the out-of-focus area of ​​the infrared image can be the third image from step 110, i.e., the currently acquired third image (infrared image) is used for out-of-focus area determination immediately after the switch is completed and the third image processing is performed. Alternatively, it can be other infrared images taken before the third image was captured after the switch. Considering that the changes in the business scenario are relatively slow, it is not necessary to determine the out-of-focus area for each third image (infrared image) captured. Instead, out-of-focus areas determined in the previous short period (within the set effective duration) can be used for subsequent processing. The method is not limited to a specific approach and can be flexibly selected based on the characteristics of the business scenario and the system's processing capabilities.

[0050] It is understandable that after switching to night mode and turning on infrared fill light, if a defocused area is determined according to step 120 after acquiring the third image (infrared image), the subsequent steps are performed based on the defocused area to perform fusion processing on the defocused area and obtain the third image with improved quality. Alternatively, based on the defocused area determined this time, the corresponding defocused areas in subsequent third images (infrared images) can be fused to output the third image with improved image quality.

[0051] In some exemplary embodiments, the out-of-focus area of ​​the infrared image of the camera in the third shooting scenario is determined according to the following method:

[0052] Step 120210: According to the same division rules, the existing black and white images in the first shooting scene, the infrared-free images in the second shooting scene, and the infrared images in the third shooting scene are divided into blocks to obtain multiple black and white image blocks, multiple infrared-free image blocks, and multiple infrared image blocks.

[0053] Step 120220: Based on the black and white image block, the non-infrared image block, and the infrared image block corresponding to each same block position, calculate the sharpness deviation ratio of each block position, and determine the block position with a sharpness deviation ratio greater than the set deviation threshold as the suspected out-of-focus block position.

[0054] Step 120230: Calculate the red component gain of the infrared image block corresponding to each suspected out-of-focus block position, and determine the suspected out-of-focus block positions where the red component gain is less than the set red component gain threshold as out-of-focus block positions.

[0055] Step 120240: The infrared image blocks corresponding to all the out-of-focus block positions constitute the out-of-focus area.

[0056] As can be seen, in some embodiments, the sharpness evaluation value may be inaccurate due to hardware limitations of the device itself or insufficient scene detail. Therefore, in addition to using sharpness to detect whether there is local out-of-focus areas in the image, based on the day-night defocusing principle, the RGB components provided by the image sensor are further used to determine whether the image is out of focus. Compared to the previous embodiments, this embodiment first determines the suspected out-of-focus block positions based on the sharpness deviation ratio of each block position, and then further determines the out-of-focus blocks based on the red component gain of the blocks, which can further improve the accuracy of out-of-focus area determination.

[0057] In some exemplary embodiments, the partitioning rule is as follows: dividing into M*N blocks; that is, each image is divided into its own M*N blocks according to the same partitioning rule, for example, all are divided into 6*8 blocks, that is, the image is divided into 6 rows and 8 columns, a total of 48 blocks. The number and rules of partitioning can be flexibly determined according to the image processing accuracy and system processing capability. Unless otherwise specified, all aspects of image partitioning in this disclosure adopt a consistent partitioning rule to ensure that the blocks of each image correspond one-to-one in their positions.

[0058] The sharpness deviation ratio for each block location is calculated using the following method:

[0059]

[0060] Wherein, ΔFV mn FV is the sharpness deviation ratio of the block position in the m-th row and n-th column. Amn FV is the sharpness evaluation value of the black and white image block at the m-th row and n-th column position. Bmn FV is the sharpness evaluation value of the infrared-free image block at the m-th row and n-th column position. Cmn The sharpness evaluation value of the infrared image block at the m-th row and n-th column position;

[0061] M and N are both integers greater than 1, where m = 1, ..., M and n = 1, ..., N.

[0062] The sharpness evaluation value of each image block can be determined according to relevant schemes or relevant sharpness evaluation functions in the field, and the specific aspects are not discussed in this application.

[0063] In some exemplary embodiments, the red component gain of the image block corresponding to each block location is the average red component gain of the pixels contained in that image block;

[0064] Among them, the red component gain of each pixel R represents the grayscale value of the red component, and G represents the grayscale value of the green component.

[0065] In some exemplary embodiments, after the night mode is fully switched, i.e., in a shooting scenario with infrared supplementary light after the night mode dual filter switch, the out-of-focus area of ​​the infrared image is reacquired and updated according to a preset period T, and steps 120110-120130 or 120210-120240 are performed based on the latest infrared image to update the out-of-focus area of ​​the infrared image in the third shooting scenario.

[0066] In some exemplary embodiments, the region to be enhanced in the third image determined in step 140 includes infrared image blocks in the third image corresponding to all out-of-focus block positions; the region to be enhanced in the first image determined in step 150 includes black and white image blocks in the first image corresponding to all out-of-focus block positions.

[0067] Those skilled in the art will understand that, according to a consistent partitioning rule, each image is divided into blocks with corresponding block positions. The defocused area obtained in step 120 indicates one or more block positions. These block positions can be mapped to each image to obtain the corresponding image blocks, thereby forming the desired region. Specifically, mapping to the first image yields the enhanced region, and mapping to the third image yields the region to be enhanced.

[0068] In some exemplary embodiments, the region to be enhanced in the third image determined in step 140 includes infrared image blocks in the third image corresponding to all out-of-focus block positions, and the region to be enhanced in the first image determined in step 150 includes black and white image blocks in the first image corresponding to all out-of-focus block positions. Accordingly, in step 160, each black and white image block corresponds to a fusion coefficient a for a black and white image block, and infrared image blocks at the same block position correspond to a fusion coefficient b for an infrared image block; wherein, a and b are both numbers less than or equal to 1 and greater than or equal to 0; the higher the degree of out-of-focus in the region to be enhanced, the larger a and the smaller b; the lower the degree of out-of-focus in the region to be enhanced, the smaller a and the larger b.

[0069] In some exemplary embodiments, the fusion coefficient a of a black and white image block is (1-κ)*λ, and the fusion coefficient b of an infrared image block at the same position is κ*(1-λ).

[0070] This is the sharpness deviation coefficient corresponding to the location of this segment;

[0071] R represents the gain percentage of the infrared component. GAR represents the red component gain of the black-and-white image block corresponding to the block location in the enhancement region. GC The red component gain corresponds to the infrared image block in the region to be enhanced at the block location.

[0072] FV Amn FV is the sharpness evaluation value of the black and white image block at the m-th row and n-th column position. Bmn FV is the sharpness evaluation value of the infrared-free image block at the m-th row and n-th column position. Cmn This is the sharpness evaluation value for the infrared image block at the m-th row and n-th column block position.

[0073] As can be seen, the clearer the infrared image block (the lower the degree of blurriness), the larger k is, and the smaller 1-k is; conversely, the less clear the infrared image block (the higher the degree of blurriness), the smaller k is, and the larger 1-k is. Similarly, the clearer the infrared image block (the lower the degree of blurriness), the smaller λ is, and the larger 1-λ is; conversely, the less clear the infrared image block (the higher the degree of blurriness), the larger λ is, and the smaller 1-λ is. Alternatively, the clearer the infrared image block (the lower the degree of blurriness), the larger κ*(1-λ) is, and the smaller (1-κ)*λ is; conversely, the less clear the infrared image block (the higher the degree of blurriness), the smaller κ*(1-λ) is, and the larger (1-κ)*λ is.

[0074] In some exemplary embodiments, the fusion coefficient a of a black and white image block is 1-k, and the fusion coefficient b of an infrared image block at the same location is k.

[0075] In some exemplary embodiments, the fusion coefficient a = λ for a black and white image block and the fusion coefficient b = 1-λ for infrared image blocks at the same location.

[0076] In some exemplary embodiments, step 160 involves fusing the enhanced region and the region to be enhanced according to the fusion ratio coefficient to obtain a processed third image, including:

[0077] Step 160110: For each out-of-focus block location, the black-and-white image block corresponding to that location in the enhancement area and the infrared image block corresponding to that location in the area to be enhanced are fused together as follows:

[0078] f R (x,y)=f C (x,y)*a+f ir (x,y)*b;

[0079] Among them, f R (x,y) represents the image block after fusion at the location of the defocused block, f C (x,y) represents the black-and-white image block in the enhancement region corresponding to the location of the defocused block, fir (x,y) represents the infrared image block in the region to be enhanced corresponding to the position of the defocused block;

[0080] Step 160120: The image blocks after fusing all out-of-focus blocks are merged with the image blocks of the non-enhanced areas in the third image to obtain the processed third image.

[0081] Where 'a' is the fusion coefficient of the black and white image block in the enhancement area corresponding to the defocused block position, and 'b' is the fusion coefficient of the infrared image block in the area to be enhanced corresponding to the defocused block position; where both 'a' and 'b' are numbers less than or equal to 1 and greater than or equal to 0; the higher the degree of defocus of the defocused block in the area to be enhanced, the larger 'a' and the smaller 'b'; the lower the degree of defocus of the defocused block in the area to be enhanced, the smaller 'a' and the larger 'b'.

[0082] In some exemplary embodiments, the fusion coefficients a and b for different block positions are determined according to the degree of defocusing of their respective corresponding blocks, and can be the same or different.

[0083] In some exemplary embodiments, a = 1 and b = 0, that is, the infrared image block of the region to be enhanced in the third image is replaced with the black and white image block of the region to be enhanced in the first image.

[0084] In some exemplary embodiments, the fusion coefficient of a black-and-white image block is a = 1 - k, and the fusion coefficient of an infrared image block at the same location is b = k; or, a = λ, b = 1 - λ; or, a = (1 - κ) * λ, b = κ * (1 - λ). It can be seen that for the region to be enhanced, the higher the degree of defocus, the higher the weight of the black-and-white image block in the enhanced region during block image fusion, and the lower the weight of the infrared image block at the corresponding location in the region to be enhanced; conversely, the lower the degree of defocus, the lower the weight of the black-and-white image block in the enhanced region during block image fusion, and the higher the weight of the infrared image block at the corresponding location in the region to be enhanced.

[0085] Alternatively, those skilled in the art may also employ other methods for evaluating the degree of defocus, and set the fusion coefficients of the black and white image blocks and the infrared image blocks accordingly, so as to perform image fusion to obtain fused image blocks.

[0086] In some exemplary embodiments, step 160 involves fusing the enhanced region and the region to be enhanced according to the fusion ratio coefficient to obtain a processed third image, including:

[0087] Step 160210: For each out-of-focus block location, the black-and-white image block corresponding to that location in the enhancement area and the infrared image block corresponding to that location in the area to be enhanced are fused together as follows:

[0088] f R (x,y)=f C (x,y)*(1-κ)*λ+f ir (x,y)*κ*(1-λ);

[0089] Among them, f R (x,y) represents the image block after fusion at the location of the defocused block, f C (x,y) represents the black-and-white image block in the enhancement region corresponding to the location of the defocused block, f ir (x,y) represents the infrared image block in the region to be enhanced corresponding to the position of the defocused block; (1-κ)*λ represents the fusion coefficient of the black and white image block in the region to be enhanced corresponding to the position of the defocused block; and κ*(1-λ) represents the fusion coefficient of the infrared image block in the region to be enhanced corresponding to the position of the defocused block.

[0090] This is the sharpness deviation coefficient corresponding to the position of the out-of-focus block;

[0091] R represents the gain percentage of the infrared component. GA R represents the red component gain of the black-and-white image block in the enhancement region corresponding to the defocused block location. GC The red component gain of the infrared image block in the region to be enhanced, corresponding to the location of the defocused block;

[0092] Step 160220: Merge the image blocks after fusing all out-of-focus areas with the image blocks of the non-enhanced areas in the third image to obtain the processed third image.

[0093] It should be noted that the sharpness deviation coefficient κ and the infrared component gain ratio λ corresponding to the out-of-focus block position in step 160210 are calculated according to the above method. The sharpness evaluation value and infrared component gain involved are determined using the same method as those involved in the sharpness evaluation value and infrared component gain obtained in step 120 for the out-of-focus area determination process. The three image blocks involved in determining the sharpness deviation coefficient κ corresponding to the out-of-focus block position are the same image blocks used in step 120 for determining the sharpness deviation ratio of the block position.

[0094] In some exemplary embodiments, step 130 includes: acquiring a stored black-and-white image of the camera last captured before the most recent night mode dual filter switch.

[0095] Optionally, step 130 includes: selecting the black and white image with the closest gain value from a plurality of stored black and white images of the first shooting scene, based on the gain value of the third image.

[0096] Among them, the multiple black-and-white images in the first scene include black-and-white images with different gains taken in the first scene. That is, black-and-white images with different gains taken before the day-night transition.

[0097] Based on the principle, if the image is in infrared mode, non-confocal lenses typically exhibit out-of-focus conditions in areas where visible light is greater than infrared light. Infrared lamps, especially wide-angle infrared lamps, have limited illumination distances, generally resulting in insufficient illumination at farther locations, increasing the probability of out-of-focus conditions and necessitating image fusion. Aside from the out-of-focus condition, the infrared image at this location (where the visible light component is approximately equal to the infrared component) can be considered almost indistinguishable from the visible light image; that is, the infrared image is close to the visible light monochrome image before the ICR switch. Therefore, the visible light monochrome image before the ICR switch and the stabilized infrared image should be fused preferentially. For this purpose, the visible light monochrome image before the ICR switch needs to be saved beforehand.

[0098] Modern cameras have good low-light performance, so day-night switching occurs relatively late, and the environment before and after the switch is generally stable, meaning there's no noticeable switching of lights. However, as the environment gradually brightens, there will be a significant brightness difference between the infrared image and the stored visible light monochrome image, affecting the overall consistency of the fused image. Therefore, some embodiments require pre-storing 2-3 visible light monochrome images with different gains before the day-night switch to serve as the source of visible light data for image fusion in brighter environments.

[0099] It should be noted that, generally speaking, as long as the ambient lighting does not change significantly, the black and white image taken just before the day-night switch is selected, that is, the last black and white image taken before the most recent night mode dual filter switch. Optionally, before image fusion, the gain value of the current infrared image (third image) is obtained, and a matching stored visible light black and white image (first image) before the ICR switch is selected based on the gain value. This includes: performing a gain value matching degree detection on the current infrared image and multiple cached visible light black and white images, and selecting the one with the gain value closest to the existing visible light black and white image as another fusion image.

[0100] In some exemplary embodiments, for cases where a moving object appears in the third image (infrared image), step 160 includes:

[0101] By subtracting the first image from the third image, the moving part, i.e. the foreground part, in the third image is obtained;

[0102] The enhanced region in the first image is used to fuse the region to be enhanced in the third image to obtain a processed quasi-third image.

[0103] The motion portion is superimposed onto the quasi-third image to obtain the superimposed third image.

[0104] In some exemplary embodiments, the method further includes:

[0105] Step 100-1: Before switching to night mode, acquire and cache one or more black and white images;

[0106] Step 100-2: If it is determined that day-night switching is required, execute the dual filter to switch to night mode, do not turn on infrared fill light, and acquire and cache images without infrared light.

[0107] Step 100-3: Turn on the infrared fill light.

[0108] In step 100-1, the black and white image acquired and cached is also called the visible light black and white image. The corresponding shooting scene is the first shooting scene, so it is also called the first image. In step 100-2, the infrared-free image acquired and cached is the corresponding shooting scene, so it is also called the second image. Step 100-3 completes the night switching, that is, the dual filters are switched to night mode and the infrared fill light is turned on. The shooting scene after this is the third shooting scene.

[0109] Whether or not a night mode needs to be switched can be determined based on the relevant implementation plan, and specific aspects are not discussed in this application.

[0110] This disclosure also provides an image quality improvement method, such as... Figure 2 As shown, it includes:

[0111] Step 210: Before the day-night cycle, acquire and save black and white images with different gains;

[0112] Step 220: Determine whether the day-night switching conditions are met. If not, return to step 210. If met, proceed to step 230.

[0113] Step 230: Obtain and save the last black and white image before the switch;

[0114] Step 240: Switch the dual filters to night mode, turn off the infrared fill light, and acquire and save the image without infrared light.

[0115] Step 250: Turn on the infrared fill light and acquire an infrared image;

[0116] Step 260: Determine the out-of-focus area of ​​the infrared image;

[0117] Step 270: Based on the determined defocused area, perform image fusion to obtain an improved infrared image.

[0118] Step 260 includes: for each of the divided block positions, performing the following steps respectively:

[0119] Step 26010: Calculate the sharpness deviation ratio of the segment location;

[0120] Step 26020: Determine whether the sharpness deviation ratio is greater than the set deviation threshold. If it is greater, proceed to step 26030; otherwise, proceed to step 26060.

[0121] Step 26030: Calculate the red component gain of the infrared image block corresponding to the block location;

[0122] Step 26040: Determine whether the red component gain is less than the set red component gain threshold. If it is less, proceed to step 26050; otherwise, proceed to step 26060.

[0123] Step 26050: Determine that the block location is the defocus block location;

[0124] Step 26060: Determine that the block position is a non-focus block position.

[0125] It should be noted that the black and white images captured before the ICR switches to night mode are fully focused and sharp because they only have visible light illumination. These are used as one frame in the fusion image (the first image) and are also used to calculate the sharpness deviation ratio and the gain value of the red (R) component. The image without infrared illumination after the ICR switch (no infrared image), meaning only visible light illumination, is also in focus because the ICR has already switched, meaning visible light can no longer produce a clear image. This image is used to calculate the sharpness deviation ratio and determine if there are any localized areas of focus in the final infrared image. The stabilized infrared image with infrared illumination after the ICR switch is used to determine if there are any localized areas of focus and to calculate the gain value of the red (R) component, creating another frame in the fusion image (the third image). Finally, the first and third images are fused based on the identified out-of-focus areas to obtain the improved infrared image.

[0126] In some exemplary embodiments, when the camera detects the need to switch to night mode, it saves a visible light black-and-white image before the ICR switch, denoted as img1. After the day-night switch is completed, it saves an image after the ICR switch but with the infrared lights off, denoted as img2. Then, it saves an image after the infrared lights are on, denoted as img3. img3 needs to be updated periodically to determine whether there is local blurring in the image. The image is divided into m×n blocks, and the sharpness evaluation value of img1, img2, and img3 is calculated using a sharpness evaluation function, denoted as FV. A FV B and FV C It also calculates the sharpness rating for each image patch, denoted as FV. A11 FV A12 …FV Amn FV B11 FV B12 …FV Bmn ,FV C11 FV C12 …FV Cmn .

[0127] When the camera switches to infrared mode, if the scene is small, the lens has a large focal length, or the infrared illumination is strong, the infrared component in the image will be high, and the image should be clear. When visible light such as streetlights appears in the scene and is far from the camera, if the infrared illumination is sufficient to cover most of the visible light, the image will have no obvious problems. However, if the infrared illumination is insufficient, the image will appear out of focus within the range of distant visible light radiation. Therefore, the sharpness evaluation value and the proportion of infrared component are used to determine whether there is localized blurring in the image.

[0128] In the same scene, the image sharpness of img1, img2, and img3 should meet the FV requirement. A >FV C >FV B A resolution threshold, denoted as FV, ​​is preset. THR This is used to determine if an image has significant out-of-focus issues. It calculates the sharpness deviation ratio of image blocks. When a certain block of img3 is out of focus, then FV Cmn It will be relatively small, almost infinitely close to FV. Bmn Then ΔFV mn It will be quite large, exceeding the set threshold FV. THR If so, it is considered that there is obvious blurring.

[0129] Calculate a coefficient When a certain block of img3 is out of focus, FV Cmn -FV Bmn It will be relatively small, while |FV Amn -FVBmn The larger the value, the smaller the final coefficient κ will be, which represents the proportion of infrared components during image fusion.

[0130] In some exemplary embodiments, the sharpness evaluation value may be inaccurate due to hardware limitations of the device itself or insufficient scene detail. Therefore, in addition to using sharpness to detect the presence of localized blurring in the image, based on the principle of day and night blurring, it is also necessary to utilize the RGB components provided by the image sensor to determine whether the image is out of focus, which offers better real-time performance. Areas with sufficient infrared illumination generally do not experience blurring, and the R component is relatively large in such areas. When the camera switches to infrared, if there is no visible light in the scene initially, but then visible light suddenly increases (e.g., streetlights are turned on), the sharpness judgment will not be entirely accurate because the scene has changed. In this case, incorporating the infrared component will make the judgment more accurate. Let the gain of the red R component of a pixel be denoted as... R represents the grayscale value of the red component, and G represents the grayscale value of the green component. (Note: The original text contains some formatting errors and inconsistencies. A more accurate translation would require the full context.) R GA R represents the red component gain value of the corresponding block in the visible light black and white image before ICR switching. GC This represents the infrared component gain value for the corresponding block of the current image. Following this method, the average component gain of each block of the infrared image can be obtained, denoted as R. Gmn When R Gmn Greater than or equal to the preset red component gain threshold R GTH If the infrared component is high and the visible light component is low, then image processing is not required; if the infrared component gain is less than the preset infrared component gain threshold R, then... GTH Then, image fusion is performed according to the rules.

[0131] If it is determined from step 120 or step 260 that there is a defocused area in the infrared image, then image fusion processing is required to update the current or subsequent output infrared image in order to improve the local defocusing phenomenon.

[0132] In step 270, the image fusion process is performed in the same manner as in steps 160110-160120 or 160210-160220.

[0133] In some exemplary embodiments, for each defocused block location, the black-and-white image block corresponding to the defocused block location in the enhancement region and the infrared image block corresponding to the defocused block location in the region to be enhanced are respectively fused in the following manner:

[0134] f R (x,y)=f C (x,y)*(1-κ)*λ+f ir (x,y)*κ*(1-λ);

[0135] Among them, fR (x,y) represents the image block after fusion at the location of the defocused block, f C (x,y) represents the black-and-white image block in the enhanced region corresponding to the location of the defocused block, f ir (x,y) represents the infrared image block in the region to be enhanced corresponding to the position of the defocused block;

[0136] This is the sharpness deviation coefficient corresponding to the out-of-focus block position;

[0137] R represents the gain percentage of the infrared component. GA R represents the red component gain of the black-and-white image block in the enhancement region corresponding to the defocused block location. GC The red component gain corresponds to the infrared image block in the region to be enhanced at the location of the defocused block.

[0138] As can be seen, the larger the calculated sharpness deviation ratio and the smaller the proportion of red component gain, the greater the degree of image defocus. The larger the proportion of the stored visible light black and white image in the fused image, or even all of it, in the final fused image. Conversely, the larger the proportion of the current infrared image.

[0139] This disclosure also provides an image quality improvement method when the camera is switched to night mode 2, such as... Figure 3 As shown, it includes:

[0140] Step 310: Obtain the third image;

[0141] Step 320: Obtain the out-of-focus area of ​​the infrared image;

[0142] Step 330: Determine if the out-of-focus area is empty. If it is, proceed to step 340; otherwise, proceed to step 350.

[0143] Step 340: Output the third image;

[0144] Step 350: Obtain the first image;

[0145] Step 360: Based on the out-of-focus area, determine the corresponding area to be enhanced in the third image;

[0146] Step 370: Determine the enhancement region in the first image based on the out-of-focus area;

[0147] Step 380: Use the enhanced region to perform a fusion process on the region to be enhanced to obtain the processed third image and output it.

[0148] In some exemplary embodiments, the out-of-focus area obtained in step 330 is calculated and determined by the camera according to step 260 after the camera completes the night switching. In some exemplary embodiments, after the night switching is completed, the infrared image is updated according to a preset period T, and the out-of-focus area is recalculated and updated based on the updated infrared image.

[0149] This disclosure also provides an electronic device, including:

[0150] One or more processors;

[0151] Storage device for storing one or more programs.

[0152] When the one or more programs are executed by the one or more processors, the one or more processors implement the image quality improvement method as described in any embodiment of this disclosure.

[0153] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, the program being implemented by a processor as the image quality improvement method described in any embodiment of this disclosure.

[0154] As can be seen, based on the image quality improvement scheme provided in this embodiment, according to the sharpness deviation value between the cached visible light black and white image, the infrared image without lights and the infrared image with lights on, and the red component gain value of each block of the infrared image, it is possible to determine whether there is local day and night defocus in the infrared image after the ICR is switched to night mode, and to determine the specific defocus area. Then, after the night mode is switched, the cached visible light image and the current infrared image are fused according to certain rules based on the sharpness deviation value and the red component gain ratio, and a frame of image with improved sharpness is output, which significantly improves the local day and night defocus problem.

[0155] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0156] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for improving image quality, characterized in that, include: Acquire the image captured by the camera in the third shooting scene, and denote it as the third image; Obtain the out-of-focus area of ​​the infrared image of the camera in the third shooting scene; Acquire the existing black and white image of the first shooting scene and denote it as the first image; Based on the defocused area, determine the corresponding area to be enhanced in the third image; Based on the defocused area, determine the corresponding enhancement area in the first image; Based on the degree of defocus in the region to be enhanced, the fusion coefficients of the enhanced region and the region to be enhanced are determined respectively. The enhanced region and the region to be enhanced are then fused according to the fusion coefficients to obtain the processed third image. The third shooting scene is the shooting scene with infrared fill light after the camera switches to night mode dual filter, and the first shooting scene is the shooting scene before the camera last switched to night mode dual filter. The out-of-focus area of ​​the infrared image in the third shooting scenario is determined by the following method: Following the same division rules, the existing black and white images in the first shooting scene, the infrared-free images in the second shooting scene, and the infrared images in the third shooting scene are divided into blocks to obtain multiple black and white image blocks, multiple infrared-free image blocks, and multiple infrared image blocks. Based on the black and white image block, the non-infrared image block, and the infrared image block corresponding to each identical block position, calculate the sharpness deviation ratio for each block position; determine the block positions with a sharpness deviation ratio greater than a set deviation threshold as out-of-focus block positions; The infrared image blocks corresponding to all the out-of-focus block positions constitute the out-of-focus region; The second shooting scenario is a shooting scenario without infrared fill light after the camera switches to night mode dual filters.

2. The image quality improvement method as described in claim 1, characterized in that, The location of the virtual focus block is also determined according to the following method: The locations of blocks where the sharpness deviation ratio is greater than the set deviation threshold are identified as suspected out-of-focus blocks. Calculate the red component gain of the infrared image block corresponding to each suspected out-of-focus block location, and determine the suspected out-of-focus block location where the red component gain is less than the set red component gain threshold as the out-of-focus block location.

3. The image quality improvement method as described in claim 1 or 2, characterized in that, The partitioning rule is: divide into M N blocks; The sharpness deviation ratio for each block location is calculated using the following method: in, This represents the sharpness deviation ratio at the m-th row and n-th column block position. This is the sharpness evaluation value for the black and white image block at the m-th row and n-th column position. The sharpness evaluation value is the infrared-free image block at the m-th row and n-th column block position. The sharpness evaluation value of the infrared image block at the m-th row and n-th column position; M and N are both integers greater than 1, m=1,…,M, n=1,…,N.

4. The image quality improvement method as described in claim 2, characterized in that, The red component gain of the image block corresponding to each block position is the average red component gain of the pixels contained in that image block; Among them, the red component gain of each pixel , This represents the grayscale value of the red component. This represents the grayscale value of the green component.

5. The image quality improvement method as described in claim 1, characterized in that, The step of acquiring the stored black-and-white image of the first shooting scene includes: Retrieve the last black and white image captured by the camera before the most recent night mode dual filter switch; or, Based on the gain value of the third image, select the black and white image with the closest gain value from a plurality of stored black and white images of the first shooting scene.

6. The image quality improvement method as described in claim 1 or 2, characterized in that, The region to be enhanced includes infrared image blocks in the third image that correspond to all out-of-focus block positions; the region to be enhanced includes black and white image blocks in the first image that correspond to all out-of-focus block positions. The process of fusing the enhanced region and the region to be enhanced according to the fusion coefficient to obtain the processed third image includes: For each out-of-focus block location, the black-and-white image block corresponding to that location in the enhancement area and the infrared image block corresponding to that location in the area to be enhanced are respectively fused together in the following manner: ； in, The image is divided into blocks after fusion at the location of the out-of-focus block. The location of the out-of-focus block corresponds to a black-and-white image block in the enhanced region. Let be the infrared image block in the region to be enhanced corresponding to the position of the defocused block; 'a' is the fusion coefficient of the black and white image block in the region to be enhanced corresponding to the position of the defocused block; and 'b' is the fusion coefficient of the infrared image block in the region to be enhanced corresponding to the position of the defocused block. Wherein, both 'a' and 'b' are numbers less than or equal to 1 and greater than or equal to 0. The higher the degree of defocus in the region to be enhanced, the larger 'a' and the smaller 'b'; conversely, the lower the degree of defocus in the region to be enhanced, the smaller 'a' and the larger 'b'. The image blocks after fusing all out-of-focus areas are merged with the image blocks of the non-enhanced areas in the third image to obtain the processed third image.

7. The image quality improvement method as described in claim 3, characterized in that, The region to be enhanced includes infrared image blocks in the third image that correspond to all out-of-focus block positions; the region to be enhanced includes black and white image blocks in the first image that correspond to all out-of-focus block positions. The process of fusing the enhanced region and the region to be enhanced according to the fusion coefficient to obtain the processed third image includes: For each out-of-focus block location, the black-and-white image block corresponding to that location in the enhancement area and the infrared image block corresponding to that location in the area to be enhanced are respectively fused together in the following manner: ； in, The image is divided into blocks after fusion at the location of the out-of-focus block. The location of the out-of-focus block corresponds to a black-and-white image block in the enhanced region. The location of the defocused block corresponds to the infrared image block in the area to be enhanced; This represents the fusion coefficient of the black-and-white image block in the enhancement region corresponding to the location of the out-of-focus block. The fusion coefficient is the infrared image block in the region to be enhanced corresponding to the position of the defocused block. This is the sharpness deviation coefficient corresponding to the position of the out-of-focus block; This represents the percentage of infrared component gain corresponding to the location of the defocused block. The red component gain corresponds to the black-and-white image block in the enhancement region at the location of the out-of-focus block. The red component gain of the infrared image block in the region to be enhanced corresponds to the location of the defocused block. The image blocks after fusing all out-of-focus areas are merged with the image blocks of the non-enhanced areas in the third image to obtain the processed third image.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the image quality improvement method as described in any one of claims 1-7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image quality improvement method as described in any one of claims 1-7.

Images