A fusion method based on multispectral non-registered images

Abstract

The application discloses a fusion method based on multispectral non-registration images, which comprises the following steps: after visible light images and thermal infrared images are preprocessed, the visible light images are subjected to large core mean filtering to generate a weight map based on brightness, and a preliminary fusion image is obtained by using a nonlinear function to adjust and weight fusion; then, a significant target mask is extracted based on the brightness and contrast information of the infrared image, and non-registration fusion is carried out through twice expansion of different types to eliminate artifacts generated during the fusion of the significant target; finally, the connected domain in the significant target mask is divided into three categories, and the local brightness and contrast of the fusion image are adjusted by mean and variance transmission of the non-registration tolerance area, so that the result is clear and natural. The application can effectively highlight the thermal target in the infrared image while retaining the texture details of the visible light image, can effectively eliminate the artifacts generated by the non-registration problem, and can improve the quality of the fusion image and generate a result with natural visual observation.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for unregistered fusion of multispectral images. Background Technology

[0002] Multispectral image fusion technology plays a crucial role in the field of vision. Visible light sensors describe rich image textures but are susceptible to environmental influences; infrared sensors can capture the feature information of prominent targets in dark or foggy environments. Therefore, the fusion of visible light and infrared images can compensate for the shortcomings of a single sensor. However, due to the different positions and shooting angles of the two sensors, the paired images captured exhibit deregistration. This deregistration typically manifests as artifacts such as ghosting and halos in the fused image, severely degrading its quality. Although this problem can be suppressed through registration methods, precise registration requires significant computational time and resources, and pixel deviations are still unavoidable after registration. Therefore, this invention aims to employ a fusion method based on preliminary registration, balancing the visual appeal and information richness of the fused image while tolerating a certain degree of pixel offset. This minimizes artifacts caused by deregistration during the fusion process, enabling the generated fused image to highlight prominent targets in the infrared image and produce a natural-looking, artifact-free image.

[0003] Chinese invention patent CN108765358A discloses a method for dual-light fusion of visible and infrared light, as well as a plug-in thermal imager system. The system employs a Laplacian pyramid for layer-by-layer fusion, replacing the Laplacian pyramid obtained from decomposing the infrared image with a fused pyramid containing visible light details, and then reconstructing the fused pyramid to obtain the fused image. However, the fused image generated by this system requires a large amount of computation, fails to highlight salient targets in the infrared image, and does not consider artifacts caused by non-registration during the fusion process.

[0004] Chinese invention patent CN111539902A discloses an image processing method, system, device, and computer-readable storage medium. The method first performs low-pass filtering on a visible light image to obtain detail information, then further filters to obtain detail intensity, and uses the reciprocal of the detail intensity to obtain a detail gain, weighted and fused to the corresponding infrared image. This method produces a fused image with poor visual quality, while also avoiding artifacts caused by non-registration issues in the fused image. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art by designing a multispectral image non-registration fusion method.

[0006] A fusion method based on multispectral unregistered images comprises three parts: after image preprocessing, a preliminary fused image is formed by weighting the brightness distribution; after extracting the mask of salient targets in the infrared image, dilation fusion is used to eliminate unregistered artifacts; and finally, local brightness adjustment is used to obtain a natural fusion result.

[0007] The specific implementation steps of the method are as follows:

[0008] Step 1: Image preprocessing. The infrared image is subjected to transmission transformation using the transmission transformation matrix obtained by the checkerboard calibration method, so that the infrared image and the visible light image are initially registered.

[0009] Step 2: Process the preprocessed infrared and visible light images as follows:

[0010] 2.1 A large kernel mean filter is used to perform mean filtering on the visible light image to obtain a weighted map of the brightness distribution of the filtered visible light image;

[0011] 2.2 A nonlinear function with appropriate parameter settings is used to process the luminance distribution weight map to obtain the corrected luminance distribution weight map;

[0012] 2.3 Using the corrected brightness distribution weight map obtained in step 2.2, the preprocessed infrared image and visible light image are weighted and fused to obtain a preliminary fused image;

[0013] Step 3: Extract the salient target mask from the infrared image following these steps:

[0014] 3.1 Extracting a brightness target weight map based on infrared image brightness;

[0015] 3.2 Extraction of contrast target weight map based on infrared image contrast;

[0016] 3.3 The luminance target weight map and the contrast target weight map are fused to generate a target weight map;

[0017] 3.4 Obtain a salient target mask by removing noise, discontinuous short line segments, and other small connected regions;

[0018] Step 4: Perform fusion of non-registered salient targets according to the following steps:

[0019] 4.1 Obtain each connected component in the salient target mask image. Based on the set pixel deviation factor, perform a first-stage dilation operation with equal weight on the outermost pixels of each connected component. Specifically: first, determine the pixel deviation factor based on the scene disparity. The greater the scene parallax, the greater the pixel deviation factor. Larger (can be selected as needed) (Value selection). Use a filter kernel to process the connected components along the outermost pixel. First dilation with a weight of 1; pixel deviation factor This ensures that pixel deviations caused by non-registration will not result in artifacts within this tolerance range;

[0020] 4.2 Obtain the dilation result from step 4.1. Based on the set transition factor, perform a second dilation operation on the outermost edge pixels of each connected component with gradually decreasing weights to obtain a weight map of the non-registered salient target.

[0021] 4.3 The preliminary fused image obtained in step 2 and the preprocessed infrared image are weighted and fused based on the unregistered salient target weight map to generate a fused image;

[0022] Step 5: Use local brightness adjustment to compensate for the brightness and contrast of the merged image in different areas to obtain the final natural merged image.

[0023] In step 2, this invention uses a nonlinear function to correct the brightness distribution weight map, dynamically adjusting the amount of visible light image information added to different brightness regions of the infrared image. It then generates a preliminary fused image that highlights salient targets in the infrared image while preserving texture details in the visible light image through a brightness distribution weighting method. This is simpler and faster than existing image pyramid and wavelet transform fusion techniques. Existing non-registration techniques often use deep learning methods to design iterative modules for registration, thus requiring significant computational time and resources. In step 4, this invention uses two dilation operations with different weights to eliminate artifacts that may occur during salient target fusion. The calculation process is simple and can tolerate pixel shifts within a certain range, suppressing non-registration artifacts generated during fusion. In step 5, this invention divides the salient target mask connected regions and compensates for local brightness and contrast in the fused image through mean and variance propagation of the non-registration tolerance region, ultimately generating a natural-looking, clear, artifact-free fused image. This invention helps reduce computational and hardware costs, promoting the development of multispectral image non-registration fusion. Attached Figure Description

[0024] Figure 1 This is a flowchart of a multispectral image unregistration fusion method implemented in this invention.

[0025] Figure 2 This is a diagram illustrating the visible light image and infrared image fusion process according to an embodiment of the present invention.

[0026] Figure 3 This is a graph showing the significant target connected component partitioning in an embodiment of the present invention. Detailed Implementation

[0027] The following description, in conjunction with specific embodiments and accompanying drawings, provides further details.

[0028] Step 1: As per the process Figure 1 As shown, due to the different positions and viewing angles of the binocular cameras, the infrared camera and the visible light camera are calibrated in advance using the checkerboard calibration method to obtain the transmission transformation matrix, and the acquired infrared image is preprocessed by performing transmission transformation.

[0029] Step 2: As Figure 2 As shown in (a) and (b), the infrared image after transmission transformation and the visible light image of the same frame in the scene captured by the camera are fused based on the brightness distribution.

[0030] Since visible light image modalities are more suitable for visual perception and contain rich texture details, this invention assigns greater weight to pixel areas with sufficient brightness in the visible light image during the fusion process, and supplements the remaining darker pixel areas with almost no texture details with greater weight from the infrared image.

[0031] 2.1 Generate a brightness distribution weight map of a visible light image.

[0032] This invention first focuses on visible light images. Mean filtering is performed to obtain the brightness distribution weight map of the visible light image. The formula is as follows:

[0033]

[0034] in The size is at least Mean filter, This represents a two-dimensional filtering operation. Then, for... Use maximum-minimum normalization to make its range... :

[0035]

[0036] In the above formula , These are operations to retrieve the maximum and minimum values, respectively.

[0037] 2.2 A nonlinear function is used to correct the normalized brightness distribution weight map.

[0038] In order to avoid In visible light images Darker pixel areas produce large infrared images. The weights cause the fusion result to exhibit two distinct modal differences, which correspond to the fusion process. Appropriate weighting should be added to areas with low weight. Weight to reduce The modal behavior, to a certain extent, is equivalent to introducing the texture information that may exist in the visible light region. Therefore, this invention designs the following nonlinear correction function:

[0039]

[0040] Parameters in the above formula Used to control the maximum value of nonlinear functions. Characterizes the steepness of the rise in a nonlinear function. Represents the threshold of the independent variable, This is a visible light image information compensation coefficient, used to balance the amount of visible light image information added to the high-weighted infrared image regions. The parameter value is related to the specific scene. In this invention, scenes with a normalized visible light image brightness mean not higher than 0.3 are considered low-brightness scenes. Low-brightness scenes include nighttime environments and low-light environments. For scenes with low visible light image brightness, such as nighttime, the brightness distribution weight map generated by mean filtering... With lower weights, in order to improve the modal representation of visible light images and make the initial fusion result appear more natural, a nonlinear function is used to enhance the performance. The weight should be larger in this case. and and using smaller , The values, and the specific range of values ​​for each parameter are as follows: , , , Conversely, a smaller one should be used. and and use larger The values, and the specific range of values ​​for each parameter are as follows: , , , This invention takes a nighttime scene as an example. , , .

[0041] 2.3 Utilizing the generated Weighted fusion is performed to obtain a preliminary fused image.

[0042] The brightness distribution weight map corresponding to the infrared image is .use right and Perform the following weighted fusion:

[0043]

[0044] In the above formula For the generated preliminary fused image, such as Figure 2 As shown in (c).

[0045] Step 3: Extract the salient target mask from the infrared image.

[0046] 3.1 Extract the brightness target weight map of the infrared image using the following formula :

[0047]

[0048] in, The set brightness threshold, This represents the brightness gain coefficient. The minimum value operation aims to prevent negative values. The brightness threshold was determined experimentally using this invention. It can eliminate interference from low-brightness objects.

[0049] 3.2 Extract the contrast target weight map of the infrared image according to the following formula. :

[0050]

[0051] in For standard deviation filtering kernel, This represents a two-dimensional filtering operation on an image. The set contrast threshold, This represents the contrast gain coefficient. The contrast threshold was determined experimentally using this invention. The interference of weak contrast objects is eliminated, and the brightness and contrast of prominent objects are re-stretched to extract prominent targets.

[0052] 3.3 Infrared Image Brightness Weighting Map Contrast weighting map The fusion method using addition is represented as follows:

[0053] .

[0054] The minimum and maximum value operations are to ensure that the pixel range is within... Inside, The generated salient target fusion weight map is as follows: Figure 2 As shown in (d).

[0055] 3.4 pairs Processing to obtain a salient target mask image :

[0056] This invention uses a segmentation threshold ,right Binary segmentation is performed to obtain the mask. Furthermore, determine the connected component area threshold. Remove By removing noise from connected components with an area smaller than a threshold of 12, a salient target mask image is obtained. , where the threshold The value of depends on the specific scenario and varies slightly depending on the scenario. Generally speaking, there is more noise in nighttime environments and more environmental information in complex scenarios, so there is more noise and small connected components that need to be removed. In this case, the threshold value will be higher. A slightly larger value needs to be set.

[0057] Step 4: Perform fusion of non-registered salient targets according to the following method:

[0058] 4.1 pairs Encoding the connected components in the middle, denoted as ,in The number of connected components for the salient target. First, determine the pixel bias factor based on the scene disparity. sequentially for the salient target connected components Perform along the outermost pixel The second expansion with equal weights generates the connected component after the first expansion. The expression is as follows:

[0059]

[0060] In the above formula For salient target connected components, For filter , For the number of expansions with a weight of 1, the method used in this invention is... Pixel deviations caused by non-registration will not produce artifacts within this tolerance range. For expansion operation, For connected components The result of the expansion.

[0061] 4.2 Then determine the scene transition factor according to the scene type. sequentially for connected components Perform along the outermost pixel The expansion process continues with gradually decreasing weights until the weights decay to zero, generating the connected component after the second expansion. The expression is as follows:

[0062]

[0063] In the above formula For the connected components after the first expansion, the present invention adopts... Scene transition factor The presence of this ensures a uniform transition at the edges after merging salient targets. For connected components The result of the expansion.

[0064] Unregistered salient target weight map For connected components set The constructed mask has a value range of 1. .

[0065] Connected components in real-world scenarios Often irregular, this invention uses Figure 3 Taking a circular connected region as an example, the dilation operation is described (the dilation process is the same regardless of whether the shape of the connected region is regular). After the above two dilation operations, the... It is divided into three regions from the inside out, namely the salient target mask area. Non-registration tolerance zone Background transition area These correspond to the connected components in the salient target mask. Use expansion factor The region generated by the first dilation operation (this region can tolerate and eliminate artifacts caused by non-registration), and the dilation factor are used. The region generated by the second dilation operation (this region can make the edges of the merged significant targets have a uniform and natural transition).

[0066] 4.3 Weighted Map Based on Unregistered Satisfactory Targets Weighted generation of fused images:

[0067]

[0068] in This is a gain factor for significant targets, used to highlight significant targets. For the initial image fusion Weighted graph The resulting fused image.

[0069] Step 5: Apply a local modal compensation method based on regional brightness and contrast to the fused image. Perform regional brightness and contrast compensation.

[0070] Although the non-registration tolerance zone It eliminated the artifacts caused by fusion, but introduced... Local modal information will be included in the fused image. This creates a bright, jarring "halo" around a prominent target. To address this issue, the present invention uses the following formula sequentially... Around the salient target, especially in the non-registration tolerance region that produces halos. Perform modal adjustment:

[0071]

[0072] In the above formula, Representative fused image A connected component Non-registration tolerance region Representative fused image A connected component Regional mean Representative fused image A connected component Regional variance. These are the mean and variance of the corresponding region in the visible light image. This represents the connected components of the fused image after local modality compensation. Based on this, the fused image can be eliminated. The abruptness of a bright "halo" around a prominent target is eliminated, resulting in a visually natural and integrated image.

[0073] Step 5 of the present invention can […]. The overall brightness and contrast around the salient target are similar to the surrounding environment, thus achieving a better modal transition and generating the final fused image. Image fusion like Figure 2 As shown in (e).

Claims

1. A fusion method based on multispectral unregistered images, characterized in that, The method includes the following steps: S1: Image preprocessing. The transmission transformation matrix is ​​obtained using the checkerboard calibration method. The transmission transformation matrix is ​​then used to perform a transmission transformation on the obtained infrared image, so that the infrared image after transmission transformation is initially registered with the visible light image of the same frame in the scene captured by the camera. S2: Process the visible light image and the preprocessed infrared image as follows: S21: Apply a large kernel mean filter to the visible light image to obtain a weighted brightness distribution map of the filtered visible light image; the size of the large kernel mean filter is at least [value missing]. ; S22: The luminance distribution weight map is processed using a nonlinear function to obtain the corrected luminance distribution weight map; S23: Using the corrected brightness distribution weight map obtained in step S22, the preprocessed infrared image and visible light image are weighted and fused to obtain a preliminary fused image; S3: Extract the salient target mask image from the infrared image processed in step S1 according to the following steps: S31: Extract the brightness target weight map based on the brightness of the infrared image processed in step S1; S32: Extract the contrast target weight map based on the contrast of the infrared image processed in step S1; S33: Fuse the luminance target weight map and the contrast target weight map to generate a target weight map; S34: Obtain a salient target mask by removing noise and connected components with an area smaller than a threshold; S4: Perform fusion of non-registered salient targets according to the following steps: S41: Obtain each connected region in the salient target mask image, and perform a first dilation operation with the same weight on the outermost pixel of each connected region according to the set pixel deviation factor. S42: Obtain the dilation result of step S41, and perform a second dilation operation on the outermost edge pixels of each connected component with gradually decreasing weight according to the set transition factor to obtain the non-registered salient target weight map. S43: Based on the unregistered salient target weight map, the preliminary fusion image obtained in step S2 and the infrared image preprocessed in step S1 are weighted and fused to generate a fusion image; S5: Compensate for regional brightness and contrast in the fused image to obtain the final fused image.

2. The fusion method based on multispectral unregistered images according to claim 1, characterized in that, Step S22 uses a nonlinear function to correct the normalized luminance distribution weight map to obtain the visible light luminance distribution weight map, based on the following formula: in, This is a weighted map of the brightness distribution in a visible light image, with parameters... Used to control the maximum value of nonlinear functions. Characterizes the steepness of the rise in a nonlinear function. Represents the threshold of the independent variable. The visible light image information compensation coefficient; parameter , , and The value depends on the scenario: For low-light scenes, a larger [size] should be used. and Value, and use the smaller one. and value; , , , The range of values ​​for are as follows: , , , The low-brightness scene refers to a scene where the average brightness of the normalized visible light image is not higher than 0.

3. For scenes with a brightness higher than 0.3, a smaller value should be selected. and Value, and use a larger one and value; , , , The range of values ​​for are as follows: , , , .

3. The fusion method based on multispectral unregistered images according to claim 1, characterized in that, Specifically, step S31 involves extracting the brightness target weight map of the infrared image using the following formula. : in, The set brightness threshold, Indicates the brightness gain coefficient; This is an infrared image.

4. The fusion method based on multispectral unregistered images according to claim 1, characterized in that, Specifically, step S41 involves: first determining the pixel deviation factor based on the scene disparity magnitude. Use a filter kernel to filter the connected components along the outermost pixels. The second-weighted expansion generates connected components.

5. The fusion method based on multispectral unregistered images according to claim 1, characterized in that, Step S42 specifically involves: determining the scene transition factor according to the scene type. According to the set transition factor Perform on the outermost edge pixels of each connected component The weights gradually decrease as the expansion continues until the weights decay to 0, thus obtaining a weight map of the non-registered salient target.

6. The fusion method based on multispectral unregistered images according to claim 1, characterized in that, In step S5, a local modal compensation method based on regional brightness and contrast is proposed to compensate for regional brightness and contrast in the fused image. Specifically, this method involves compensating for regional brightness and contrast in the fused image by adjusting the connected components. Divided into prominent target mask areas from the inside out Non-registration tolerance zone Background transition area Three areas, among which Connected components in a salient target mask This is the region generated by the first dilation operation. The region generated by the second dilation operation; based on the corresponding region in the visible light image. The mean and variance of the regions are calculated using the following formula for the fused image. of Adjusting brightness and contrast in different areas: In the above formula, Representative fused image The non-registration tolerance region generated by the first expansion operation of a connected component. Representative fused image A connected component Regional mean Representative fused image A connected component Regional variance This corresponds to the visible light image. The mean and variance of the region This represents a connected region of the fused image after local modality compensation.

Images