Infrared super-resolution dual light fusion method

Abstract

The application discloses an infrared super-resolution dual-light fusion method. The method performs super-resolution processing on an infrared image to obtain an infrared super-resolution image; according to a preset basic mapping relationship, determines the corresponding visible light source coordinates of each position in the target domain of the infrared super-resolution image, samples a visible light image according to the visible light source coordinates to obtain a mapped visible light image; performs structure-related search at a target scale and a half scale respectively to obtain a first translation amount and a second translation amount, and determines a correction translation amount according to the matching effectiveness of the two scales and the cross-scale consistency; determines a visible light structure residual according to the mapped visible light image, samples the visible light structure residual according to the correction translation amount, fuses the visible light structure residual to the infrared super-resolution image in a common effective area, and keeps the infrared super-resolution image unchanged outside the common effective area. Thus, the double edges, ghosting and texture misplacement caused by the basic mapping residual and unreliable translation estimation can be reduced.

Description

Technical Field

[0001] This application relates to the field of infrared image fusion technology, and in particular to an infrared super-resolution dual-light fusion method. Background Technology

[0002] Infrared images can reflect differences in thermal radiation, but their spatial resolution is usually lower than that of visible light images. Existing two-light fusion schemes can use basic two-light mapping relationships to overlay visible light contours or textures onto infrared images, but the basic mapping usually only corresponds to the calibration state; assembly and adjustment, temperature, mechanical stress, timing differences, distortion correction errors, and calibration residuals can all cause overall residual translation during operation, and direct overlay can easily produce double edges, ghosting, and texture misalignment.

[0003] In related technologies, single-scale correlation search is also susceptible to low texture, repetitive texture, cross-modal grayscale differences, and fine pseudo-structures in learned super-resolution. When the infrared structure is weak or the visible light mapping exceeds the limit, using unreliable displacement results will result in erroneous structures in the fusion region; if visible light structure modulation is turned off when matching fails, it will cause discontinuities in the display style.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of at least one of the above technical problems, this application provides an infrared super-resolution two-light fusion method to solve the problems of overall residual translation, insufficient single-scale matching reliability, discontinuous fusion process when matching fails, and invalid or out-of-bounds regions affecting the fusion results during the basic two-light mapping process.

[0006] This application provides an infrared super-resolution dual-light fusion method, including: S100, perform super-resolution processing on the infrared image at a preset super-resolution ratio to obtain an infrared super-resolution image; S200: Based on the preset basic mapping relationship, determine the visible light source coordinates corresponding to each position in the target domain of the infrared super-resolution image, and sample the visible light image according to the visible light source coordinates to obtain the mapped visible light image; S300, determine the common effective region based on the infrared effective region corresponding to the infrared super-resolution image and the visible light effective region corresponding to the mapped visible light image; S400: Based on the infrared super-resolution image and the mapped visible light image, perform structure correlation search at the target scale and half-scale respectively to obtain the first translation amount and the second translation amount, and determine the matching validity of the target scale and the half-scale respectively; determine the correction translation amount based on the matching validity of the target scale and the half-scale and the cross-scale consistency of the first translation amount and the second translation amount; wherein, when at least one scale matching is invalid or the first translation amount and the second translation amount do not meet the preset cross-scale consistency condition, the correction translation amount is determined to be zero translation; S500: Determine the visible light structural residual based on the mapped visible light image, sample the visible light structural residual based on the correction translation amount, and fuse the sampled visible light structural residual into the infrared super-resolution image within the common effective region to obtain the fused image; wherein, the infrared super-resolution image remains unchanged outside the common effective region.

[0007] This application reduces the impact of single-scale mismatches on residual translation estimation by performing global structure correlation searches at the target scale and half-scale respectively, and judging the matching validity and cross-scale consistency of the two scales.

[0008] The present application will be further described below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart of the infrared super-resolution dual-light fusion method provided in the embodiments of this application. Detailed Implementation

[0011] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the specific embodiments of this application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0012] Infrared images can reflect differences in thermal radiation within a target scene, while visible light images typically possess richer spatial structural information such as edges, contours, and textures. During dual-light fusion, the structural information in visible light images can be used to enhance the visual details of infrared images.

[0013] However, due to differences in installation location, imaging resolution, and optical parameters between infrared and visible light imaging devices, a basic mapping relationship between the two images usually needs to be established beforehand. Furthermore, even after establishing this basic mapping relationship, during actual equipment operation, factors such as adjustments, temperature variations, mechanical stress, timing differences, distortion correction errors, and calibration residuals can still affect the infrared and visible light images, resulting in a slight overall positional deviation between the mapped infrared and visible light images. If visible light structural information is directly used to fuse the infrared image, double edges, ghosting, or texture misalignment may occur at common strong edge locations.

[0014] Based on this, embodiments of this application perform super-resolution processing on the infrared image, mapping the visible light image to the target domain of the infrared super-resolution image, and estimating the positional deviation between the two images at the target scale and half-scale respectively. When the matching results at the two scales meet preset requirements, a correction translation amount is determined based on the translation results at the two scales; otherwise, the correction translation amount is reset to zero. Furthermore, the sampling position of the visible light structural residual is adjusted only using the correction translation amount, and the sampled visible light structural residual is fused into the infrared super-resolution image.

[0015] like Figure 1 As shown, one embodiment provides an infrared super-resolution dual-light fusion method, which may include the following steps: S100 to S500.

[0016] S100 performs super-resolution processing on the infrared image at a preset super-resolution ratio to obtain an infrared super-resolution image.

[0017] In step S100, the infrared image to be processed can be acquired and super-resolution processing can be performed on the infrared image to obtain an infrared super-resolution image. The spatial resolution of the infrared super-resolution image is higher than that of the input infrared image, and it serves as the target image used in subsequent two-light mapping, structure matching, and fusion processing. By unifying subsequent processing to the target coordinate domain of the infrared super-resolution image, direct structural position comparisons between images of different resolutions can be avoided, and the infrared super-resolution image can be used as the base image for the final fusion result.

[0018] S200: Based on the preset basic mapping relationship, determine the visible light source coordinates corresponding to each position in the target domain of the infrared super-resolution image, and sample the visible light image according to the visible light source coordinates to obtain the mapped visible light image; In step S200, a basic mapping relationship corresponding to the current dual-light imaging device can be obtained. This basic mapping relationship describes the geometric correspondence between the target domain of the infrared super-resolution image and the original visible light image. Based on this basic mapping relationship, corresponding visible light data can be obtained from the visible light image for each position in the target domain of the infrared super-resolution image, thus forming a mapped visible light image located in the same target coordinate domain. Therefore, in subsequent processing, the infrared super-resolution image and the mapped visible light image can be compared in structural position within a unified target domain.

[0019] S300: Determine the common effective region based on the infrared effective region corresponding to the infrared super-resolution image and the visible light effective region corresponding to the mapped visible light image.

[0020] In step S300, the effective infrared region corresponding to the infrared super-resolution image and the effective visible light region corresponding to the mapped visible light image can be determined separately, and a common effective region can be determined based on the effective infrared region and the effective visible light region. The common effective region can be understood as the target domain region that has effective infrared data in the infrared super-resolution image and can also obtain effective visible light data through the basic mapping relationship.

[0021] Subsequent structure-related searches and visible light structure fusion can be limited to a common effective region or a scale-effective region determined based on the common effective region. For locations outside the common effective region, since it cannot be guaranteed that both infrared and visible light information are effective, visible light structure information can be omitted, while keeping the image data of the corresponding location in the infrared super-resolution image unchanged.

[0022] S400: Based on the infrared super-resolution image and the mapped visible light image, perform structure correlation search at the target scale and half-scale respectively to obtain the first translation amount and the second translation amount, and determine the matching validity of the target scale and the half-scale respectively; determine the correction translation amount based on the matching validity of the target scale and the half-scale and the cross-scale consistency of the first translation amount and the second translation amount; wherein, when at least one scale matching is invalid or the first translation amount and the second translation amount do not meet the preset cross-scale consistency condition, the correction translation amount is determined to be zero translation.

[0023] In step S400, structural information from the infrared super-resolution image and the mapped visible light image can be acquired respectively, and structural correlation searches can be performed at the target scale and the half-scale respectively. The target scale reflects the finer structure in the current target domain, while the half-scale reduces the influence of some fine textures or pseudo-structures on the matching results by lowering the spatial scale. Performing a structural correlation search at the target scale yields a first translation amount; performing a structural correlation search at the half-scale yields a second translation amount.

[0024] Furthermore, the reliability of the matching results at the target scale and the half-scale can be determined separately. When the matching results at both scales are valid, it can be further determined whether the first translation and the second translation satisfy the preset cross-scale consistency condition. When both scales are matched validly and the first and second translations satisfy the cross-scale consistency condition, the correction translation can be determined based on the first and second translations. When at least one scale match is invalid, or the first and second translations do not satisfy the cross-scale consistency condition, the correction translation is determined to be zero translation.

[0025] It should be noted that the correction translation is used to correct the overall positional deviation that still exists after the basic mapping, and is not a replacement for the basic mapping relationship used in step S200.

[0026] S500: Determine the visible light structural residual based on the mapped visible light image, sample the visible light structural residual based on the correction translation amount, and fuse the sampled visible light structural residual into the infrared super-resolution image within the common effective region to obtain the fused image; wherein, the infrared super-resolution image remains unchanged outside the common effective region.

[0027] In step S500, visible light structural residuals, used to characterize local visible light edges, contours, and texture changes, can be extracted from the mapped visible light image. On the other hand, the sampling position of the visible light structural residuals is adjusted based on the correction translation amount obtained in step S400. That is, this embodiment does not use the correction translation amount to re-transform the infrared super-resolution image, nor does it use the correction translation amount to change the basic mapping relationship; instead, it only changes the reading position of the visible light structural residuals. Within the common effective region, the visible light structural residuals sampled according to the correction translation amount can be fused into the infrared super-resolution image; outside the common effective region, the infrared super-resolution image remains unchanged, thereby obtaining the final fused image.

[0028] It should be noted that the generation of visible light structural residuals and the determination of the correction translation amount can be performed separately. One path of data from the mapped visible light image can be used for the structural correlation search in step S400, and the other path of data can be used for the construction of visible light structural residuals in step S500. The two paths are finally combined during the visible light structural residual sampling and fusion process.

[0029] Through the above steps, the embodiments of this application can correct the position of the visible light structural information used for fusion when there is a small amount of basic mapping residual between the infrared super-resolution image and the mapped visible light image; and when the position deviation estimation lacks reliability, it will fall back to the basic mapping position for structural fusion, reducing the possibility of significant fusion misalignment caused by erroneous translation correction.

[0030] In some embodiments, S100 may include steps S110 to S120.

[0031] S110, obtain the preset learning super-resolution model.

[0032] The learning-based super-resolution model can use an existing super-resolution network or a super-resolution model trained specifically for current infrared imaging equipment. This application does not limit the specific network structure or training method of the learning-based super-resolution model.

[0033] S120: Input the infrared image into the learning super-resolution model and perform super-resolution processing according to the preset super-resolution ratio to obtain the infrared super-resolution image; wherein, the super-resolution ratio is 2 or 4.

[0034] Specifically, using a learned super-resolution model to perform super-resolution processing on infrared images can be represented as follows: ,in, For the input infrared image, For infrared super-resolution images, For the corresponding preset super-resolution ratio Learning-based super-resolution models, These are the model parameters for the learning-based super-resolution model.

[0035] The super-resolution ratio is 2 or 4, meaning that a 2x or 4x learning super-resolution model can be used depending on the equipment configuration. The preset super-resolution ratio remains constant during operation of the same equipment, and the model parameters of the learning super-resolution model remain fixed during operation.

[0036] In some embodiments, S200 may include steps S210 to S230.

[0037] S210, Obtain the basic mapping relationship corresponding to the preset super-resolution ratio. .

[0038] Basic mapping relationship Used to map the location in the target domain of an infrared super-resolution image to the source coordinates in a visible light image.

[0039] S220: For the position in the target domain of the infrared super-resolution image, determine the coordinates of the visible light source according to the basic mapping relationship, and sample the visible light image according to the coordinates of the visible light source to obtain the mapped visible light image.

[0040] For the location in the target domain of the infrared super-resolution image Its corresponding source coordinates in a visible light image can be represented as: Therefore, it can be done according to: This yields a mapped visible light image. Visible light image, For image sampling operations.

[0041] S230: When the coordinates of the visible light source exceed the effective range of the visible light image, the corresponding target domain position is determined as an invalid position for visible light sampling.

[0042] when When the target location is outside the effective range of a visible light image, the position within the target domain can be determined. This location was identified as an invalid visible light sampling point.

[0043] In some embodiments, the 2x and 4x super-resolution ratios can be pre-calibrated with their respective underlying mapping relationships. When calibrating the underlying mapping relationships, the processing conditions, such as the infrared preprocessing state, distortion correction state, super-resolution model, input size, and preset super-resolution ratio, can be kept consistent with those during device operation.

[0044] In some embodiments, S300 may include steps S310 to S350.

[0045] S310: Determine the effective infrared input mask based on the infrared image, and obtain the effective infrared mask of the target domain based on the effective infrared input mask and the super-resolution geometric mapping corresponding to the preset super-resolution magnification.

[0046] In step S310, the effective infrared input mask can be determined based on the effective pixel information of the infrared image. It should be noted that this embodiment does not limit the specific method for generating the effective infrared input mask. For example, the effective infrared input mask can be determined based on the pixel positions in the input infrared image that are still valid after bad pixel correction, the pixel positions within the effective imaging range of the sensor, and the pixel positions that have not saturated after automatic gain control. For example, positions that simultaneously meet the above valid conditions can be marked as valid positions, while positions that do not meet the corresponding conditions can be marked as invalid positions.

[0047] Subsequently, an effective mask can be established based on the infrared input. The actual distortion correction state and the super-resolution geometric mapping corresponding to the current preset super-resolution ratio will effectively mask the infrared input. Mapping the target domain onto the infrared super-resolution image yields the effective infrared mask for the target domain. .

[0048] S320: Determine the visible light source coordinates corresponding to each target domain position based on the basic mapping relationship, and determine the target domain position whose visible light source coordinates are within the effective range of the visible light image as the effective position, so as to obtain the visible light effective mask.

[0049] In step S320, the basic mapping relationship can be used as a basis. Determine the location of the target domain Whether the corresponding visible light source coordinates are within the effective range of the original visible light image. The target domain location where visible light data can be effectively sampled is determined as the effective location, thus obtaining the effective visible light mask. .

[0050] S330 obtains a common effective mask based on the target domain infrared effective mask and the visible light effective mask.

[0051] In step S330, the intersection of the effective infrared mask and the effective visible light mask in the target domain is calculated: Thus, a common and effective mask is obtained.

[0052] S340, according to the preset boundary margin, the common effective mask is etched to obtain the common effective area.

[0053] To prevent subsequent candidate displacement sampling, filtering, gradient calculation, and other neighborhood operations from accessing data outside the effective range, the common effective mask can be further eroded according to a preset boundary margin to obtain the common effective region. This boundary margin can include lateral boundary margins and longitudinal boundary margins. The lateral boundary margin can satisfy: ,in, Search for the maximum allowed lateral absolute displacement for the following candidate displacements. This provides the horizontal support radius required by subsequent image processing operators. The vertical boundary margin can satisfy: , Search for the maximum allowed longitudinal absolute displacement for the following candidate displacements. This is the support radius required in the vertical direction for subsequent image processing operators.

[0054] S350 uses the common effective region as the target scale effective region and performs conservative downsampling on the common effective region to obtain the half-scale effective region.

[0055] In step S350, the common effective region can be... As the effective region of the target scale Conservative downsampling is then performed based on the common effective region to obtain the half-scale effective region. .

[0056] For scale The number of effective pixels in the corresponding scale's effective region can be expressed as: .

[0057] Conservative downsampling can refer to determining a half-scale position as a valid position only when the target scale region corresponding to the half-scale position and its subsequent filtering, gradient, and candidate displacement sampling neighborhood are all within the valid region.

[0058] In some embodiments, S400 may include steps S401 to S403.

[0059] S401, acquire the infrared brightness map of the infrared super-resolution image and the visible light brightness map of the mapped visible light image, and normalize the infrared brightness map and the visible light brightness map.

[0060] In step S401, the infrared brightness map of the infrared super-resolution image and the visible light brightness map of the mapped visible light image can be obtained respectively, and the brightness data used for structure correlation search is normalized to the range of 0 to 1.

[0061] S402, at the target scale, Gaussian smoothing and Sobel gradient calculation are performed on the normalized infrared brightness map and visible light brightness map respectively to obtain the target scale infrared structure map and the target scale visible light structure map.

[0062] Understandably, regarding scale Brightness diagram Its structure diagram can be represented as follows: ,in, Gaussian smoothing was applied. Sobel gradient calculation in the horizontal direction For Sobel gradient calculation in the vertical direction, For absolute value operations, Absolute value operation is used to take the absolute value of the Sobel gradient response, thereby reducing the influence of the positive and negative gradient directions on the structural strength characterization, so that the structural diagram mainly reflects the edge and contour response strength at the current position.

[0063] Therefore, the infrared structure diagram at the target scale can be obtained. And the visible light structure diagram at the target scale The scale factor for the target scale can be set as follows: .

[0064] S403, at a half-scale, performs low-pass filtering and half-scale downsampling on the normalized infrared brightness map and visible light brightness map respectively, and performs Gaussian smoothing and Sobel gradient calculation on the downsampled brightness map to obtain the half-scale infrared structure map and the half-scale visible light structure map.

[0065] In step S403, low-pass filtering and half-scale downsampling can be applied to the normalized infrared brightness map and visible light brightness map, respectively. A Gaussian low-pass filter with a size of 7×7 and a standard deviation of 1 can be used, and half-scale downsampling can be performed based on area relationships.

[0066] Subsequently, Gaussian smoothing and Sobel gradient processing were applied to the half-scale brightness map to obtain the half-scale infrared structure map. and half-scale visible light structure diagram The scale factor at a half-scale can be set as follows: .

[0067] In some embodiments, S400, the structure-related search may include steps S404 to S408.

[0068] S404. Based on the maximum horizontal translation and the maximum vertical translation, a candidate displacement set is constructed, wherein the candidate displacements in the candidate displacement set are all in units of target domain pixels.

[0069] The target-scale effective region for target-scale structure correlation search and the half-scale effective region for half-scale structure correlation search can be determined based on the common effective region.

[0070] Furthermore, a candidate displacement set can be constructed based on the preset maximum lateral and longitudinal translation values: The candidate displacement can be searched in steps of one target domain pixel.

[0071] S405, based on each candidate displacement in the candidate displacement set, sample the target-scale visible light structure map at the target scale, and calculate the global zero-mean normalized cross-correlation value between the target-scale infrared structure map and the sampled target-scale visible light structure map.

[0072] S406. Based on the half-translation of each candidate displacement in the candidate displacement set, the half-scale visible light structure map is sampled at the half-scale, and the global zero-mean normalized cross-correlation value between the half-scale infrared structure map and the sampled half-scale visible light structure map is calculated.

[0073] Steps S405 and S406 above involve sampling the visible light structure map at different scales and then calculating the global zero-mean normalized cross-correlation value.

[0074] Specifically, regarding scale and any candidate displacement It can be performed in the corresponding scale effective area according to: The visible light structure map is sampled.

[0075] For target scale Therefore, according to: Perform sampling.

[0076] For half-scale Therefore, according to: Perform sampling.

[0077] When the corresponding position is a non-integer position, bilinear interpolation can be used to determine the structural response.

[0078] For scale The average structural response of the infrared structure map can be expressed as: .

[0079] For candidate displacement The average structural response of the sampling results corresponding to the visible light structure map can be expressed as: .

[0080] The global zero-mean normalized cross-correlation can be expressed as: , in, It is a preset small positive number, used to reduce the possibility of numerical anomalies caused by the normalization correlation denominator being zero or too close to zero.

[0081] For the same scale, the position of the infrared structure map does not change with the candidate displacement, therefore the mean infrared structure value... It can be calculated once.

[0082] The sampling positions of the visible light structure change corresponding to different candidate displacements, therefore the mean of the visible light structure... Calculate each candidate displacement separately.

[0083] S407, at the target scale, determine the first translation amount based on the global zero-mean normalized cross-correlation value corresponding to each candidate displacement in the candidate displacement set.

[0084] For the target scale, the cross-correlation value can be normalized to the global zero mean. The candidate displacement with the maximum value is determined as the first translation, which can be expressed as: max .

[0085] When multiple candidate displacements with the same maximum correlation exist, the candidate displacement that is closer to the zero vector can be selected first.

[0086] S408, at the half-scale, the second translation is determined based on the global zero-mean normalized cross-correlation value corresponding to the half-translation of each candidate displacement in the candidate displacement set.

[0087] For a half-scale, the cross-correlation value can be normalized to the global zero mean. The candidate displacement with the maximum value is determined as the second translation, which can be expressed as: max .

[0088] When multiple candidate displacements with the same maximum correlation exist, the candidate displacement that is closer to the zero vector can be selected first.

[0089] In some embodiments, determining the matching validity between the target scale and the half-scale in S400 may include steps S409 to S410.

[0090] S409, for the target scale or half-scale, determine the effective number of pixels, infrared structure energy, visible light structure energy, normalized correlation denominator, and maximum correlation for the corresponding scale; wherein, the infrared structure energy is determined based on the structural response of the infrared structure map at the corresponding scale within the effective region of the corresponding scale, the visible light structure energy is determined based on the structural response of the visible light structure map after sampling the corresponding translation amount within the effective region of the corresponding scale, the normalized correlation denominator is the normalized denominator of the global zero-mean normalized cross-correlation under the corresponding translation amount, and the maximum correlation is the global zero-mean normalized cross-correlation value corresponding to the translation amount at the corresponding scale.

[0091] After obtaining the first translation amount Second translation amount Then, the matching validity of the target scale and the half-scale can be judged separately.

[0092] For the target scale, the number of effective pixels at the target scale can be determined. Target-scale infrared structure energy Visible light structure energy at the target scale Target scale normalization related denominator Maximum correlation with target scale For a half-scale, the number of effective pixels at that half-scale can be determined. Half-scale infrared structure energy Half-scale visible light structure energy Half-scale normalized correlation denominator Maximum correlation with half-scale .

[0093] Specifically, in order to make the structural energy have a clear and calculable form, the average square value of the structural response of the structure diagram can be used to characterize the infrared structural energy and the visible light structural energy.

[0094] For example, target-scale infrared structure energy It can be represented as: .

[0095] Target-scale visible light structure energy It can be represented as: .

[0096] Half-scale infrared structure energy It can be represented as: .

[0097] Half-scale visible light structure energy It can be represented as: .

[0098] By using the mean-free structural response mean square value, the structural energy can primarily reflect the degree of spatial structural change in the effective region at the corresponding scale, and reduce the impact of the number of effective pixels and the overall baseline of the structural response on the structural energy statistics.

[0099] Furthermore, the first translation amount can be... The corresponding ZNCC normalization denominator is determined to be the target scale normalization correlation denominator: , And determine = The maximum relevance is the target scale.

[0100] For the half-scale, it can be determined that: , And determine = The maximum correlation is at the half-scale.

[0101] S410: When the number of effective pixels, infrared structure energy, visible light structure energy, normalized correlation denominator and maximum correlation all meet their respective preset thresholds, the corresponding scale match is determined to be valid; otherwise, the corresponding scale match is determined to be invalid.

[0102] Among them, when , , , , If all conditions are met, the target scale matching is considered valid; if any one condition is not met, the target scale matching is considered invalid.

[0103] Among them, when , , , , If all conditions are met, the half-scale match is considered valid; if any one condition is not met, the half-scale match is considered invalid.

[0104] By simultaneously determining the number of effective pixels, the structural information of each of the two modalities, the statistical stability of the correlation, and the final correlation degree, the risk of false acceptance caused by judging solely based on the maximum correlation value can be reduced.

[0105] In some embodiments, S400, determining the corrected translation amount based on the matching validity of the target scale and the half-scale and the cross-scale consistency of the first translation amount and the second translation amount may include steps S411 to S412.

[0106] S411, calculate the translation difference between the first translation amount and the second translation amount.

[0107] After determining the target scale and the matching validity of the half-scale, the first translation amount can be calculated. With the second translation amount Differences in translation between them: Where x is the horizontal component and y is the vertical component.

[0108] S412, when both the target scale and the half-scale are effectively matched, and the translation difference is less than or equal to the preset consistency threshold, the average of the first translation amount and the second translation amount is determined as the correction translation amount.

[0109] When the target scale and half-scale matching are valid, and The average of the first and second translations can be used as the corrected translation. .in, This is a preset consistency threshold. Since the first and second translation amounts can be represented using integer target domain pixels, their average result can include sub-pixel components.

[0110] When at least one scale match is invalid, or At that time, the correction translation amount is determined as: .

[0111] After verifying the matching validity at each of the two scales, a cross-scale consistency assessment is performed. This is primarily to address the issue that even if a single scale passes its own validity assessment, its translation result may still be miscalculated due to the unique structure of that scale. Since the target scale and the half-scale address the overall positional deviation between the same infrared super-resolution image and the mapped visible light image, the translation results obtained at both scales should have a certain degree of consistency, provided that the positional deviation can be reliably estimated at both scales.

[0112] Based on this, this embodiment only uses the average of the two translations as the correction translation amount when both scales are validly matched and the difference between the first and second translation amounts does not exceed a preset consistency threshold; otherwise, the correction translation amount is rolled back to zero. This ensures that the additional translation correction only takes effect when there is mutual verification between the two scales, reducing the possibility of single-scale anomalous results directly affecting the fused image and causing structural misalignment.

[0113] Meanwhile, zero translation is used when cross-scale consistency is not met, and subsequent visible light structure residual fusion is not turned off. Therefore, visible light structure enhancement at the base mapping location can be maintained while avoiding the use of unreliable additional displacements.

[0114] In some embodiments, determining the visible light structural residual based on the mapped visible light image in S500 may include steps S501 to S505.

[0115] S501, Obtain the visible light brightness of the mapped visible light image. .

[0116] S502, determine the fine-scale residual and the mesoscale residual based on the visible light brightness; Gaussian filtering of visible light brightness at different scales is performed, and local structural information at different spatial scales is extracted by the differences between the filtering results. The fine-scale residual can be expressed as: ; Mesoscale residuals can be expressed as: .

[0117] S503, based on preset scale weights and visible light structure gain Weighting the fine-scale residuals and the mesoscale residuals can be expressed as: .

[0118] S504, applying continuous noise attenuation to the weighted residuals, can be expressed as: , in, This represents the noise attenuation parameter.

[0119] S505 performs soft limiting on the residual after noise attenuation to obtain the visible light structure residual.

[0120] Applying soft limiting can be expressed as: , in, This is the soft limiting parameter.

[0121] The final visible light structure residual was determined to be: , Continuous noise attenuation can more significantly attenuate structural residuals with smaller amplitudes, thereby reducing the degree to which noise participates in the fusion process; soft limiting can reduce the possibility of overshoot in the fused brightness caused by larger structural residuals. Under the same device configuration and within the same common effective region, the same preset scale weights and total gain can be used without dynamically changing these parameters based on statistical results from different local regions within the image.

[0122] The aforementioned visible light structural residuals are mainly used to provide structural information such as local edges, contours, and textures from the visible light image to the infrared super-resolution image, rather than directly replacing or significantly altering the brightness information of the infrared super-resolution image with the overall brightness of the visible light.

[0123] Fine-scale residuals mainly extract relatively high-frequency detailed structures, while meso-scale residuals are used to supplement edge and contour information at a wider scale. By weighting residuals at different scales, visible light structures at different spatial scales can be taken into account.

[0124] Furthermore, continuous noise attenuation allows for more significant attenuation of structural residuals with smaller amplitudes, thereby reducing the likelihood of fine noise being introduced into infrared images as structural information; soft limiting, on the other hand, can continuously limit structural residuals with larger amplitudes, reducing the possibility of excessive brightness modulation caused by strong structural residuals.

[0125] Therefore, while preserving the effective local structure of visible light, it is possible to suppress small noise responses and excessive structural responses, making the visible light structural residuals used for fusion more suitable for local structural enhancement of infrared super-resolution images.

[0126] In some embodiments, in S500, the visible light structural residual is sampled and fused according to the correction translation amount, including steps S506 to S509.

[0127] S506, the visible light structure residual is sampled based on the correction translation amount and the position in the common effective region. When the position in the common effective region where the correction translation amount is superimposed is a non-integer position, bilinear interpolation is used for sampling. After obtaining the visible light structure residual and correction translation amount Then, the visible light structural residuals can be sampled according to the following formula: , in, This represents the structural residual obtained after adjusting the sampling position based on the correction translation amount.

[0128] when When the position is a non-integer position, bilinear interpolation can be used to determine the residual value at the corresponding position.

[0129] S507, acquire the infrared brightness of the infrared super-resolution image. and infrared chromaticity ; S508, for the location of the common effective region, the sampled visible light structure residual is superimposed on the infrared brightness while keeping the infrared chromaticity unchanged; For locations within a common effective region, the fused brightness can be determined using the following formula, while preserving the infrared chromaticity: , , in, This indicates saturation processing, used to limit the blended brightness to the effective brightness range corresponding to the current image processing chain.

[0130] S509: For locations outside the common effective area, the infrared super-resolution image remains unchanged.

[0131] For locations outside the common effective area, the infrared super-resolution image is directly preserved, i.e.: .

[0132] When the correction translation amount is: Sometimes, .

[0133] Therefore, zero-shift back-off does not mean turning off visible light structure fusion, but only cancels the additional shift correction relative to the base mapping position. At this time, the visible light structure residual can still participate in infrared brightness modulation according to its original position after the base mapping.

[0134] In the above steps, structural residual sampling and fusion methods are mainly used to avoid changing the already calibrated basic mapping relationship or moving the infrared super-resolution image itself in order to compensate for a small amount of residual position deviation during operation.

[0135] Since step S200 has already established the basic geometric correspondence between the two modes through the basic mapping relationship, the correction translation amount obtained in step S400 is only used to compensate for the small overall positional deviation that still exists after the basic mapping. This embodiment can correct the position of the visible light structure participating in the final fusion without changing the basic mapping relationship and the infrared super-resolution target canvas by only changing the sampling address of the visible light structure residual.

[0136] Simultaneously, within the common effective region, only the sampled visible light structural residuals are superimposed onto the infrared brightness, while maintaining the infrared chromaticity unchanged; for locations outside the common effective region, the infrared super-resolution image is directly preserved. Thus, the effect of visible light information can be limited to local structural enhancement within the common effective region, and the output of the infrared super-resolution image can be avoided by regions with ineffective visible light coverage.

[0137] When the correction shift amount is reduced to zero, the visible light structure residual still participates in the same fusion relationship according to the basic mapping position. Therefore, it is not necessary to switch to another fusion branch in the event of a matching failure, which helps to maintain the continuity of the processing path and the fusion output method.

[0138] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0139] In the detailed description of this specification, reference is made to the accompanying drawings, which form a part thereof, wherein like reference numerals always denote like parts, and wherein exemplary embodiments are shown by way of example that may be implemented. It should be understood that other embodiments may be utilized, and structural or logical changes may be made, without departing from the scope of this application. Therefore, the following detailed description should not be considered limiting.

[0140] The various operations in the specification may be described sequentially as multiple discrete actions or operations in a manner most conducive to understanding the claimed subject matter. However, the order of description should not be construed as implying that these operations must be sequentially related. Specifically, these operations may not be performed in the order presented. The described operations may be performed in a different order than in the described embodiments. Various additional operations may be performed in additional embodiments and / or the described operations may be omitted.

[0141] The above are merely preferred embodiments of this application and do not constitute any limitation on this application. Any person skilled in the art can make many possible variations and modifications to the technical solution of this application, or modify it into equivalent embodiments, without departing from the scope of the technical solution of this application. Therefore, all equivalent changes made based on the shape, structure, and principle of this application without departing from the content of the technical solution of this application should be covered within the protection scope of this application.

Claims

1. An infrared super-resolution dual-light fusion method, characterized in that, include: S100, perform super-resolution processing on the infrared image at a preset super-resolution ratio to obtain an infrared super-resolution image; S200: Based on the preset basic mapping relationship, determine the visible light source coordinates corresponding to each position in the target domain of the infrared super-resolution image, and sample the visible light image according to the visible light source coordinates to obtain the mapped visible light image; S300, determine the common effective region based on the infrared effective region corresponding to the infrared super-resolution image and the visible light effective region corresponding to the mapped visible light image; S400, based on the infrared super-resolution image and the mapped visible light image, perform structural correlation search at the target scale and half-scale respectively to obtain the first translation amount and the second translation amount, and determine the matching validity of the target scale and the half-scale respectively; The correction translation amount is determined based on the matching validity between the target scale and the half-scale, as well as the cross-scale consistency between the first translation amount and the second translation amount; wherein, when at least one scale matching is invalid or the first translation amount and the second translation amount do not meet the preset cross-scale consistency condition, the correction translation amount is determined to be zero translation; S500, the visible light structural residual is determined according to the mapped visible light image, the visible light structural residual is sampled according to the correction translation amount, and the sampled visible light structural residual is fused to the infrared super-resolution image within the common effective region to obtain a fused image; wherein, the infrared super-resolution image remains unchanged outside the common effective region.

2. The infrared super-resolution dual-light fusion method according to claim 1, characterized in that, S100 includes: Obtain a pre-defined learning-based super-resolution model; The infrared image is input into the learning super-resolution model and super-resolution is performed according to a preset super-resolution ratio to obtain the infrared super-resolution image; wherein the super-resolution ratio is 2 or 4.

3. The infrared super-resolution dual-light fusion method according to claim 1, characterized in that, S200 includes: when the coordinates of the visible light source exceed the effective range of the visible light image, the corresponding target domain position is determined as an invalid position for visible light sampling.

4. The infrared super-resolution dual-light fusion method according to claim 3, characterized in that, The S300 includes: The effective infrared input mask is determined based on the infrared image, and the effective infrared mask for the target domain is obtained based on the effective infrared input mask and the super-resolution geometric mapping corresponding to the preset super-resolution magnification. Based on the basic mapping relationship, the visible light source coordinates corresponding to each target domain position are determined, and the target domain positions whose visible light source coordinates are within the effective range of the visible light image are determined as effective positions, so as to obtain an effective visible light mask; Based on the target domain infrared effective mask and the visible light effective mask, a common effective mask is obtained; The common effective mask is etched according to the preset boundary margin to obtain the common effective region; The common effective region is taken as the target scale effective region, and conservative downsampling is performed on the common effective region to obtain a half-scale effective region.

5. The infrared super-resolution dual-light fusion method according to claim 1, characterized in that, The S400 includes: Obtain the infrared brightness map of the infrared super-resolution image and the visible light brightness map of the mapped visible light image, and normalize the infrared brightness map and the visible light brightness map; At the target scale, Gaussian smoothing and Sobel gradient calculation are performed on the normalized infrared brightness map and visible light brightness map respectively to obtain the target scale infrared structure map and the target scale visible light structure map. At the half-scale, the normalized infrared brightness map and visible light brightness map are subjected to low-pass filtering and half-scale downsampling, respectively. Gaussian smoothing and Sobel gradient calculation are then performed on the downsampled brightness map to obtain the half-scale infrared structure map and the half-scale visible light structure map.

6. The infrared super-resolution dual-light fusion method according to claim 5, characterized in that, In S400, the structure-related search includes: A candidate displacement set is constructed based on the maximum horizontal and vertical translations, wherein the candidate displacements in the candidate displacement set are all in units of the target domain pixels; Based on each candidate displacement in the candidate displacement set, the visible light structure map at the target scale is sampled at the target scale, and the global zero-mean normalized cross-correlation value between the infrared structure map at the target scale and the sampled visible light structure map at the target scale is calculated. Based on the half-translation of each candidate displacement in the candidate displacement set, the half-scale visible light structure map is sampled at the half-scale, and the global zero-mean normalized cross-correlation value between the half-scale infrared structure map and the sampled half-scale visible light structure map is calculated. At the target scale, the first translation amount is determined based on the global zero-mean normalized cross-correlation value corresponding to each candidate displacement in the candidate displacement set; At the half-scale, the second translation amount is determined based on the global zero-mean normalized cross-correlation value corresponding to the half-translation amount of each candidate displacement in the candidate displacement set.

7. The infrared super-resolution dual-light fusion method according to claim 6, characterized in that, In step S400, determining the matching validity between the target scale and the half-scale includes: For the target scale or the half-scale, determine the effective number of pixels, infrared structure energy, visible light structure energy, normalized correlation denominator, and maximum correlation for the corresponding scale; wherein, the infrared structure energy is determined based on the structural response of the infrared structure map at the corresponding scale within the effective region of the corresponding scale, the visible light structure energy is determined based on the structural response of the visible light structure map after sampling the corresponding translation amount within the effective region of the corresponding scale, the normalized correlation denominator is the normalized denominator of the global zero-mean normalized cross-correlation under the corresponding translation amount, and the maximum correlation is the global zero-mean normalized cross-correlation value corresponding to the translation amount at the corresponding scale. When the effective number of pixels, infrared structure energy, visible light structure energy, normalized correlation denominator, and maximum correlation all meet their respective preset thresholds, the corresponding scale match is determined to be valid; otherwise, the corresponding scale match is determined to be invalid.

8. The infrared super-resolution dual-light fusion method according to claim 7, characterized in that, In step S400, determining the corrected translation amount based on the matching validity between the target scale and the half-scale, and the cross-scale consistency between the first translation amount and the second translation amount, includes: Calculate the translation difference between the first translation and the second translation; When both the target scale and the half-scale are effectively matched, and the translation difference is less than or equal to a preset consistency threshold, the average of the first translation amount and the second translation amount is determined as the corrected translation amount.

9. The infrared super-resolution dual-light fusion method according to claim 1, characterized in that, In step S500, determining the visible light structural residual based on the mapped visible light image includes: Obtain the visible light brightness of the mapped visible light image; Determine the fine-scale residual and the mesoscale residual based on the visible light brightness; The fine-scale residual and the mesoscale residual are weighted according to a preset scale weight and visible light structure gain; Continuous noise attenuation is applied to the weighted residuals; The visible light structure residual is obtained by soft-limiting the residual after noise attenuation.

10. The infrared super-resolution dual-light fusion method according to claim 9, characterized in that, In step S500, sampling and fusing the visible light structural residual based on the correction translation amount includes: The visible light structure residual is sampled based on the correction translation amount and the position in the common effective region. When the position in the common effective region where the correction translation amount is superimposed is a non-integer position, bilinear interpolation is used for sampling. Obtain the infrared luminance and infrared chromaticity of the infrared super-resolution image; For the location of the common effective region, the sampled visible light structure residual is superimposed on the infrared brightness, while keeping the infrared chromaticity unchanged; For locations outside the common effective region, the infrared super-resolution image remains unchanged.

Images