An infrared image enhancement method and system based on a deep learning convolutional neural network

Abstract

This application provides an infrared image enhancement method and system based on deep learning convolutional neural networks, relating to the field of infrared image enhancement technology. The method includes: acquiring infrared and visible light images of a target monitoring area and constructing a feature transformation space; using a target feature mapping network to map the infrared image from a second feature domain to an intermediate feature domain to obtain a first intermediate feature, and then mapping it back to the first feature domain to obtain a first mapped image; using the same network to map the visible light image from the first feature domain to an intermediate feature domain to obtain a second intermediate feature, and then mapping it back to the second feature domain to obtain a second mapped image; determining a first feature difference between the first mapped image and the visible light image, and determining a second feature difference between the second mapped image and the infrared image; and enhancing the infrared image based on the two feature differences to obtain a target-enhanced infrared image. This solves the technical problem of poor reliability in infrared image enhancement based on single-modality processing in related technologies.

Description

Technical Field

[0001] This application relates to the field of infrared image enhancement technology, and in particular to an infrared image enhancement method and system based on deep learning convolutional neural networks. Background Technology

[0002] With the development of computer vision technology, infrared imaging is increasingly widely used in night vision surveillance, security early warning, industrial inspection and other fields. Infrared image enhancement, as a key technology for improving infrared imaging quality, directly affects the performance of downstream vision tasks.

[0003] In related technologies, infrared image quality is mainly improved through image grayscale transformation and local information enhancement. Specifically, firstly, multi-scale grayscale histogram analysis is performed on the infrared image to obtain the brightness distribution characteristics and grayscale statistics of the image at different spatial scales. Then, a nonlinear grayscale mapping function is designed based on global grayscale features, and adaptive weight coefficients are constructed by combining the statistical characteristics of local regions. The parameters of the mapping function are dynamically adjusted. Next, the image is partitioned, and the enhancement intensity is determined according to the local features of each partition. Then, grayscale transformation and contrast enhancement processing are performed on each partition image based on the adjusted mapping function. Finally, the processed partition images are fused, and the image edge regions are smoothed to obtain the enhanced infrared image.

[0004] However, due to the inherent limitations of infrared imaging, relying solely on the grayscale information of the image itself for enhancement is clearly insufficient. On the one hand, multi-scale grayscale analysis and local feature extraction in related technologies struggle to accurately identify key target structures in the image, making it easy for the enhancement process to destroy the original features of the image. On the other hand, adaptive weight adjustment based on preset rules lacks an understanding of the semantic information of the image, making it difficult to achieve effective enhancement while maintaining the true features of the image. Especially under complex backgrounds and low signal-to-noise ratio conditions, problems such as blurred image contours and loss of details often occur, resulting in poor reliability of infrared image enhancement based on single-modality processing in related technologies. Summary of the Invention

[0005] This application provides an infrared image enhancement method and system based on deep learning convolutional neural networks, which is used to improve the reliability of infrared image enhancement.

[0006] In a first aspect, this application provides an infrared image enhancement method based on a deep learning convolutional neural network, applied to the aforementioned infrared image enhancement system. The method includes: upon acquiring an infrared image and a visible light image of a target monitoring area, constructing a feature transformation space, wherein the infrared image and the visible light image are spatiotemporally aligned registered images of the target monitoring area; the feature transformation space includes a first feature domain for storing the visible light image, a second feature domain for storing the infrared image, and an intermediate feature domain for storing information shared by the first and second feature domains; using a target feature mapping network to map the infrared image from the second feature domain to the intermediate feature domain to obtain a first intermediate feature, mapping the first intermediate feature back to the first feature domain to obtain a first mapped image; and using the target feature mapping network to map the visible light image from the first feature domain to the intermediate feature domain to obtain a second intermediate feature, mapping the second intermediate feature back to the second feature domain to obtain a second mapped image; determining a first feature difference between the first mapped image and the visible light image, and determining a second feature difference between the second mapped image and the infrared image; and performing enhancement processing on the infrared image based on the first feature difference and the second feature difference to obtain a target-enhanced infrared image.

[0007] By employing the aforementioned technical solution, a feature transformation space is constructed using visible light and infrared images, including a first feature domain, a second feature domain, and an intermediate feature domain, providing a basic framework for feature mapping of images of different modalities. Based on this, the target feature mapping network can achieve bidirectional feature mapping transformation between infrared and visible light images. By calculating the feature differences between the mapped image and the original image, a reliable evaluation basis can be provided for subsequent enhancement processing. This enhancement method based on dual-modal information and feature mapping not only fully utilizes the detailed information in the visible light image to compensate for the deficiencies of the infrared image, but also ensures the controllability of the enhancement process through the calculation of feature differences, thereby achieving high-quality enhancement of infrared images and effectively avoiding detail loss and over-enhancement in traditional single-modal enhancement methods. This solves the technical problem of poor reliability in infrared image enhancement based on single-modal processing in related technologies, achieving the technical effect of improving the reliability of infrared image enhancement.

[0008] Optionally, after acquiring infrared and visible light images of the target monitoring area, a feature transformation space is constructed, specifically including: inputting the infrared and visible light images into a preset feature extraction network to obtain infrared deep feature maps and visible light deep feature maps output by the preset feature extraction network; performing texture feature processing on the visible light deep feature map to obtain a first feature domain, wherein the first feature domain has visible light texture detail characteristics; performing thermal intensity feature processing on the infrared deep feature map to obtain a second feature domain, wherein the second feature domain has infrared thermal radiation distribution characteristics; performing feature channel splicing and convolution fusion processing on the infrared deep feature map and the visible light deep feature map to extract cross-modal common structural features to obtain an intermediate feature domain, wherein the intermediate feature domain includes the common structural characteristics of the infrared image and the visible light image.

[0009] By employing the aforementioned technical solution, deep feature extraction is performed on infrared and visible light images using a pre-defined feature extraction network. Then, texture feature processing and thermal intensity feature processing are performed separately to obtain a first feature domain with visible light texture details and a second feature domain with infrared thermal radiation distribution characteristics. Specifically, by performing feature channel concatenation and convolution fusion on the infrared and visible light deep feature maps, cross-modal shared structural features can be extracted to obtain an intermediate feature domain containing the shared structural characteristics of both images. This depth-based feature-based domain construction method effectively separates and preserves the feature attributes of images from different modalities. Simultaneously, the intermediate feature domain can establish the correlation between the two modalities, providing a more accurate feature representation for subsequent feature mapping.

[0010] Optionally, texture feature processing is performed on the visible light deep feature map to obtain the first feature domain. Specifically, this includes: using multiple convolutional layers of different scales to extract multi-scale features from the visible light deep feature map to obtain a texture feature map, wherein the texture feature map includes global texture structure features and local texture detail features; performing texture gradient analysis on the global texture structure features to generate a texture gradient statistical histogram; determining a texture segmentation threshold based on the peak distribution of the texture gradient statistical histogram; dividing the texture feature map into M texture regions based on the texture segmentation threshold and the global texture structure features, where M is a positive integer; performing first boundary gradient detection on the M texture regions to determine the texture transition zone region between every two texture regions, wherein the texture transition zone region is a blurred boundary region caused by texture gradation; comparing and enhancing the first texture feature of each texture transition zone region with the second texture feature of the first adjacent texture region and the third texture feature of the second adjacent texture region, and fusing local texture detail features to generate a texture boundary enhancement feature map; performing hierarchical division of the texture boundary enhancement feature map according to the texture segmentation threshold to generate a texture layered feature map; and determining the texture layered feature map as the first feature domain.

[0011] By employing the aforementioned technical solution, multi-scale feature extraction is performed on the visible light deep feature map using multiple convolutional layers of different scales, resulting in a texture feature map that includes both global texture structure features and local texture detail features. Through texture gradient analysis and segmentation, the texture feature map can be divided into multiple texture regions, and boundary enhancement is applied to the texture transition zones. This multi-level texture feature processing method can comprehensively capture texture information in visible light images, providing rich texture detail references for infrared image enhancement.

[0012] Optionally, thermal intensity feature processing is performed on the infrared deep feature map to obtain a second feature domain. Specifically, this includes: using multiple convolutional layers with different expansion rates to perform multi-scale feature extraction on the infrared deep feature map to obtain a thermal radiation feature map, where the thermal radiation feature map includes global thermal radiation distribution features and local thermal radiation detail features; performing thermal intensity range analysis on the global thermal radiation distribution features to generate a thermal intensity statistical histogram; determining the thermal intensity segmentation threshold based on the peak distribution of the thermal intensity statistical histogram; dividing the thermal radiation feature map into N thermal intensity regions based on the thermal intensity segmentation threshold and the global thermal radiation distribution features, where N is a positive integer; and processing the N regions... A second boundary gradient detection is performed on the thermal intensity region to determine the thermal intensity transition zone region between every two thermal intensity regions in N thermal intensity regions, where the thermal intensity transition zone region is a blurred boundary region caused by heat diffusion; the first thermal radiation feature of each thermal intensity transition zone region is compared and reconstructed with the second thermal radiation feature of the first adjacent thermal intensity region and the third thermal radiation feature of the second adjacent thermal intensity region, and local thermal radiation detail features are fused to generate a thermal boundary enhancement feature map; the thermal boundary enhancement feature map is temperature-layered according to the thermal intensity segmentation threshold to generate a thermal intensity layered feature map; the thermal intensity layered feature map is determined as the second feature domain.

[0013] By employing the aforementioned technical solution, multi-scale feature extraction is performed on the infrared deep feature map using multiple convolutional layers with different dilation rates, resulting in a thermal radiation feature map that includes both global thermal radiation distribution features and local thermal radiation detail features. Through thermal intensity range analysis and segmentation, the thermal radiation feature map can be divided into multiple thermal intensity regions, and the thermal intensity transition zones are reconstructed through comparison. This multi-level thermal radiation feature processing method can accurately preserve the temperature distribution characteristics of infrared images and provide a reliable thermal radiation feature benchmark for feature mapping.

[0014] Optionally, the infrared image is mapped from the second feature domain to the intermediate feature domain using a target feature mapping network to obtain a first intermediate feature, and the first intermediate feature is mapped back to the first feature domain to obtain a first mapped image. Similarly, the visible light image is mapped from the first feature domain to the intermediate feature domain using the target feature mapping network to obtain a second intermediate feature, and the second intermediate feature is mapped back to the second feature domain to obtain a second mapped image. Specifically, this includes: constructing a first feature extraction encoder and a second feature extraction encoder using a multi-layer convolutional neural network, wherein the first feature extraction encoder and the second feature extraction encoder share the convolutional layer parameters of the multi-layer convolutional neural network; and constructing a first feature reconstruction decoder and a second feature reconstruction decoder using a multi-layer deconvolutional neural network, wherein the first feature reconstruction decoder and the second feature reconstruction decoder share the multi-layer deconvolutional neural network. The parameters of the deconvolution layer are determined; a first feature extraction encoder and a first feature reconstruction decoder are concatenated to construct a first mapping sub-network, wherein the first mapping sub-network is used to implement feature mapping from the second feature domain to the first feature domain; a second feature extraction encoder and a second feature reconstruction decoder are concatenated to construct a second mapping sub-network, wherein the second mapping sub-network is used to perform feature mapping from the first feature domain to the second feature domain; the infrared image is mapped from the second feature domain to the intermediate feature domain using the first mapping sub-network to obtain a first intermediate feature, and the first intermediate feature is mapped back to the first feature domain using the first mapping sub-network to obtain a first mapped image; the visible light image is mapped from the first feature domain to the intermediate feature domain using the second mapping sub-network to obtain a second intermediate feature, and the second intermediate feature is mapped back to the second feature domain using the second mapping sub-network to obtain a second mapped image.

[0015] By employing the above technical solution, a feature extraction encoder and a feature reconstruction decoder are constructed using a multi-layer convolutional neural network and a deconvolutional neural network with shared parameters, respectively. A first mapping sub-network and a second mapping sub-network are then constructed by concatenation. The parameter-sharing mechanism between the feature extraction encoder and the feature reconstruction decoder ensures the consistency of bidirectional feature mapping. This mapping structure design based on deep neural networks can establish nonlinear mapping relationships between different feature domains, enabling accurate conversion between infrared and visible light image features.

[0016] Optionally, determining a first feature difference between the first mapped image and the visible light image, and determining a second feature difference between the second mapped image and the infrared image, specifically includes: extracting a first shared feature vector corresponding to the first mapped image from an intermediate feature domain, and extracting a second shared feature vector corresponding to the visible light image from the intermediate feature domain; calculating the cosine similarity between the first shared feature vector and the second shared feature vector in the feature channel dimension to generate a visible light domain feature difference map; determining a first dynamic threshold based on the local variance distribution of the visible light domain feature difference map; and marking regions in the visible light domain feature difference map below the first dynamic threshold as first feature difference regions. The pseudo-infrared feature vector corresponding to the second mapped image is extracted from the second feature domain, and the real infrared feature vector corresponding to the infrared image is extracted from the second feature domain. The KL divergence of the temperature distribution between the pseudo-infrared feature vector and the real infrared feature vector is calculated to generate a thermal radiation fidelity difference map. A second dynamic threshold is determined based on the thermal intensity distribution of the thermal radiation fidelity difference map. Regions in the thermal radiation fidelity difference map that are higher than the second dynamic threshold are marked as second feature difference regions. Connectivity analysis is performed on the first feature difference region to determine the area distribution and local difference intensity of the first feature difference region. The first feature difference is determined based on the area distribution and local difference intensity. Thermal radiation intensity gradient analysis is performed on the second feature difference region to determine the gradient change rate and thermal intensity difference amplitude of the second feature difference region. The second feature difference is determined based on the gradient change rate and thermal intensity difference amplitude.

[0017] By employing the aforementioned technical solution, feature vectors are extracted from the intermediate and second feature domains respectively. Cosine similarity and KL divergence are then calculated to determine the feature difference regions. Connectivity analysis and thermal radiation intensity gradient analysis are performed on these feature difference regions to obtain accurate feature difference information. This multi-dimensional feature difference analysis method can comprehensively evaluate the quality of feature mapping and provide precise difference guidance for enhancement processing.

[0018] Optionally, the infrared image is enhanced based on the first feature difference and the second feature difference to obtain a target enhanced infrared image. Specifically, this includes: constructing a visible light detail compensation template based on the first feature difference, wherein the visible light detail compensation template includes edge contour enhancement information and texture structure enhancement information; constructing an infrared fidelity correction template based on the second feature difference, wherein the infrared fidelity correction template includes thermal radiation distribution preservation information and temperature gradient correction information; and performing a weighted fusion of the visible light detail compensation template and the infrared fidelity correction template to generate a comprehensive enhancement template, wherein the weighting coefficients of the weighted fusion are based on the first feature difference and the second feature difference. The relative intensity of the difference is determined; the infrared image is spatially divided into several image sub-blocks; the enhancement intensity value of the corresponding region of each image sub-block in the comprehensive enhancement template is determined, and the local enhancement parameters of each image sub-block are determined based on the enhancement intensity value; adaptive enhancement processing is performed on each image sub-block using the local enhancement parameters to obtain an enhanced image sub-block set; all enhanced image sub-blocks in the enhanced image sub-block set are seamlessly stitched together, and the stitching boundary is smoothed to obtain an enhanced stitched image; global tone mapping and dynamic range compression are performed on the enhanced stitched image to generate a target enhanced infrared image.

[0019] By employing the above technical solution, a visible light detail compensation template and an infrared fidelity correction template are constructed based on feature differences, and a comprehensive enhancement template is generated through weighted fusion. The infrared image is then spatially partitioned and adaptively enhanced based on the comprehensive enhancement template. This template-based local adaptive enhancement method can effectively fuse the detail information of the visible light image while maintaining the thermal radiation characteristics of the infrared image, thereby improving the visual quality of the infrared image.

[0020] In a second aspect, embodiments of this application provide an infrared image enhancement system, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the infrared image enhancement system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an infrared image enhancement system, cause the infrared image enhancement system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an infrared image enhancement system, cause the infrared image enhancement system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. The infrared image enhancement method based on deep learning convolutional neural networks provided in this application utilizes visible light images and infrared images to construct a feature transformation space, including a first feature domain, a second feature domain, and an intermediate feature domain, providing a basic framework for feature mapping of images of different modalities. Based on this, a target feature mapping network can achieve bidirectional feature mapping transformation between infrared and visible light images. By calculating the feature differences between the mapped image and the original image, a reliable evaluation basis can be provided for subsequent enhancement processing. This enhancement method based on dual-modal information and feature mapping can not only fully utilize the detailed information in the visible light image to compensate for the deficiencies of the infrared image, but also ensure the controllability of the enhancement process through the calculation of feature differences, thereby achieving high-quality enhancement of infrared images and effectively avoiding detail loss and over-enhancement in traditional single-modal enhancement methods.

[0025] 2. The infrared image enhancement method based on deep learning convolutional neural networks provided in this application utilizes a pre-defined feature extraction network to extract deep features from infrared and visible light images, followed by texture feature processing and thermal intensity feature processing, respectively. This yields a first feature domain containing visible light texture details and a second feature domain containing infrared thermal radiation distribution characteristics. Specifically, by performing feature channel concatenation and convolutional fusion on the infrared and visible light deep feature maps, cross-modal shared structural features can be extracted to obtain an intermediate feature domain containing the shared structural characteristics of both images. This domain construction method based on deep features effectively separates and preserves the feature attributes of images from different modalities, and simultaneously establishes the correlation between the two modalities through the intermediate feature domain, providing a more accurate feature representation for subsequent feature mapping.

[0026] 3. The infrared image enhancement method based on deep learning convolutional neural networks provided in this application extracts features at multiple scales using convolutional layers of different scales from the deep feature map of visible light, obtaining a texture feature map containing both global texture structure features and local texture detail features. Through texture gradient analysis and segmentation, the texture feature map can be divided into multiple texture regions, and boundary enhancement is performed on the texture transition zone regions. This multi-level texture feature processing method can comprehensively capture the texture information in visible light images, providing rich texture detail references for infrared image enhancement. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating an infrared image enhancement method based on a deep learning convolutional neural network in an embodiment of this application.

[0028] Figure 2 This is a schematic diagram of the physical device structure of an infrared image enhancement system in the embodiments of this application. Detailed Implementation

[0029] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to any or all possible combinations including one or more of the listed items.

[0030] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0031] This application provides an infrared image enhancement method based on deep learning convolutional neural networks, see reference. Figure 1 , Figure 1 This is a flowchart illustrating an infrared image enhancement method based on a deep learning convolutional neural network, as described in this application, which includes the following steps:

[0032] Step S101: After obtaining the infrared image and the visible light image of the target monitoring area, a feature transformation space is constructed. The infrared image and the visible light image are spatiotemporally aligned registered images of the target monitoring area. The feature transformation space includes a first feature domain for storing the visible light image, a second feature domain for storing the infrared image, and an intermediate feature domain for storing information shared by the first feature domain and the second feature domain.

[0033] Step S102: Using a target feature mapping network, the infrared image is mapped from the second feature domain to the intermediate feature domain to obtain a first intermediate feature, and the first intermediate feature is mapped to the first feature domain to obtain a first mapped image. Also, using a target feature mapping network, the visible light image is mapped from the first feature domain to the intermediate feature domain to obtain a second intermediate feature, and the second intermediate feature is mapped to the second feature domain to obtain a second mapped image.

[0034] Step S103: Determine the first feature difference between the first mapped image and the visible light image, and determine the second feature difference between the second mapped image and the infrared image;

[0035] Step S104: Enhance the infrared image based on the first feature difference and the second feature difference to obtain an enhanced infrared image of the target.

[0036] In the above embodiments, the target monitoring area refers to the specific scene range that needs to be image acquired and processed; the infrared image represents an image reflecting the thermal radiation characteristics of the target acquired by an infrared sensor; the visible light image refers to an image reflecting the visual characteristics of the target acquired by a visible light camera; the spatiotemporally aligned registered image represents an image pair that has undergone spatial and temporal synchronization processing; the feature transformation space represents a three-domain structure used to store image features of different modalities, including a first feature domain storing visible light image features, a second feature domain storing infrared image features, and an intermediate feature domain storing information shared by the two types of features; the first feature domain and the second feature domain respectively represent feature representation structures with specific attributes; the intermediate feature domain refers to a transitional feature space used to store information shared between different feature domains; feature mapping refers to the feature transformation relationship established between different feature domains; and feature difference represents the degree of distinction between different feature representations.

[0037] In the above embodiments, the problem of insufficient detail in infrared images during nighttime surveillance is addressed. First, image pairs are acquired using an infrared camera and a visible light camera installed in the surveillance area. The infrared camera acquires a 640×480 resolution thermal image, while the visible light camera acquires a 1920×1080 resolution color image. A SIFT-based registration algorithm is used to perform spatiotemporal alignment on the two images, downsampling the visible light image to the same resolution as the infrared image, resulting in a registered image pair. To achieve effective conversion of image features from different modalities, a three-domain feature transformation space is constructed. Specifically, a pre-trained VGG16 network is used to extract the conv4_3 layer features of the infrared and visible light images, respectively, resulting in 256-channel feature maps. The visible light feature map is subjected to 3×3 convolution and ReLU activation to obtain the first feature domain, and the infrared feature map is subjected to 3×3 convolution and ReLU activation to obtain the second feature domain. By concatenating the two feature maps along the channel dimension and then performing a 1×1 convolution, the common structural features of 128 channels are extracted as the intermediate feature domain.

[0038] In the above embodiment, a target feature mapping network with an encoder-decoder structure is used to achieve feature transformation. The encoder consists of four 3×3 convolutional layers, each followed by BatchNorm and ReLU; the decoder uses four 3×3 deconvolutional layers, also configured with BatchNorm and ReLU. Infrared features are mapped to an intermediate feature domain by the encoder to obtain the first intermediate feature, and then mapped back to the first feature domain by the decoder to obtain the first mapped image. Similarly, visible light features are mapped through the same network to obtain the second intermediate feature and the second mapped image. When calculating the feature difference between the mapped image and the original image, the features of the conv4_3 layer are extracted and the cosine similarity is calculated to obtain a difference map. Adaptive threshold segmentation is performed on the difference map to mark the regions with significant differences. Combining statistical features such as the area ratio of the difference regions and the average difference degree, the feature difference value is quantified. Finally, an adaptive enhancement strategy is designed based on the feature differences: enhancing details and contrast in regions with significant differences, and maintaining the original features in regions with smaller differences, thereby improving the quality of the infrared image.

[0039] Through the above steps, a feature transformation space is constructed using visible light and infrared images, including a first feature domain, a second feature domain, and an intermediate feature domain, providing a basic framework for feature mapping of images of different modalities. Based on this, the target feature mapping network can achieve bidirectional feature mapping transformation between infrared and visible light images. By calculating the feature differences between the mapped image and the original image, a reliable evaluation basis can be provided for subsequent enhancement processing. This enhancement method based on dual-modal information and feature mapping not only fully utilizes the detailed information in the visible light image to compensate for the deficiencies of the infrared image, but also ensures the controllability of the enhancement process through the calculation of feature differences, thereby achieving high-quality enhancement of infrared images and effectively avoiding detail loss and over-enhancement in traditional single-modal enhancement methods. This solves the technical problem of poor reliability in infrared image enhancement based on single-modal processing in related technologies, achieving the technical effect of improving the reliability of infrared image enhancement.

[0040] The entity performing the above steps may be a system with infrared image enhancement capabilities, such as an infrared image enhancement system, or a device with infrared image enhancement capabilities, or a controller or processor in the device or system, or a separate controller or processor, or other processing devices or processing units with similar processing functions, but is not limited to these.

[0041] In an optional embodiment, after acquiring infrared and visible light images of the target monitoring area, a feature transformation space is constructed, specifically including: inputting the infrared and visible light images into a preset feature extraction network to obtain infrared deep feature maps and visible light deep feature maps output by the preset feature extraction network; performing texture feature processing on the visible light deep feature map to obtain a first feature domain, wherein the first feature domain has visible light texture detail characteristics; performing thermal intensity feature processing on the infrared deep feature map to obtain a second feature domain, wherein the second feature domain has infrared thermal radiation distribution characteristics; performing feature channel splicing and convolution fusion processing on the infrared deep feature map and the visible light deep feature map to extract cross-modal common structural features to obtain an intermediate feature domain, wherein the intermediate feature domain includes the common structural characteristics of the infrared image and the visible light image.

[0042] In the above embodiments, the preset feature extraction network refers to a neural network model that can extract deep features of an image, obtained through large-scale data training; the infrared deep feature map refers to the high-dimensional feature representation of an infrared image extracted by the preset feature extraction network; the visible light deep feature map refers to the high-dimensional feature representation of a visible light image extracted by the preset feature extraction network; texture feature processing refers to feature extraction and preliminary enhancement operations for the texture structure of an image; thermal intensity feature processing refers to the extraction and analysis process of thermal radiation intensity features; feature channel stitching refers to combining the channel dimensions of different feature maps; convolution fusion processing refers to the mixing and integration of features through convolution operations; cross-modal shared structural features refer to structural features that coexist in images of different modalities.

[0043] In the above embodiments, to construct an effective feature transformation space, the following specific implementation scheme is adopted: First, a ResNet50 network pre-trained on ImageNet is selected as the feature extraction network, and the registered infrared image and visible light image with a size of 640×480 are input into the network respectively. Deep features are extracted from the conv4_x layer of the network to obtain 40×30×1024-dimensional feature maps, which are used as infrared deep feature maps and visible light deep feature maps respectively. The texture feature processing of the visible light deep feature map adopts a multi-scale convolutional structure, which includes three parallel branches: the first branch uses 1×1 convolution to extract point-level features, the second branch uses 3×3 convolution to extract local texture features, and the third branch uses 5×5 convolution to extract texture structure features over a larger range. The output feature maps of the three branches are concatenated by channel dimension, and then fused by 1×1 convolution to obtain a 512-channel feature map. The original features and the fused features are added through residual connections, and finally ReLU activation is applied to obtain a first feature domain with rich texture details.

[0044] In the above embodiments, the thermal intensity feature processing of the infrared deep feature map uses a multi-branch structure with dilated convolution: the first branch uses a standard 3×3 convolution, the second branch uses a 3×3 convolution with a dilation rate of 2, and the third branch uses a 3×3 convolution with a dilation rate of 4. This design can expand the receptive field without increasing the number of parameters, and better capture the distribution characteristics of thermal radiation. Similarly, the feature maps of the three branches are concatenated and fused with 1×1 convolution to obtain 512 channels. After adding residual connections, ReLU activation is performed to obtain a second feature domain that retains the thermal radiation distribution characteristics. To extract cross-modal shared structural features, the infrared deep feature map and the visible light deep feature map are concatenated in the channel dimension to obtain a 2048-channel feature map. Two layers of 3×3 convolution are used for feature fusion. The first layer outputs 512 channels, and the second layer outputs 256 channels. BatchNorm and ReLU are used after each layer. The final 256-channel feature map is the intermediate feature domain, which contains structural information shared by the two modal images, such as target contours and scene layout features. This feature domain construction method can effectively separate and retain the feature attributes of different modalities, and at the same time establish the relationship between the two modalities through the intermediate feature domain, providing a good foundation for subsequent feature mapping.

[0045] In an optional embodiment, texture feature processing is performed on the visible light deep feature map to obtain a first feature domain. Specifically, this includes: using multiple convolutional layers of different scales to extract multi-scale features from the visible light deep feature map to obtain a texture feature map, wherein the texture feature map includes global texture structure features and local texture detail features; performing texture gradient analysis on the global texture structure features to generate a texture gradient statistical histogram; determining a texture segmentation threshold based on the peak distribution of the texture gradient statistical histogram; and dividing the texture feature map into M texture regions based on the texture segmentation threshold and the global texture structure features, where M is a positive integer. The process involves: counting; performing first boundary gradient detection on M texture regions to determine the texture transition zone between every two texture regions, where the texture transition zone is a blurred boundary region caused by texture gradation; comparing and enhancing the first texture feature of each texture transition zone with the second texture feature of the first adjacent texture region and the third texture feature of the second adjacent texture region, and fusing local texture detail features to generate a texture boundary enhancement feature map; dividing the texture boundary enhancement feature map into layers according to a texture segmentation threshold to generate a texture layered feature map; and determining the texture layered feature map as the first feature domain.

[0046] In the above embodiments, multi-scale feature extraction refers to the process of extracting features at different spatial scales; texture feature map refers to a feature representation that includes image texture information; global texture structure feature represents the overall texture organization features of the image; local texture detail feature refers to the fine texture features of local areas of the image; texture gradient analysis represents the analysis of the degree of texture change; texture gradient statistical histogram refers to a statistical graph that characterizes the distribution of texture gradients; texture segmentation threshold refers to a threshold used to distinguish different texture regions; texture transition zone region refers to the transition region between different texture regions; boundary gradient detection represents the detection of changes in region boundaries; texture boundary enhancement feature map refers to a texture feature representation that has undergone boundary enhancement; texture hierarchical feature map represents a texture feature representation with a hierarchical structure.

[0047] In the above embodiments, to achieve effective texture feature processing of visible light images, the following specific implementation scheme is adopted: First, a multi-scale feature extraction network is constructed, containing three parallel convolutional branches, using 1×1, 3×3, and 5×5 convolutional kernels respectively, with a convolutional stride of 1, and output channels of 128, 256, and 128 respectively. The output feature maps of each branch are activated by BatchNorm and ReLU, and then concatenated along the channel dimension to obtain a 512-channel texture feature map, where the first 256 channels correspond to global texture structure features, and the last 256 channels correspond to local texture detail features. When performing gradient analysis on the global texture structure features, the Sobel gradients in the horizontal and vertical directions are calculated to obtain gradient magnitude maps and direction maps. A histogram of gradient magnitudes is statistically analyzed, using 32 uniformly distributed histogram intervals to obtain a texture gradient statistical histogram. The peak distribution of the histogram is analyzed using the Ostu's method (OTSU) algorithm, and three key thresholds are determined as texture segmentation thresholds: 0.15, 0.35, and 0.65. Based on these thresholds, the texture feature map is divided into four texture regions (M=4): weak texture region (gradient value < 0.15), medium texture region (0.15 ≤ gradient value < 0.35), strong texture region (0.35 ≤ gradient value < 0.65), and complex texture region (gradient value ≥ 0.65). A 3×3 Sobel operator is used to perform boundary gradient detection on each region, and regions with gradient magnitudes greater than 0.1 and less than 0.3 are marked as texture transition zones.

[0048] In the above embodiment, feature enhancement is performed on each transition zone region: First, the first texture feature (transition zone feature) of the region, as well as the second and third texture features of the two adjacent texture regions, are extracted. The mean and standard deviation of these three sets of features are calculated, and the mean of the transition zone feature is adjusted to a weighted average of the mean of the features of the adjacent regions, and the standard deviation is adjusted to the maximum of the standard deviations of the adjacent regions. Then, the enhanced transition zone feature is weighted and fused with the local texture detail features, with weights of 0.7 and 0.3 respectively, to obtain a texture boundary enhancement feature map. Finally, the enhanced feature map is divided into four levels according to the previously determined segmentation threshold to generate a texture layered feature map: the first layer is the basic texture layer (value range 0-0.15), the second layer is the detail texture layer (value range 0.15-0.35), the third layer is the main texture layer (value range 0.35-0.65), and the fourth layer is the fine texture layer (value range 0.65-1.0). This layered feature map is used as the first feature domain for subsequent feature mapping. Experiments show that this multi-level texture feature processing method can effectively preserve and enhance texture information in visible light images, providing a reliable texture reference for infrared image enhancement.

[0049] In an optional embodiment, thermal intensity feature processing is performed on the infrared deep feature map to obtain a second feature domain. Specifically, this includes: using multiple convolutional layers with different expansion rates to perform multi-scale feature extraction on the infrared deep feature map to obtain a thermal radiation feature map, wherein the thermal radiation feature map includes global thermal radiation distribution features and local thermal radiation detail features; performing thermal intensity range analysis on the global thermal radiation distribution features to generate a thermal intensity statistical histogram; determining a thermal intensity segmentation threshold based on the peak distribution of the thermal intensity statistical histogram; and dividing the thermal radiation feature map into N thermal intensity regions based on the thermal intensity segmentation threshold and the global thermal radiation distribution features, where N is a positive integer. A second boundary gradient detection is performed on N thermal intensity regions to determine the thermal intensity transition zone between every two thermal intensity regions, where the thermal intensity transition zone is a blurred boundary region caused by heat diffusion. The first thermal radiation feature of each thermal intensity transition zone is compared and reconstructed with the second thermal radiation feature of the first adjacent thermal intensity region and the third thermal radiation feature of the second adjacent thermal intensity region, and local thermal radiation detail features are fused to generate a thermal boundary enhancement feature map. The thermal boundary enhancement feature map is then temperature-layered according to the thermal intensity segmentation threshold to generate a thermal intensity layered feature map. The thermal intensity layered feature map is determined as the second feature domain.

[0050] In the above embodiments, the expansion rate represents the expansion coefficient of the convolution kernel; the thermal radiation feature map refers to the feature representation characterizing the thermal radiation distribution; the global thermal radiation distribution feature represents the distribution characteristics of the overall temperature field; the local thermal radiation detail feature refers to the thermal radiation detail information of a local region; the thermal intensity range analysis represents the statistical analysis of the temperature intensity range; the thermal intensity statistical histogram refers to the statistical graph characterizing the temperature distribution; the thermal intensity segmentation threshold represents the threshold used to distinguish different temperature regions; the thermal intensity transition zone region refers to the transition region between different temperature regions; heat diffusion refers to the phenomenon of heat propagation in space; the thermal boundary enhancement feature map represents the thermal feature representation after boundary enhancement; and the thermal intensity layering feature map refers to the feature representation with a temperature hierarchy structure.

[0051] In the above embodiments, to achieve thermal intensity feature processing of infrared images, the following specific implementation scheme is adopted: First, a multi-branch dilated convolutional network is constructed for multi-scale feature extraction, containing four parallel branches. Each branch uses a 3×3 convolutional kernel with dilation rates of 1, 2, 4, and 8, respectively, to ensure the capture of thermal radiation features under different receptive fields. The number of output channels for each branch are 128, 128, 128, and 128, respectively. After activation by BatchNorm and LeakyReLU (slope 0.2), the channels are concatenated to obtain a 512-channel thermal radiation feature map, where the first 256 channels represent global thermal radiation distribution features, and the last 256 channels represent local thermal radiation detail features. When analyzing the thermal intensity range of the global thermal radiation distribution features, the feature values ​​are first normalized to the [0,1] interval, and then the distribution histogram of thermal intensity values ​​is statistically analyzed. Statistical histograms are obtained by using 64 uniformly distributed histogram intervals. An adaptive threshold algorithm is applied to analyze the peak distribution characteristics of the histograms to identify the main temperature distribution intervals. Four key thermal intensity segmentation thresholds were determined using K-means clustering: 0.2, 0.4, 0.6, and 0.8. Based on these thresholds, the thermal radiation feature map was divided into five thermal intensity regions (N=5): background region (intensity value < 0.2), low-temperature region (0.2 ≤ intensity value < 0.4), medium-temperature region (0.4 ≤ intensity value < 0.6), high-temperature region (0.6 ≤ intensity value < 0.8), and extremely high-temperature region (intensity value ≥ 0.8). An improved Canny operator was used to detect boundary gradients in each thermal intensity region, with dual thresholds of 0.1 and 0.3. The width of the detected thermal intensity transition band was approximately 5% of the feature map size.

[0052] In the above embodiments, feature reconstruction is performed on each thermal intensity transition zone region: the first thermal radiation feature (transition zone feature) of the region and the second and third thermal radiation features of the two adjacent thermal intensity regions are extracted. A feature reconstruction method based on Gaussian mixture model is adopted to adjust the distribution parameters of the transition zone feature to a weighted combination of the distribution parameters of the features of adjacent regions. Specifically, the mean is a distance-weighted average, and the variance is the minimum value to ensure a smooth transition. The reconstructed transition zone feature is adaptively fused with the local thermal radiation detail features, and the fusion weight is dynamically adjusted according to the local temperature gradient to generate a thermal boundary enhancement feature map. Finally, the enhancement feature map is divided into five temperature layers according to the thermal intensity segmentation threshold to obtain a thermal intensity layered feature map: the first layer is the background temperature layer (0-0.2), the second layer is the low temperature feature layer (0.2-0.4), the third layer is the medium temperature feature layer (0.4-0.6), the fourth layer is the high temperature feature layer (0.6-0.8), and the fifth layer is the extremely high temperature feature layer (0.8-1.0). Using this layered feature map as the second feature domain, this multi-layered thermal intensity feature processing method can effectively maintain the temperature distribution characteristics of infrared images while enhancing the clarity of thermal target boundaries.

[0053] In an optional embodiment, the infrared image is mapped from a second feature domain to an intermediate feature domain using a target feature mapping network to obtain a first intermediate feature, and the first intermediate feature is mapped back to the first feature domain to obtain a first mapped image. Similarly, the visible light image is mapped from the first feature domain to an intermediate feature domain using the same network to obtain a second intermediate feature, and the second intermediate feature is mapped back to the second feature domain to obtain a second mapped image. Specifically, this includes: constructing a first feature extraction encoder and a second feature extraction encoder using a multi-layer convolutional neural network, wherein the first and second feature extraction encoders share the convolutional layer parameters of the multi-layer convolutional neural network; and constructing a first feature reconstruction decoder and a second feature reconstruction decoder using a multi-layer deconvolutional neural network, wherein the first and second feature reconstruction decoders share the parameters of the multi-layer deconvolutional neural network. The parameters of the deconvolution layer of the network are used to construct a first mapping sub-network by concatenating the first feature extraction encoder and the first feature reconstruction decoder, wherein the first mapping sub-network is used to realize the feature mapping from the second feature domain to the first feature domain; the second feature extraction encoder and the second feature reconstruction decoder are concatenated to construct a second mapping sub-network, wherein the second mapping sub-network is used to perform feature mapping from the first feature domain to the second feature domain; the infrared image is mapped from the second feature domain to the intermediate feature domain using the first mapping sub-network to obtain a first intermediate feature, and the first intermediate feature is mapped back to the first feature domain using the first mapping sub-network to obtain a first mapped image; the visible light image is mapped from the first feature domain to the intermediate feature domain using the second mapping sub-network to obtain a second intermediate feature, and the second intermediate feature is mapped back to the second feature domain using the second mapping sub-network to obtain a second mapped image.

[0054] In the above embodiments, a multi-layer convolutional neural network represents a deep learning model composed of multiple convolutional layers; a feature extraction encoder refers to a network module that converts input features into latent representations; a feature reconstruction decoder refers to a network module that reconstructs latent representations into target features; convolutional layer parameters refer to the weights and biases in a convolutional neural network; a deconvolutional neural network represents a network that reconstructs features through deconvolution operations; deconvolutional layer parameters refer to the weights and biases in a deconvolutional network; a mapping subnetwork represents a network structure that implements mapping between specific feature domains; parameter sharing refers to different network modules using the same network parameters.

[0055] In the above embodiments, to achieve bidirectional mapping and transformation between feature domains, an encoder-decoder network structure based on shared parameters is designed. Specifically, a feature extraction encoder is first constructed, consisting of 5 convolutional layers, each with a kernel size of 3×3 and a stride of 1. The number of output channels is 64, 128, 256, 512, and 256, respectively. Each layer is followed by BatchNorm and PReLU activation functions. To enhance feature extraction capabilities, residual connections are added between the third and fourth convolutional layers. The first and second feature extraction encoders fully share these convolutional layer parameters, ensuring consistency in feature extraction across different feature domains. The feature reconstruction decoder employs a symmetrical 5-layer deconvolutional structure with a kernel size of 3×3 and a stride of 1. The number of output channels is 512, 256, 128, 64, and the original feature channel number, respectively. Each deconvolutional layer is also followed by BatchNorm and PReLU activation, and skip connections are added between corresponding layers to preserve more feature details. The first and second feature reconstruction decoders share all deconvolution layer parameters, ensuring consistency in the feature reconstruction process.

[0056] In the above embodiments, a first feature extraction encoder and a first feature reconstruction decoder are concatenated to construct a first mapping sub-network for mapping from the infrared feature domain to the visible light feature domain. A 1×1 convolutional layer is added between the encoder and decoder for feature channel reorganization and alignment. Similarly, a second feature extraction encoder and a second feature reconstruction decoder are concatenated to construct a second mapping sub-network for mapping from the visible light feature domain to the infrared feature domain. The two mapping sub-networks adopt a symmetrical structural design to ensure the balance of bidirectional feature mapping. During feature mapping, the second feature domain features of the infrared image are first input into the encoder of the first mapping sub-network to obtain a 256-channel first intermediate feature, which contains the main structural information of the infrared image. Then, the first intermediate feature is input into the decoder, and a first mapped image with the same dimension as the first feature domain is reconstructed through a deconvolution operation. This image has the texture features of the visible light image. Similarly, the first feature domain features of the visible light image are input into the second mapping sub-network, and after passing through the encoder, a 256-channel second intermediate feature is obtained. Then, the second mapped image with infrared features is reconstructed through the decoder. To improve the stability of feature mapping, a combination of cycle consistency loss and adversarial loss is employed during network training. Cycle consistency ensures the invertibility of feature mapping, while adversarial loss promotes the generated features to conform to the distribution characteristics of the target domain. This bidirectional mapping network structure based on shared parameters can effectively achieve feature transformation between different feature domains, providing a reliable feature foundation for subsequent image enhancement.

[0057] In an optional embodiment, determining a first feature difference between the first mapped image and the visible light image, and determining a second feature difference between the second mapped image and the infrared image, specifically includes: extracting a first shared feature vector corresponding to the first mapped image from an intermediate feature domain, and extracting a second shared feature vector corresponding to the visible light image from the intermediate feature domain; calculating the cosine similarity between the first shared feature vector and the second shared feature vector in the feature channel dimension to generate a visible light domain feature difference map; determining a first dynamic threshold based on the local variance distribution of the visible light domain feature difference map; and marking regions in the visible light domain feature difference map that are below the first dynamic threshold as first feature difference regions. The pseudo-infrared feature vector corresponding to the second mapped image is extracted from the second feature domain, and the real infrared feature vector corresponding to the infrared image is extracted from the second feature domain. The KL divergence of the temperature distribution between the pseudo-infrared feature vector and the real infrared feature vector is calculated to generate a thermal radiation fidelity difference map. A second dynamic threshold is determined based on the thermal intensity distribution of the thermal radiation fidelity difference map. Regions in the thermal radiation fidelity difference map that are higher than the second dynamic threshold are marked as second feature difference regions. Connectivity analysis is performed on the first feature difference region to determine the area distribution and local difference intensity of the first feature difference region. The first feature difference is determined based on the area distribution and local difference intensity. Thermal radiation intensity gradient analysis is performed on the second feature difference region to determine the gradient change rate and thermal intensity difference amplitude of the second feature difference region. The second feature difference is determined based on the gradient change rate and thermal intensity difference amplitude.

[0058] In the above embodiments, the shared feature vector represents the feature representation extracted from the intermediate feature domain; the feature channel dimension refers to the channel direction of the feature representation; cosine similarity represents an index that measures the similarity of feature vectors; the visible light domain feature difference map refers to an image characterizing the feature differences in the visible light domain; the local variance distribution represents the degree of change in local regions of the image; the first dynamic threshold represents a threshold dynamically determined based on the local variance distribution of the visible light domain feature difference map for dividing the feature difference region; the second dynamic threshold represents a threshold dynamically determined based on the thermal intensity distribution of the thermal radiation fidelity difference map for dividing the feature difference region; the pseudo-infrared feature vector represents the infrared domain feature generated through mapping; the true infrared feature vector refers to the feature extracted from the original infrared image; the KL divergence represents an index that measures the difference in probability distribution; the thermal radiation fidelity difference map refers to an image characterizing the difference in thermal radiation features; connected component analysis represents the analysis of the connectivity of image regions; and the gradient change rate refers to the rate of change of the feature gradient.

[0059] In the above embodiments, to accurately evaluate the effect of feature mapping, an evaluation scheme based on multi-dimensional feature difference analysis is designed. First, feature vectors are extracted from the intermediate feature domain. For the first mapped image and the original visible light image, 256-dimensional first shared feature vectors and second shared feature vectors are extracted, respectively. The cosine similarity of these two vectors is calculated along the feature channel dimension, resulting in a visible light domain feature difference map with the same size as the original image. The local 9×9 window variance of this difference map is calculated, and an adaptive thresholding method is used to determine the first dynamic threshold, taking the lower quartile value of the local variance distribution, approximately 0.3. Regions in the difference map with similarity below this threshold are marked as first feature difference regions. Simultaneously, pseudo-infrared feature vectors of the second mapped image and true infrared feature vectors of the original infrared image are extracted from the second feature domain, each vector having a dimension of 512. The Kullback-Leibler divergence of the temperature distribution of these two feature vectors is calculated, generating a thermal radiation fidelity difference map. The thermal intensity histogram distribution of the difference map is analyzed, and the upper quartile value is selected as the second dynamic threshold, approximately 0.7. Regions in the difference map with scatter values ​​higher than the threshold are marked as second feature difference regions.

[0060] In the above embodiment, connected component analysis is performed on the first feature difference region: First, the difference region is binarized, and then an 8-connected region labeling algorithm is used to identify connected regions. The area percentage of each connected region is calculated, and the average difference intensity within the region is calculated. Regions with an area percentage greater than 5% and an average difference intensity greater than 0.5 are identified as the main difference regions. Based on the number of main difference regions, the total area percentage, and the average difference intensity, a weighted summation method is used to calculate the first feature difference value. Thermal radiation intensity gradient analysis is performed on the second feature difference region: The gradient map of the difference region is calculated using the Sober operator, and the spatial distribution of the gradient change rate is statistically analyzed. Simultaneously, the difference magnitude between the thermal intensity value within the difference region and the surrounding region is calculated. Specifically, the gradient change rate is taken as the local mean of the gradient magnitude, and the thermal intensity difference magnitude is taken as the mean of the intensity difference between the target region and the background region. The gradient change rate and the thermal intensity difference magnitude are combined through a nonlinear mapping function to obtain the second feature difference value. This multi-dimensional feature difference analysis method can comprehensively evaluate the quality of feature mapping, especially in terms of preserving texture details and thermal radiation characteristics. The first feature difference can reflect the accuracy of texture mapping, while the second feature difference can assess the degree of preservation of thermal radiation characteristics, providing a reliable reference for subsequent adaptive enhancement.

[0061] In an optional embodiment, the infrared image is enhanced based on a first feature difference and a second feature difference to obtain a target enhanced infrared image. Specifically, this includes: constructing a visible light detail compensation template based on the first feature difference, wherein the visible light detail compensation template includes edge contour enhancement information and texture structure enhancement information; constructing an infrared fidelity correction template based on the second feature difference, wherein the infrared fidelity correction template includes thermal radiation distribution preservation information and temperature gradient correction information; and performing weighted fusion of the visible light detail compensation template and the infrared fidelity correction template to generate a comprehensive enhancement template, wherein the weighting coefficients of the weighted fusion are based on the first feature difference and the second feature difference. The relative intensity of the two feature differences is determined; the infrared image is spatially segmented into several image sub-blocks; the enhancement intensity value of the corresponding region of each image sub-block in the comprehensive enhancement template is determined, and the local enhancement parameters of each image sub-block are determined based on the enhancement intensity value; adaptive enhancement processing is performed on each image sub-block using the local enhancement parameters to obtain an enhanced image sub-block set; all enhanced image sub-blocks in the enhanced image sub-block set are seamlessly stitched together, and the stitching boundary is smoothed to obtain an enhanced stitched image; global tone mapping and dynamic range compression are performed on the enhanced stitched image to generate a target enhanced infrared image.

[0062] In the above embodiments, the detail compensation template represents a reference template used to enhance image details; edge contour enhancement information refers to feature information used to enhance target edges; texture structure enhancement information represents feature information used to enhance texture details; fidelity correction template refers to a reference template used to maintain thermal radiation characteristics; thermal radiation distribution preservation information represents feature information used to maintain temperature distribution; temperature gradient correction information refers to feature information used to correct temperature transitions; weighted fusion represents a feature combination method based on weights; spatial domain segmentation refers to the process of dividing the image into sub-regions; local enhancement parameters represent parameters used for sub-region enhancement; seamless stitching refers to image fusion that eliminates stitching artifacts; tone mapping represents the method of adjusting image tone; and dynamic range compression refers to the process of adjusting the dynamic range of the image.

[0063] In the above embodiments, to achieve adaptive enhancement of infrared images, a multi-level enhancement scheme based on dual-template fusion is designed. First, a visible light detail compensation template is constructed based on the first feature difference. This template contains two parts of information: edge contour enhancement information is extracted by using the Laplacian operator to extract target edges, and the enhancement intensity is adaptively adjusted according to local contrast; texture structure enhancement information is extracted by using the Local Binary Pattern (LBP) operator to extract texture features, and the texture enhancement degree is adjusted by wavelet transform. An infrared fidelity correction template is constructed based on the second feature difference. This template also contains two types of information: thermal radiation distribution preservation information is obtained based on thermal intensity histogram equalization to ensure that the original temperature distribution characteristics are maintained after enhancement; temperature gradient correction information is calculated by an improved gradient operator to correct the transition characteristics of thermal target boundaries. Both templates are normalized to the [0,1] interval to ensure the consistency of the numerical range. The two templates are adaptively weighted and fused to generate a comprehensive enhancement template. The weighting coefficients are dynamically determined based on the relative intensity of the feature differences: when the first feature difference is dominant, the visible light detail compensation template has a larger weight, ranging from 0.6 to 0.8; when the second feature difference is significant, the infrared fidelity correction template has a larger weight, ranging from 0.5 to 0.7. The specific weights are obtained by mapping the feature difference ratio using the sigmoid function.

[0064] In the above embodiment, the original infrared image is spatially divided into blocks using overlapping sliding windows of 32×32 pixels, with an overlap rate of 25% between adjacent windows. For each image sub-block, local enhancement parameters are designed based on the average enhancement intensity value of its corresponding region in the comprehensive enhancement template. These enhancement parameters include a contrast gain coefficient (range 1.0-2.0), a detail enhancement coefficient (range 0.5-1.5), and a smoothing factor (range 0.1-0.3). Each image sub-block is adaptively enhanced using the determined local enhancement parameters: first, contrast adjustment is performed, then detail enhancement, and finally, smoothing is applied to suppress noise. The enhanced sub-blocks are seamlessly stitched together using an image fusion algorithm based on the Poisson equation, with a weighted averaging strategy used in overlapping areas to ensure smooth transitions. Guided filtering is used to smooth the stitching boundaries, eliminating possible stitching artifacts. Finally, the enhanced stitched image is globally optimized: adaptive gamma correction is used for tone mapping, with a gamma value range of 0.6-1.4; improved local histogram equalization is used to achieve dynamic range compression, ensuring detail visibility while avoiding over-enhancement. This enhancement scheme can effectively improve the visual quality of infrared images, while maintaining thermal radiation characteristics and enhancing image details, providing better image input for subsequent tasks such as target recognition.

[0065] This application's embodiments utilize visible light and infrared images to construct a feature transformation space, including a first feature domain, a second feature domain, and an intermediate feature domain, providing a basic framework for feature mapping of images from different modalities. Based on this, a target feature mapping network can achieve bidirectional feature mapping transformation between infrared and visible light images. By calculating the feature differences between the mapped image and the original image, a reliable evaluation basis can be provided for subsequent enhancement processing. This enhancement method based on dual-modal information and feature mapping not only fully utilizes the detailed information in the visible light image to compensate for the deficiencies of the infrared image, but also ensures the controllability of the enhancement process through the calculation of feature differences, thereby achieving high-quality enhancement of infrared images and effectively avoiding detail loss and over-enhancement in traditional single-modal enhancement methods.

[0066] The infrared image enhancement system in the embodiments of this invention is described below from the perspective of hardware processing. (See attached document.) Figure 2 , Figure 2 This is a schematic diagram of the physical device structure of an infrared image enhancement system in the embodiments of this application.

[0067] It should be noted that, Figure 2 The structure of the infrared image enhancement system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0068] like Figure 2 As shown, the infrared image enhancement system includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in Read-Only Memory (ROM) 202 or a program loaded from storage portion 208 into Random Access Memory (RAM) 203, such as performing the methods described in the above embodiments. The RAM 203 also stores...

[0069] It contains various programs and data required for system operation. CPU 201, ROM 202, and RAM 203 are interconnected via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0070] The following components are connected to I / O interface 205: input section 206 including audio input devices, push-button switches, etc.; output section 207 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 208 including a hard disk, etc.; and communication section 209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 209 performs communication processing via a network such as the Internet. Drive 210 is also connected to I / O interface 205 as needed. Removable media 211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 210 as needed so that computer programs read from them can be installed into storage section 208 as needed.

[0071] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 209, and / or installed from removable medium 211. When the computer program is executed by central processing unit (CPU) 201, it performs the various functions defined in the present invention.

[0072] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0073] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0074] Specifically, the infrared image enhancement system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the infrared image enhancement method based on deep learning convolutional neural networks provided in the above embodiment.

[0075] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the infrared image enhancement system described in the above embodiments; or it may exist independently and not assembled into the infrared image enhancement system. The storage medium carries one or more computer programs that, when executed by a processor of the infrared image enhancement system, cause the infrared image enhancement system to implement the infrared image enhancement method based on deep learning convolutional neural networks provided in the above embodiments.

[0076] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An infrared image enhancement method based on deep learning convolutional neural networks, characterized in that, include: Having acquired infrared and visible light images of the target monitoring area A feature transformation space is constructed, wherein the infrared image and the visible light image are spatiotemporally aligned registered images of the target monitoring area. The feature transformation space includes a first feature domain for storing the visible light image, a second feature domain for storing the infrared image, and an intermediate feature domain for storing information shared by the first feature domain and the second feature domain. The infrared image is mapped from the second feature domain to the intermediate feature domain using a target feature mapping network to obtain a first intermediate feature, and the first intermediate feature is mapped back to the first feature domain to obtain a first mapped image. The visible light image is also mapped from the first feature domain to the intermediate feature domain using the target feature mapping network to obtain a second intermediate feature, and the second intermediate feature is mapped back to the second feature domain to obtain a second mapped image. Determine a first feature difference between the first mapped image and the visible light image, and determine a second feature difference between the second mapped image and the infrared image; The infrared image is enhanced based on the first feature difference and the second feature difference to obtain an enhanced infrared image of the target.

2. The method according to claim 1, characterized in that, The construction of a feature transformation space after acquiring infrared and visible light images of the target monitoring area specifically includes: The infrared image and the visible light image are input into a preset feature extraction network to obtain the infrared deep feature map and the visible light deep feature map output by the preset feature extraction network; The visible light deep feature map is subjected to texture feature processing to obtain the first feature domain, wherein the first feature domain has visible light texture detail characteristics; The infrared deep feature map is subjected to thermal intensity feature processing to obtain the second feature domain, wherein the second feature domain has infrared thermal radiation distribution characteristics; The infrared deep feature map and the visible light deep feature map are spliced ​​and convolutionally fused using feature channels to extract cross-modal common structural features to obtain the intermediate feature domain, wherein the intermediate feature domain includes the common structural characteristics of the infrared image and the visible light image.

3. The method according to claim 2, characterized in that, The process of performing texture feature processing on the visible light deep feature map to obtain the first feature domain specifically includes: Multiple convolutional layers of different scales are used to perform multi-scale feature extraction on the visible light deep feature map to obtain a texture feature map, wherein the texture feature map includes global texture structure features and local texture detail features; Perform texture gradient analysis on the global texture structure features to generate a texture gradient statistical histogram; The texture segmentation threshold is determined based on the peak distribution of the texture gradient statistical histogram; The texture feature map is divided into M texture regions based on the texture segmentation threshold and the global texture structure features, where M is a positive integer; A first boundary gradient detection is performed on the M texture regions to determine the texture transition zone region between every two texture regions in the M texture regions, wherein the texture transition zone region is a blurred boundary region caused by texture gradation; The first texture feature of each texture transition zone region is compared and enhanced with the second texture feature of the first adjacent texture region and the third texture feature of the second adjacent texture region, and the local texture detail features are fused to generate a texture boundary enhancement feature map. The texture boundary enhancement feature map is hierarchically divided according to the texture segmentation threshold to generate a texture layered feature map; The texture layer feature map is determined as the first feature domain.

4. The method according to claim 2, characterized in that, The step of performing thermal intensity feature processing on the infrared deep feature map to obtain the second feature domain specifically includes: Multiple convolutional layers with different dilation rates are used to perform multi-scale feature extraction on the infrared deep feature map to obtain a thermal radiation feature map, wherein the thermal radiation feature map includes global thermal radiation distribution features and local thermal radiation detail features. A thermal intensity range analysis is performed on the global thermal radiation distribution characteristics to generate a thermal intensity statistical histogram; The heat intensity segmentation threshold is determined based on the peak distribution of the heat intensity statistical histogram; The thermal radiation feature map is divided into N thermal intensity regions based on the thermal intensity segmentation threshold and the global thermal radiation distribution characteristics, where N is a positive integer; A second boundary gradient detection is performed on the N thermal intensity regions to determine the thermal intensity transition zone region between every two thermal intensity regions in the N thermal intensity regions, wherein the thermal intensity transition zone region is a fuzzy boundary region caused by heat diffusion; The first thermal radiation feature of each thermal intensity transition zone region is compared and reconstructed with the second thermal radiation feature of the first adjacent thermal intensity region and the third thermal radiation feature of the second adjacent thermal intensity region, and the local thermal radiation detail features are fused to generate a thermal boundary enhancement feature map. The thermal boundary enhancement feature map is temperature-layered according to the thermal intensity segmentation threshold to generate a thermal intensity layered feature map; The thermal intensity layer feature map is determined as the second feature domain.

5. The method according to claim 1, characterized in that, The process of mapping the infrared image from the second feature domain to the intermediate feature domain using a target feature mapping network to obtain a first intermediate feature, mapping the first intermediate feature back to the first feature domain to obtain a first mapped image, and mapping the visible light image from the first feature domain to the intermediate feature domain using the target feature mapping network to obtain a second intermediate feature, and mapping the second intermediate feature back to the second feature domain to obtain a second mapped image, specifically includes: A first feature extraction encoder and a second feature extraction encoder are constructed using a multi-layer convolutional neural network, wherein the first feature extraction encoder and the second feature extraction encoder share the convolutional layer parameters of the multi-layer convolutional neural network; A first feature reconstruction decoder and a second feature reconstruction decoder are constructed using a multi-layer deconvolutional neural network, wherein the first feature reconstruction decoder and the second feature reconstruction decoder share the deconvolutional layer parameters of the multi-layer deconvolutional neural network; The first feature extraction encoder and the first feature reconstruction decoder are concatenated to construct a first mapping sub-network, wherein the first mapping sub-network is used to implement feature mapping from the second feature domain to the first feature domain; The second feature extraction encoder and the second feature reconstruction decoder are concatenated to construct a second mapping sub-network, wherein the second mapping sub-network is used for feature mapping from the first feature domain to the second feature domain; The infrared image is mapped from the second feature domain to the intermediate feature domain using the first feature extraction encoder in the first mapping sub-network to obtain the first intermediate feature, and the first feature reconstruction decoder in the first mapping sub-network is used to map the first intermediate feature from the intermediate feature domain to the first feature domain to obtain the first mapped image. The visible light image is mapped from the first feature domain to the intermediate feature domain using the second feature extraction encoder in the second mapping sub-network to obtain the second intermediate feature, and the second feature reconstruction decoder in the second mapping sub-network is used to map the second intermediate feature from the intermediate feature domain to the second feature domain to obtain the second mapped image.

6. The method according to claim 1, characterized in that, The determination of the first feature difference between the first mapped image and the visible light image, and the determination of the second feature difference between the second mapped image and the infrared image, specifically include: Extract a first shared feature vector corresponding to the first mapped image from the intermediate feature domain, and extract a second shared feature vector corresponding to the visible light image from the intermediate feature domain; Calculate the cosine similarity between the first shared feature vector and the second shared feature vector in the feature channel dimension to generate a visible light domain feature difference map; The first dynamic threshold is determined based on the local variance distribution of the visible light domain feature difference map; The regions in the visible light domain feature difference map that are below the first dynamic threshold are marked as the first feature difference regions; Extract the pseudo infrared feature vector corresponding to the second mapped image from the second feature domain, and extract the real infrared feature vector corresponding to the infrared image from the second feature domain; Calculate the KL divergence of the temperature distribution between the pseudo-infrared feature vector and the real infrared feature vector to generate a thermal radiation fidelity difference map. The second dynamic threshold is determined based on the thermal intensity distribution of the thermal radiation fidelity difference map. The regions in the thermal radiation fidelity difference map that are higher than the second dynamic threshold are marked as the second feature difference regions; Perform connected component analysis on the first feature difference region to determine the area distribution and local difference intensity of the first feature difference region; The first characteristic difference is determined based on the area distribution and the intensity of the local difference. Perform thermal radiation intensity gradient analysis on the second characteristic difference region to determine the gradient change rate and thermal intensity difference amplitude of the second characteristic difference region; The second characteristic difference is determined based on the gradient change rate and the magnitude of the thermal intensity difference.

7. The method according to claim 1, characterized in that, The step of enhancing the infrared image based on the first feature difference and the second feature difference to obtain an enhanced infrared image of the target specifically includes: A visible light detail compensation template is constructed based on the first feature difference, wherein the visible light detail compensation template includes edge contour enhancement information and texture structure enhancement information; An infrared fidelity correction template is constructed based on the second feature difference, wherein the infrared fidelity correction template includes thermal radiation distribution preservation information and temperature gradient correction information; The visible light detail compensation template and the infrared fidelity correction template are weighted and fused to generate a comprehensive enhancement template, wherein the weight coefficients of the weighted fusion are determined based on the relative intensity of the first feature difference and the second feature difference; The infrared image is spatially partitioned into several image sub-blocks. Determine the enhancement intensity value of the corresponding region of each image sub-block in the comprehensive enhancement template, and determine the local enhancement parameters of each image sub-block based on the enhancement intensity value; The local enhancement parameters are used to perform adaptive enhancement processing on each image sub-block to obtain an enhanced image sub-block set; All enhanced image sub-blocks in the enhanced image sub-block set are seamlessly stitched together, and the stitching boundaries are smoothed to obtain an enhanced stitched image; Global tone mapping and dynamic range compression are performed on the enhanced stitched image to generate the target enhanced infrared image.

8. An infrared image enhancement system, characterized in that, The infrared image enhancement system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the infrared image enhancement system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the infrared image enhancement system, the infrared image enhancement system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the infrared image enhancement system, it causes the infrared image enhancement system to perform the method as described in any one of claims 1-7.

Images