Infrared image enhancement method based on scene segmentation

Abstract

The invention discloses an infrared image enhancement method based on scene segmentation. The method comprises the following steps: (1) inputting an original infrared image into an infrared image scene segmentation network based on edge feature guidance to obtain a scene soft segmentation result; (2) decomposing the infrared image into a background layer image and a detail layer image by adopting bilateral filtering; (3) performing adaptive enhancement on the detail layer image based on characteristics of different scenes; (4) enhancing the background layer image by adopting contrast limited adaptive histogram equalization; and (5) carrying out weighted fusion on the enhanced detail layer image and the background layer image. According to the infrared image enhancement method based on scene segmentation, the semantic segmentation algorithm is introduced, the infrared image is divided into different scene areas, and the adaptive infrared image detail enhancement parameters are set for the detail information and noise features of each scene area, so that the quality of the infrared image can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of infrared image enhancement technology, and more specifically, to an infrared image enhancement method based on scene segmentation. Background Art

[0002] With the development of infrared imaging technology in military, science and technology, medical and other fields, the quality of infrared images is crucial.

[0003] Most of the infrared images obtained by current infrared imaging devices have disadvantages such as low contrast and high noise, and they are in urgent need of image enhancement. Infrared image enhancement algorithms can be roughly divided into three main areas: spatial domain, frequency domain and convolutional neural network. However, most of the above infrared image enhancement algorithms adopt a unified processing strategy to enhance the entire image. This global method ignores the significant semantic differences and changes in detail richness between different scene regions. Specifically, the sky area in the infrared image often presents relatively flat and less detailed features, while the ground area contains rich target details and texture information. If a unified enhancement parameter is applied to the entire image, it is easy to cause the noise in low-detail areas such as the sky to be significantly amplified, while the enhancement effect of high-detail areas such as the ground is limited, thereby reducing the overall quality of the image. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides an infrared image enhancement method based on scene segmentation. By introducing a semantic segmentation algorithm, an edge feature-guided infrared image scene segmentation network is used to segment the infrared image into different scenes, and appropriate image enhancement parameters are selected based on the image features of different scenes to adaptively enhance the infrared image, reduce background noise, and improve the quality of the infrared image, so as to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above-mentioned object, the present invention provides the following technical solution: an infrared image enhancement method based on scene segmentation, comprising the following five steps: (1) inputting the original infrared image into an infrared image scene segmentation network guided by edge features to obtain a scene soft segmentation result; (2) using bilateral filtering to decompose the infrared image into a background layer image and a detail layer image; (3) adaptively enhancing the detail layer image based on the characteristics of different scenes; (4) enhancing the background layer image by using contrast-restricted adaptive histogram equalization; (5) weighted fusion of the enhanced detail layer image and the background layer image to obtain an infrared enhanced image:

[0006] Step 1: infrared image scene segmentation;

[0007] The present invention first performs scene segmentation on the infrared image, which mainly includes four steps:

[0008] (1) Preparation of infrared image dataset: Collect infrared image dataset and divide it into training set, validation set and test set, and use Labelme to draw labels of infrared images;

[0009] (2) Construction of infrared image scene segmentation network: Using the Fast-SCNN real-time semantic segmentation model as the core network framework, a infrared image scene segmentation network guided by edge features is constructed;

[0010] (3) Infrared image scene segmentation network training: The infrared image is input into the segmentation network for training, and the network parameters are optimized by constructing a supervised loss function to obtain a trained infrared image scene segmentation network;

[0011] (4) Infrared image scene segmentation network test: The infrared image test set is input into the trained infrared image scene segmentation model for segmentation to obtain the scene segmentation result of the test image;

[0012] Step 2: Use bilateral filtering to decompose the infrared image into background layer and detail layer;

[0013] The present invention adopts an unsharp mask image layering processing method, and uses bilateral filtering as a low-pass filter to divide the infrared image into a background layer image containing low-frequency information and a detail layer image containing high-frequency information;

[0014] Step 3: Adaptively enhance the detail layer image based on the characteristics of different scenes;

[0015] Considering that the sky region of the detail layer image has fewer detail features and more noise information, while the ground region has rich target features, the present invention uses the scene soft segmentation result to map the detail layer image into the detail layer sky region and the detail layer ground region, and sets appropriate enhancement parameters according to the scene features and detail information of different regions, thereby adaptively enhancing the sky region and the detail region, and finally merging the two to obtain the enhanced detail layer image;

[0016] Step 4: Use contrast-limited adaptive histogram equalization to enhance the background layer image;

[0017] The present invention adopts the limited contrast adaptive histogram equalization method to enhance the background layer image, which mainly includes four steps:

[0018] (1) Image block division and sub-block histogram statistics: The infrared image is divided into sub-blocks of equal size, and the histogram of each sub-block is calculated;

[0019] (2) Sub-block histogram clipping and redistribution: Setting the clipping threshold and clipping and redistributing the sub-block histogram to avoid excessive local contrast enhancement;

[0020] (3) Histogram equalization: Calculate the mapping function of each sub-block based on the reallocated histogram;

[0021] (4) Sub-block pixel interpolation reconstruction: In order to achieve smooth transition at the sub-block boundary, the sub-block pixels located at the four corners of the image are directly mapped using a mapping function, the other sub-block pixels located at the four edges of the image are mapped using linear interpolation, and the sub-block pixels located in the middle of the image are mapped using bilinear interpolation;

[0022] Step 5: Perform weighted fusion on the enhanced detail layer and background layer to obtain an infrared enhanced image;

[0023] The enhanced detail layer image and the enhanced background layer image are weighted fused to obtain the final infrared image enhancement result.

[0024] In a preferred embodiment, in step S1, the infrared image is annotated into two scenes, namely, a flat area with fewer details, namely, a sky area, and an area with rich details, namely, a ground area.

[0025] Technical effects and advantages of the present invention:

[0026] The infrared image enhancement method based on scene segmentation proposed in the present invention divides the infrared image into different scene areas by introducing a semantic segmentation algorithm, and sets adaptive infrared image detail enhancement parameters according to the detail information and noise characteristics of each scene area. The method not only realizes the adaptive enhancement of infrared images, but also solves the problem that noise is easily amplified during the detail enhancement process. It can effectively improve the quality of infrared images and the practical effect of current infrared imaging devices, and promote the development of infrared imaging technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the infrared image enhancement algorithm based on scene segmentation of the present invention;

[0028] Figure 2 This is a scene segmentation network framework diagram of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] As attached Figure 1 To Attachment Figure 2 The infrared image enhancement method based on scene segmentation shown in the figure comprises the following steps:

[0031] S1. Infrared image scene segmentation:

[0032] First, the infrared image scene segmentation is performed, which mainly includes four steps: infrared image dataset preparation, infrared image scene segmentation network construction, infrared image scene segmentation network training and infrared image scene segmentation network testing;

[0033] (1) Preparation of infrared image dataset:

[0034] A total of 150 original infrared data sets were collected, with an image size of 1280×1024. The original infrared data sets were divided into 120 training sets, 15 validation sets, and 15 test sets. Labelme was used to annotate infrared image scenes, and the infrared images were annotated into two scenes, namely, a flat area with fewer details (i.e., the sky area) and an area with rich details (i.e., the ground area);

[0035] (2) Construction of infrared image scene segmentation network:

[0036] Construct an infrared image scene segmentation network based on edge feature guidance. The network framework is as follows: Figure 2 As shown in the figure, the network uses the Fast-SCNN real-time semantic segmentation model as the core network framework, which mainly includes three modules: edge feature guidance module, dual-channel feature extraction module and feature classification module;

[0037] Since the original infrared image has a low resolution, a weak contrast, and a blurred boundary due to the influence of temperature, the present invention uses an edge detection network PiDiNet to obtain the edge feature image of the original infrared image, and then uses the edge feature map as the prior knowledge of the infrared image, and inputs it into the scene segmentation network after dual-channel splicing with the original image, and the auxiliary network extracts more edge feature information, thereby guiding the network to better perform scene segmentation on the infrared image and improve the scene segmentation accuracy of the infrared image;

[0038] The dual-channel feature extraction module is mainly used to extract and fuse low-level features and global features of the image, so that the network can obtain more levels of feature information. The low-level features are only obtained through the convolution layer (Conv2D) and the depth-separable convolution layer (DSConv), while the global feature extraction part introduces the Bottleneck residual module in MobileNet-V2 to capture image feature information, and uses the pyramid module (PyramidPooling) to aggregate global context features;

[0039] The feature classification module is used for image classification, which divides the original infrared image into two scenes: the sky area and the ground area;

[0040] (3) Infrared image scene segmentation network training:

[0041] The infrared image training set and the corresponding label image are input into the infrared image scene segmentation network for training. The supervised loss function is constructed according to the segmentation result predicted by the infrared image scene segmentation network and the label image corresponding to the original image. The network parameters are optimized through the supervised loss function. The supervised loss function mainly adopts the cross entropy loss function, and the formula is as follows:

[0042]

[0043] Among them, p i represents the i-th element predicted by the segmentation network, g i Represents the i-th element of the label image;

[0044] (4) Infrared image scene segmentation network test:

[0045] The infrared image test set is input into the trained infrared image scene segmentation model for prediction to obtain the scene soft segmentation result of the infrared image test set;

[0046] S2. Use bilateral filtering to decompose the infrared image into background layer and detail layer:

[0047] The unsharp mask image layering processing method is adopted, and the bilateral filter is used as a low-pass filter to divide the infrared image into a background layer image containing low-frequency information and a detail layer image containing high-frequency information. Specifically, the original infrared image I IN The background layer image I of the infrared image is obtained by bilateral filtering B , using the original image I IN Subtract background layer image I B Get detail layer image I D ,Right now:

[0048] I D =I IN -I B

[0049] Among them, bilateral filtering is a nonlinear image processing technology that realizes edge-preserving denoising of images by simultaneously considering the spatial proximity and grayscale similarity of images. The calculation formula is as follows:

[0050]

[0051] Where I is the pixel value of the input image, G is the pixel value of the image after bilateral filtering, σ s is the standard deviation of the spatial distance, σ r is the standard deviation of pixel intensity;

[0052] S3, Adaptively enhance the detail layer image based on the characteristics of different scenes:

[0053] Considering that the sky area of ​​the detail layer image has fewer detail features and more noise information, while the ground area has rich target features, we first use the scene soft segmentation result I seg The detail layer image I D Mapping to detail layer sky area I D_sky and detail layer ground area I D_ground , the formula is as follows:

[0054] I D_ground =I D ×I seg

[0055] I D_sky =I D -I D_ground

[0056] Among them, the scene soft segmentation result I seg The pixel value range is 0 to 1. The part close to 0 pixel value is the sky area, and the part close to 1 pixel value is the ground area. The soft segmentation result is used for mapping to make the enhanced image transition naturally between the sky area and the ground area.

[0057] Then, a smaller enhancement parameter λ is set for the detail layer sky area with larger noise and fewer details. 1 , set a larger enhancement parameter λ for the detail layer ground area with less noise and more details 2 , adaptively enhance the sky area and detail area respectively, and finally fuse the two to obtain the enhanced detail layer image I D_enh , the formula is as follows:

[0058] I D_enh =λ 1 ×I D_sky +λ 2 ×I D_ground

[0059] Among them, λ 1 Set to 0.5, λ 2 Set to 2;

[0060] S4, using contrast-limited adaptive histogram equalization to enhance the background layer image:

[0061] The background layer image is enhanced by using the contrast-constrained adaptive histogram equalization method, which mainly includes four steps: image block division and sub-block histogram statistics, sub-block histogram cropping and redistribution, histogram equalization and sub-block pixel interpolation reconstruction. The specific steps are as follows:

[0062] (1) Image block division and sub-block histogram statistics:

[0063] The infrared image is divided into sub-blocks of equal size, the size of each sub-block is set to w×h, and the histogram of each sub-block is calculated, that is:

[0064]

[0065] Where w and h are set to 64, k represents the gray level, L represents the total number of gray levels in the image, n(k,i,j) is used to determine whether the current pixel value I(i,j) is equal to the gray level k, and h k Represents the number of pixels with the current gray level k;

[0066] (2) Sub-block histogram cropping and redistribution:

[0067] Set the clipping threshold and clip and redistribute the sub-block histogram to avoid excessive local contrast enhancement. The clipping threshold formula is as follows:

[0068]

[0069] Where M represents the total number of pixels in the sub-block, L represents the total number of gray levels in the sub-block, ε represents the cropping factor, and U and Q represent the gray level mean and gray level variance of the sub-block;

[0070] Then the total cropped pixels N are calculated according to the cropping threshold and evenly distributed to each gray level, that is, the reallocated histogram is:

[0071]

[0072] (3) Histogram equalization:

[0073] According to the reallocated histogram h ’ k The mapping function of each sub-block can be calculated as:

[0074]

[0075] (4) Sub-block pixel interpolation reconstruction:

[0076] Finally, in order to achieve smooth transition of sub-blocks at the boundaries, the sub-block pixels located at the four corners of the image are directly mapped using the mapping function, the other sub-block pixels located at the four edges of the image are mapped by linear interpolation, and the sub-block pixels located in the middle of the image are mapped by bilinear interpolation;

[0077] S5. Perform weighted fusion on the enhanced detail layer and background layer to obtain an infrared enhanced image:

[0078] The enhanced detail layer image ID_enh and the enhanced background layer image I B_enh Perform weighted fusion to obtain the final infrared image enhancement result I E :

[0079] I E =ω d I D_enh +ω b I B_enh

[0080] Among them, ω d is the detail layer image fusion coefficient, ω b is the background layer image fusion coefficient.

[0081] Finally, a few points should be explained: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, which may refer to mechanical connection or electrical connection, or internal communication between two components, or direct connection. "upper", "lower", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may change;

[0082] Secondly: In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other;

[0083] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for infrared image enhancement based on scene segmentation, characterized in that: The following steps are included: S1. Infrared image scene segmentation: First, the infrared image scene segmentation is performed, which mainly includes four steps: infrared image dataset preparation, infrared image scene segmentation network construction, infrared image scene segmentation network training and infrared image scene segmentation network testing; (1) Preparation of infrared image dataset: A total of 150 original infrared data sets were collected, with an image size of 1280×1024. The original infrared data sets were divided into 120 training sets, 15 validation sets, and 15 test sets. Labelme was used to annotate infrared image scenes, and the infrared images were annotated into two scenes, namely, a flat area with fewer details and an area with rich details. (2) Construction of infrared image scene segmentation network: Construct an infrared image scene segmentation network based on edge feature guidance. The network uses the Fast-SCNN real-time semantic segmentation model as the core network framework and mainly includes three modules: edge feature guidance module, dual-channel feature extraction module and feature classification module. The edge detection network PiDiNet is used to obtain the edge feature image of the original infrared image. The edge feature map is then used as the prior knowledge of the infrared image and is input into the scene segmentation network after being spliced ​​with the original image in two channels. The auxiliary network extracts more edge feature information, thereby guiding the network to better segment the infrared image and improve the scene segmentation accuracy of the infrared image. The dual-channel feature extraction module is mainly used to extract and fuse the low-level features and global features of the image, so that the network can obtain more levels of feature information. The low-level features are only obtained through the convolution layer Conv2D and the depth-separable convolution layer DSConv, while the global feature extraction part introduces the Bottleneck residual module in MobileNet-V2 to capture image feature information, and uses the pyramid module PyramidPooling to aggregate global context features. The feature classification module is used for image classification, which divides the original infrared image into two scenes: the sky area and the ground area; (3) Infrared image scene segmentation network training: The infrared image training set and the corresponding label image are input into the infrared image scene segmentation network for training. The supervised loss function is constructed according to the segmentation result predicted by the infrared image scene segmentation network and the label image corresponding to the original image. The network parameters are optimized through the supervised loss function. The supervised loss function mainly adopts the cross entropy loss function, and the formula is as follows: Among them, p i represents the i-th element predicted by the segmentation network, g i Represents the i-th element of the label image; (4) Infrared image scene segmentation network test: The infrared image test set is input into the trained infrared image scene segmentation model for prediction to obtain the scene soft segmentation result of the infrared image test set; S2, use bilateral filtering to decompose the infrared image into background layer and detail layer: The unsharp mask image layering processing method is adopted, and the bilateral filter is used as a low-pass filter to divide the infrared image into a background layer image containing low-frequency information and a detail layer image containing high-frequency information. Specifically, the original infrared image I IN The background layer image I of the infrared image is obtained by bilateral filtering B , using the original image I IN Subtract background layer image I B Get detail layer image I D ,Right now: I D =I IN -I B Among them, bilateral filtering is a nonlinear image processing technology that realizes edge-preserving denoising of images by simultaneously considering the spatial proximity and grayscale similarity of images. The calculation formula is as follows: Where I is the pixel value of the input image, G is the pixel value of the image after bilateral filtering, σ s is the standard deviation of the spatial distance, σ r is the standard deviation of pixel intensity; S3, Adaptively enhance the detail layer image based on the characteristics of different scenes: Considering that the sky area of ​​the detail layer image has fewer detail features and more noise information, while the ground area has rich target features, we first use the scene soft segmentation result I seg The detail layer image I D Mapping to detail layer sky area I D_sky and detail layer ground area I D_ground , the formula is as follows: I D_ground =I D ×I seg I D_sky =I D -I D_ground Among them, the scene soft segmentation result I seg The pixel value range is 0 to 1. The part close to 0 pixel value is the sky area, and the part close to 1 pixel value is the ground area. The soft segmentation result is used for mapping to make the enhanced image transition naturally between the sky area and the ground area. Then, a smaller enhancement parameter λ1 is set for the detail layer sky area with larger noise and fewer details, and a larger enhancement parameter λ2 is set for the detail layer ground area with smaller noise and more details. The sky area and the detail area are adaptively enhanced respectively, and finally the two are fused to obtain the enhanced detail layer image I D_enh , the formula is as follows: I D_enh =λ1×I D_sky +λ2×I D_ground Among them, λ1 is set to 0.5 and λ2 is set to 2; S4, using contrast-limited adaptive histogram equalization to enhance the background layer image: The background layer image is enhanced by using the contrast-constrained adaptive histogram equalization method, which mainly includes four steps: image block division and sub-block histogram statistics, sub-block histogram cropping and redistribution, histogram equalization and sub-block pixel interpolation reconstruction. The specific steps are as follows: (1) Image block division and sub-block histogram statistics: The infrared image is divided into sub-blocks of equal size, the size of each sub-block is set to w×h, and the histogram of each sub-block is calculated, that is: Where w and h are set to 64, k represents the gray level, L represents the total number of gray levels in the image, n(k,i,j) is used to determine whether the current pixel value I(i,j) is equal to the gray level k, and h k Represents the number of pixels with the current gray level k; (2) Sub-block histogram cropping and redistribution: Set the clipping threshold and clip and redistribute the sub-block histogram to avoid excessive local contrast enhancement. The clipping threshold formula is as follows: Where M represents the total number of pixels in the sub-block, L represents the total number of gray levels in the sub-block, ε represents the cropping factor, and U and Q represent the gray level mean and gray level variance of the sub-block; Then the total cropped pixels N are calculated according to the cropping threshold and evenly distributed to each gray level, that is, the reallocated histogram is: (3) Histogram equalization: According to the reallocated histogram The mapping function of each sub-block can be calculated as: (4) Sub-block pixel interpolation reconstruction: Finally, in order to achieve smooth transition of sub-blocks at the boundaries, the sub-block pixels located at the four corners of the image are directly mapped using the mapping function, the other sub-block pixels located at the four edges of the image are mapped by linear interpolation, and the sub-block pixels located in the middle of the image are mapped by bilinear interpolation; S5. Perform weighted fusion on the enhanced detail layer and background layer to obtain an infrared enhanced image: The enhanced detail layer image I D_enh and the enhanced background layer image I B_enh Perform weighted fusion to obtain the final infrared image enhancement result I E : I E =ω d I D_enh +ω b I B_enh Among them, ω d is the detail layer image fusion coefficient, ω b is the background layer image fusion coefficient.

2. The infrared image enhancement method based on scene segmentation according to claim 1, characterized in that: In step S1, the infrared image is annotated into two scenes, namely, a flat area with fewer details, namely, the sky area, and an area with rich details, namely, the ground area.

Images