Infrared image enhancement method based on semantic segmentation

Abstract

The application provides an infrared image enhancement method and system based on semantic segmentation, which comprises the following processes: registering an infrared image and a visible light image; performing semantic segmentation on the visible light image to obtain a label image; obtaining the positions of category blocks according to different colors corresponding to different categories of the label image; since the infrared image and the visible light image have been registered, the label image corresponds to the infrared image, so the positions and pixel values of the category blocks are obtained; the histograms of each category block are counted and histogram equalization is performed on each category block; interpolation operation is performed on the category blocks to output an enhanced infrared image. To the best of our knowledge, the method of the application first introduces semantic segmentation to solve the image enhancement task. Compared with existing infrared image enhancement algorithms, the method solves the problem of insufficient local contrast and details of the infrared image, reduces the local noise of the image, and has better visual performance.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and infrared image processing technology, and more specifically to an infrared image enhancement method based on semantic segmentation. Background Technology

[0002] Currently, histogram equalization is mainly divided into global histogram equalization and local histogram equalization [4]. Global histogram equalization is simple and efficient, but it is easy to lose details. In order to solve the problem of global histogram equalization, local histogram equalization has been proposed. Local histogram equalization improves the enhancement effect by dividing the histogram into blocks or the entire input image into blocks, and then equalizing each region.

[0003] Local histogram equalization (HQE) is effective and easy to implement. However, because HQE divides the histogram or the input image into blocks, different grayscale transformations are applied to different pixels in the image, resulting in a deviation in the mapping function. This can cause objects in the output image to be overstretched or have increased noise, leading to an unnatural visual appearance.

[0004] In response, existing technologies attempt to improve the specific applications of image enhancement algorithms. For example, the paper with application number 202011323559.X proposes an adaptive contrast-limited threshold setting method (Cliplimit) that performs secondary cropping of the grayscale histogram. min =floor(theta*2^img_bit), Cliplimit min The minimum contrast limiting threshold is represented by the gray value at position theta*2^img_bit, which is used as the contrast limiting threshold Cliplimit for that sub-block. Here, 2^img_bit represents the dynamic range of the image, img_bit is the image bit depth, and theta is the truncation coefficient, ranging from [0,1]. In CN107784637A, a method for infrared image enhancement, Cliplimit = 0.1*(Kmax-Kmin), where Kmax and Kmin represent the maximum and minimum gray values ​​in each sub-block image. The maximum and minimum values ​​may be noise, which could cause the image Cliplimit to be higher than the normal value. Improper selection can lead to excessive local noise. Furthermore, these documents still use traditional block segmentation methods, only improving Cliplimit.

[0005] Existing technical documents:

[0006] Patent Document 1: 202011323559.X Infrared Image Denoising and Enhancement Method, Apparatus, System and Computer-Readable Storage Medium Based on Local Adaptive CLAHE

[0007] Patent Document 2: CN107784637A A Method for Infrared Image Enhancement Summary of the Invention

[0008] The purpose of this invention is to propose an infrared image enhancement method based on semantic segmentation. By segmenting the input image into blocks according to categories, the method solves the problem of insufficient local contrast and detail intensity in infrared images and reduces local noise in the image.

[0009] According to the present invention, an infrared image enhancement method based on semantic segmentation is proposed, comprising the following steps:

[0010] Step S1: Register the infrared image with the visible light image;

[0011] Step S2: Perform semantic segmentation on the visible light image to obtain a label map; obtain category blocks based on the different colors in the label map corresponding to different categories;

[0012] Step S3: Based on the registration of the infrared image and the visible light image, the infrared image corresponds to the label image, and the coordinate position of the type block in the infrared image and the pixel value of the coordinate position are obtained.

[0013] Step S4: Calculate the histograms of each type of block in the infrared image and perform histogram processing on each type of block;

[0014] Step S5: Perform infrared image type block interpolation calculation based on pixel position and grayscale mapping function of each type block;

[0015] Step S6: Output the enhanced infrared image.

[0016] Preferably, in step S2, the image is divided into blocks according to the label map of the semantic segmentation result to obtain the position coordinates of the category blocks.

[0017] Preferably, in step S3, obtaining the infrared image type block position and pixel value includes the following operations:

[0018] Step S31: Obtain the coordinates corresponding to different RGB values ​​of the label image. Different RGB values ​​have different colors and correspond to different types of regions. Obtain the position coordinates of different types of regions.

[0019] Step S32: Let X = {X(u, v)} represent a given input infrared image composed of L discrete gray levels, where X(u, v) represents the gray value corresponding to the spatial location (u, v). The input image is segmented into M category blocks through semantic segmentation, and the input image category is represented by Xi, where i = A, B, ..., M. X = X A ∪X B ∪...∪XM .

[0020] Preferably, in step S4, calculating the histograms of each type of block and performing histogram processing on each type of block includes the following operations:

[0021] Step S41: For a type block i, the histogram of its corresponding gray level l is defined as: hi(l)=nl l=0,1,...,L-1; where hi(l) is the total number of pixels with gray level l in type block i, nl is the total number of pixels with gray value l in the image, and L is the total number of gray levels in the original image.

[0022] Step S42: The probability distribution function for each gray level is: Pi(l) = hi(l) / Ni; where Pi(l) is the probability distribution function of gray level l of type block i;

[0023] Step S43: According to the principle of histogram equalization, the mapped gray level of gray level l for category block i is Hi:

[0024]

[0025] Where 2^img_bit represents the dynamic range of the image, and img_bit is the image bit depth;

[0026] Preferably, in step S5, the type block interpolation operation includes the following steps:

[0027] Step S51: Calculate the center coordinates of each type of block;

[0028] Step S52: For any pixel p in the image, calculate the distance di between pixel p and the center of other type blocks i;

[0029] Step S53: Calculate the weights for each type of block in p using the following formula:

[0030] D=∑di

[0031] wi=di / ∑(di / D)

[0032] Step S54: For any pixel p in the original image, its mapping value is the interpolation result of the grayscale mapping values ​​corresponding to other types of blocks. The formula for calculating the final mapping value of pixel p is:

[0033] T(X p )=∑w i H i (Xp);

[0034] Where X p Let T(X) be the gray value of pixel p in the original image. p H represents the final mapped value of pixel p.i (·) is the mapping function for type block i.

[0035] Compared with the prior art, the significant advantages of the technical solution of the present invention are as follows:

[0036] This invention proposes an infrared image enhancement method based on semantic segmentation. Compared to existing image enhancement algorithms, it provides a novel block-segmentation approach that divides the image into category blocks based on semantic segmentation, rather than the traditional spatial block division. This method can further improve image contrast, display more detail, and simultaneously address the problem of insufficient local contrast and detail intensity in existing infrared image enhancement methods, while reducing local noise in the image. It results in a superior visual effect.

[0037] This invention segments images using semantic segmentation, dividing them into different types of blocks. Histogram equalization is then applied to each type of block, followed by interpolation to output the final image. This differs from traditional image enhancement techniques, which spatially divide the image into equal-sized, non-overlapping rectangular blocks, neglecting overall global information. Furthermore, because each type of block independently calculates its mapping function, over-enhancement is easily observed. This method not only improves image contrast but also addresses the local noise amplification problem inherent in existing infrared image enhancement algorithms, resulting in superior visual performance.

[0038] In conjunction with the implementation of this invention, compared with the infrared image enhancement methods in prior art references 1 and 2, the method proposed in this invention is more effective in segmenting images into category blocks based on semantic segmentation, which can reduce image noise and enhance image contrast.

[0039] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail below may be considered part of the inventive subject matter of this disclosure, provided that such concepts do not contradict each other. Furthermore, all combinations of the claimed subject matter are considered part of the inventive subject matter of this disclosure.

[0040] The foregoing and other aspects, embodiments, and features of the teachings of the present invention will be more fully understood from the following description in conjunction with the accompanying drawings. Other additional aspects of the invention, such as features and / or beneficial effects of exemplary embodiments, will become apparent from the following description or may be learned through practice of specific embodiments according to the teachings of the present invention. Attached Figure Description

[0041] The accompanying drawings are not intended to be drawn to scale. In the drawings, every identical or nearly identical component shown in each figure may be indicated by the same reference numeral. For clarity, not every component is labeled in each figure.

[0042] Figure 1 This is a flowchart of an infrared image enhancement method based on semantic segmentation, an exemplary embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the infrared image obtained after registration in step S1.

[0044] Figure 3 This is a schematic diagram of the visible light image obtained after registration in step S1.

[0045] Figure 4 It is the label map obtained after semantic segmentation of the visible light image in step S2.

[0046] Figure 5 This is the flowchart for step S4, histogram processing.

[0047] Figure 6 This is a diagram of the interpolation process in step S5.

[0048] Figure 7 This is a schematic diagram of the final output image of an exemplary embodiment of the present invention, output in step S6.

[0049] Figure 8 This is a schematic diagram of an image processed using the traditional CLAHE technique based on existing technology.

[0050] Figure 9 This is a schematic diagram of an image processed using the method described in Patent 1 of the prior art. Detailed Implementation

[0051] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.

[0052] Various aspects of the invention are described in this disclosure with reference to the accompanying drawings, which illustrate numerous illustrative embodiments. The embodiments of this disclosure are not necessarily intended to encompass all aspects of the invention. It should be understood that the various concepts and embodiments described above, as well as those described in more detail below, can be implemented in any of many ways, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.

[0053] According to an exemplary embodiment of the present invention, an infrared image enhancement method based on semantic segmentation is proposed. The visible light image is divided into blocks by semantic segmentation, and the image is divided into blocks of various types. Then, histogram equalization is performed on each type block. The aim is to solve the problem of insufficient local contrast and detail intensity in infrared images and reduce local noise in the image.

[0054] Combination Figure 1 The illustrated flow of the semantic segmentation-based infrared image enhancement method of this invention includes the following steps:

[0055] Step S1: Register the infrared image with the visible light image;

[0056] Step S2: Perform semantic segmentation on the visible light image to obtain a label map; obtain category blocks based on the different colors in the label map corresponding to different categories;

[0057] Step S3: Based on the registration of the infrared image and the visible light image, the infrared image corresponds to the label image, and the coordinate position of the type block in the infrared image and the pixel value of the coordinate position are obtained.

[0058] Step S4: Calculate the histograms of each type of block in the infrared image and perform histogram processing on each type of block;

[0059] Step S5: Perform infrared image type block interpolation calculation based on pixel position and grayscale mapping function of each type block;

[0060] Step S6: Output the enhanced infrared image.

[0061] The exemplary implementation process of the above method will be described in more detail below with reference to the accompanying drawings.

[0062] In step S1, a suitable dataset is selected, and its visible light image and infrared image are registered.

[0063] In the exemplary embodiments of the present invention, the corresponding dataset is a Flir dataset, which contains unregistered visible light images and 8-bit infrared images. The registered visible light and infrared images are as follows: Figure 2 , Figure 3 As shown.

[0064] In step S2, the visible light image is semantically segmented to obtain a label map.

[0065] In an exemplary embodiment of the present invention, the semantic segmentation network CGnet is selected to perform semantic segmentation on visible light images. CGnet is trained using the visible light semantic segmentation street scene dataset Cityscapes, and then uses registered visible light images as input as the test dataset, outputting a semantic segmentation label map, such as... Figure 4 As shown.

[0066] In step S3, since the infrared image and the visible light image have been registered, the infrared image can correspond to the label image to obtain the category block position and pixel value.

[0067] As an example implementation method, combined with Figure 3 ,4 For example, if the RGB values ​​of the segmented sky region are [70, 130, 180], then the coordinates of all RGB values ​​in the range [70, 130, 180] can be iterated. Corresponding to the infrared image, the coordinate positions of each region and the pixel value at each coordinate position can be obtained.

[0068] In step S4, the histograms of each type of block are calculated, and histogram equalization is performed on each type of block. Each type of block undergoes its own histogram equalization process. In this embodiment, the 450×350 image is divided into 7 type regions, each of varying sizes. Histogram equalization is then performed on each type of block.

[0069] As an optional embodiment, the histogram equalization processing flow is as follows: Figure 5 As shown, the specific steps include:

[0070] Step S41: For a type block i, the histogram of its corresponding gray level l is defined as: hi(l)=nl l=0,1,...,L-1; where hi(l) is the total number of pixels with gray level l in type block i, nl is the total number of pixels with gray value l in the image, and L is the total number of gray levels in the original image.

[0071] Step S42: The probability distribution function for each gray level is: Pi(l) = hi(l) / Ni; where Pi(l) is the probability distribution function of gray level l of type block i;

[0072] Step S43: According to the principle of histogram equalization, the mapped gray level of gray level l for category block i is Hi:

[0073]

[0074] Where 2^img_bit represents the dynamic range of the image, and img_bit is the image bit depth.

[0075] In step S5, the interpolation process is as follows: Figure 6 As shown in the figure, different types of blocks are distinguished by different colors. Oi represents the center of different type i, and di is the distance between pixel p and the center of other type blocks. The type block interpolation operation includes the following steps:

[0076] Step S51: Calculate the center coordinates of each type of block;

[0077] Step S52: For any pixel p in the image, calculate the distance di between pixel p and the center of other type blocks i;

[0078] Step S53: Calculate the weights for each type of block in p using the following formula:

[0079] D = ∑di;

[0080] wi=di / ∑(di / D);

[0081] Step S54: For any pixel p in the original image, its mapping value is the interpolation result of the grayscale mapping values ​​corresponding to other types of blocks. The formula for calculating the final mapping value of pixel p is:

[0082] T(Xp)=∑w i H i (Xp)

[0083] Where Xp is the gray value of pixel p in the original image, T(Xp) is the final mapped value of pixel p, and Hi(·) is the mapping function of class block i.

[0084] Figure 7 It is the final output image of the exemplary embodiment of the present invention output in step S6, that is, the output infrared image after each pixel has been interpolated.

[0085] Combination Figure 7-9 The example shown, Figure 7 The image is the result of processing using the method of this invention. Figure 8 It is an image processed using traditional CLAHE. Figure 9 The image is processed based on the method described in Patent 1 of the prior art. The following table can be obtained by calculating the PSNR (Peak Signal-to-Noise Ratio) of the image:

[0086] PSNR (dB) 21.00 19.60 20.27

[0087] Based on the above comparative test results, one comparison is made with the traditional CLAHE enhancement algorithm; the other is a comparison with the infrared image denoising and enhancement method based on local adaptive CLAHE corresponding to Patent 1 in the prior art. As can be seen from the table, after processing by the method of this invention, the PSNR of the image is significantly improved, and the infrared image enhancement method of this invention significantly improves the background noise of the image.

[0088] Combined with the above appendix Figure 7-9 As illustrated in the embodiments, the semantic segmentation-based infrared image enhancement method proposed in this invention can not only enhance images and reduce noise, but also achieve better visual performance.

[0089] According to embodiments of the present invention, a computer system for semantic segmentation-based infrared image enhancement is also provided, comprising: one or more processors; and at least one memory. The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the processes of the semantic segmentation-based infrared image enhancement method of the above embodiments of the present invention.

[0090] According to embodiments of the present invention, a computer-readable medium for storing software including instructions executable by one or more computers, which, upon execution, cause the one or more computers to perform operations including the processes of the semantic segmentation-based infrared image enhancement method of the above embodiments of the present invention.

[0091] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.

Claims

1. An infrared image enhancement method based on semantic segmentation, characterized in that, Includes the following steps: Step S1: Register the infrared image with the visible light image; Step S2: Perform semantic segmentation on the visible light image to obtain a label map; obtain category blocks based on the different colors in the label map corresponding to different categories; Step S3: Based on the registration of the infrared image and the visible light image, the infrared image corresponds to the label image, and the coordinate position of the type block in the infrared image and the pixel value of the coordinate position are obtained. Step S4: Calculate the histograms of each type of block in the infrared image and perform histogram processing on each type of block; Step S5: Perform infrared image type block interpolation calculation based on pixel position and grayscale mapping function of each type block; Step S6: Output the enhanced infrared image; In step S3, obtaining the type block position and pixel value of the infrared image includes the following steps: Step S31: Obtain the coordinates corresponding to different RGB values ​​of the label image. Different RGB values ​​have different colors and correspond to different types of regions. Obtain the position coordinates of different types of regions. Step S32: Let X = {X(u, v)}, representing a given input infrared image composed of L discrete gray levels, where X(u, v) represents the gray value corresponding to the spatial location (u, v); the input infrared image is segmented into M category blocks through semantic segmentation, and the input image category is represented by Xi, where i = A, B, ..., M, X = X A ∪X B ∪...∪X M ; In step S4, the histograms of each type of block in the infrared image are statistically analyzed and histogram processing is performed on each type of block, including the following steps: Step S41: For a type block i, the histogram of its corresponding gray level l is defined as: hi(l) = nl, l = 0, 1, ..., L-1; where hi(l) is the total number of pixels with gray level l in type block i, nl is the total number of pixels with gray level l in the image, and L is the total number of gray levels in the original infrared image. Step S42: The probability distribution function of each gray level is: Pi(l) = hi(l) / ni, where Pi(l) is the probability distribution function of gray level l of type block i; Step S43: According to the principle of histogram equalization, the mapped gray level of gray level l for category block i is Hi: ； Where 2^img_bit represents the dynamic range of the infrared image, and img_bit is the image bit depth of the infrared image.

2. The infrared image enhancement method based on semantic segmentation according to claim 1, characterized in that, In step S2, the image is segmented into different category blocks based on the label map of the semantic segmentation result, and the position coordinates of the category blocks are obtained.

3. The infrared image enhancement method based on semantic segmentation according to claim 1, characterized in that, In step s2, the semantic segmentation uses the semantic segmentation network CGnet to perform semantic segmentation on the visible light image.

4. The infrared image enhancement method based on semantic segmentation according to claim 1, characterized in that, In step S4, the original input image with a size of 450×350 is divided into 7 categories of regions, each of which has a different size; then, histogram equalization is performed on each category of region.

5. A computer system for semantic segmentation-based infrared image enhancement, characterized in that, include: One or more processors; A memory that stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the process of the method of any one of claims 1-3.

6. A computer-readable medium for storing software, characterized in that, The software includes instructions executable by one or more computers, which, when executed, cause the one or more computers to perform operations including the process of the method described in any one of claims 1-3.

Images