An Adaptive Infrared Image Colorization Method and System Based on Human Visual Characteristics

Abstract

The present invention discloses a method and system for adaptive infrared image colorization based on human visual features, relating to the field of image processing technology. The method comprises the following steps: preparing a data set and adapting it; constructing a network model; training the network model; constructing a composite loss function to minimize the loss; fine-tuning the model; saving the model; and solidifying and outputting the finalized model parameters as a model file. This model can then be directly called upon for colorization output during future testing and model hardware deployment to obtain a colored image. By deeply mining the human visual features, applying the principle of overlap in these features, introducing a generation strategy, and designing a generator, the present invention makes the output infrared image colorization more realistic and consistent with the local and overall perception of human visual observation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method and system for adaptive infrared image colorization based on human visual characteristics. Background Art

[0002] Image colorization technology is an image processing technology that converts collected images into more visually universal images. In the field of image colorization, infrared image colorization, as an emerging technology, is being widely used in many industries such as military, medicine, security, and industry, and has important research value. However, the conversion of single-channel infrared images into three-channel RGB images still faces many challenges, including poor image quality, severe loss of detail information, and difficulty in collecting paired data sets. With the continuous deepening of research on human visual characteristics, scholars in various fields have been inspired to propose cutting-edge technologies such as convolutional layers, pooling layers, normalization layers, and attention mechanisms that are widely used in the field of deep learning. Continuously exploring the characteristics of human body is still of great significance in today's era.

[0003] The Chinese patent publication number is "CN114067018B," and its title is "A Method and System for Colorizing Infrared Images Based on Dilated Residual Generative Adversarial Networks." The method first constructs four modules: a shallow feature extraction module, a deep feature extraction module, an information fusion module, and an upsampling module. Next, the dataset is simulated downgraded, and the resulting high- and low-resolution image pairs are used to train the entire convolutional neural network. The network model is then trained and fine-tuned by minimizing the loss value. Finally, the resulting model parameters are solidified, allowing for direct use of the parameters for image colorization next time. The colorized image obtained by this method suffers from severe loss of detail, significantly different from the visual effect of the human eye, and has high requirements for the format and quality of the dataset, resulting in poor robustness.

[0004] In summary, how to propose an adaptive infrared image colorization method and system based on human visual characteristics to solve the above problems is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0005] The present invention solves the above-mentioned technical problem by providing a method for adaptive infrared image colorization based on human visual characteristics, comprising:

[0006] Step 1: Prepare the dataset and adapt: Obtain the infrared image dataset, preprocess it, and then adapt it to output fixed-size training set images. The training set is divided into the KAIST dataset and the FLIR dataset.

[0007] Step 2: Build a network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator. The first adversarial network is composed of the adversarial generative network composed of generator 1, generator 2 and the discriminator. By completing adversarial learning between the generated colorized image and the true value image, it continuously obtains more realistic colorization results. The second adversarial network is composed of the optimal sliding strategy, adversarial strategy and fusion module in the generative network. Through continuous adversarial learning between image 1 and image 2, it selects the optimal generator and completes effective dynamic segmentation and image stitching and fusion operations.

[0008] The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is as follows:

[0009]

[0010] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step size for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. n Represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation,

[0011] Follow K n and The quantitative relationship between them leads to K n The value of is:

[0012]

[0013] Among them, the value of k is a positive integer, e n Replace the expression with:

[0014]

[0015] The adversarial strategy consists of a judgment statement, which is used to complete adversarial learning and selection between image 1 and image 2. The segmentation loss is calculated for image region II of input image 1 and image region II of image 2, respectively, to obtain segmentation loss G1(II) and segmentation loss G2(II), and the sizes of G1(II) and G2(II) are compared. The following five results are obtained:

[0016]

[0017] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0018] The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy:

[0019]

[0020] in, represents the output of the fusion module, Indicates concatenating and resizing the three inputs;

[0021] The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x-direction and y-direction of the colorized image and the true value image respectively, then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map;

[0022] Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training;

[0023] Step 4: Construct and minimize the composite loss function: The composite loss function is minimized, including adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss, and sliding loss.

[0024] Furthermore, in step 2, the generator 1 and the generator 2 have the same module structure, including an initial convolution, a multi-scale feature dimensionality reduction module, a stacking module, an expanded attention module, a skip connection module, a maximum pooling layer, an addition operation, a splicing operation, and a multiplication operation;

[0025] The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information;

[0026] The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampled information to obtain high-dimensional feature information for output;

[0027] The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features;

[0028] The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module to assign different weights to different regions of the image.

[0029] The maximum pooling layer reduces the input feature map size to half of its original size, facilitating further feature calculations;

[0030] The jump connection is used to fuse different information of the same size and dimension;

[0031] The splicing operation is used to perform channel fusion on different information of the same size but different dimensions to enhance the nonlinearity of features;

[0032] The addition operation is used to add and fuse different information of the same size and dimension;

[0033] The multiplication operation is used to multiply and fuse different information of the same size and dimension.

[0034] Furthermore, the composite convolutional attention module consists of a composite attention module, a convolutional feature extraction module and an enhanced feature module;

[0035] The composite attention module consists of a fully connected layer, maximum pooling, average pooling, convolution layer 1, convolution layer 2, convolution layer 3, S-type function, splicing operation and addition operation; wherein, the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and calculates the average value of each channel, the channel maximum pooling takes the maximum value of the different channel numbers of each pixel and outputs it, the channel average pooling takes the average value of the different channel numbers of each pixel and outputs it, the convolution kernel size used in convolution layer 1 and convolution layer 2 is 1×1, the step size is 1, the convolution kernel size used in convolution layer 3 is 7×7, the step size is 3,

[0036] The convolution feature extraction module consists of average pooling, convolution layer, sigmoid function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image and finds the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1;

[0037] The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection. The convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1. The convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1.

[0038] The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. Among them, the convolution blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a stride of 2. The convolution block in convolutional layer 5 uses a convolution kernel size of 4×4 and a stride of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

[0039] Furthermore, the adaptive infrared image colorization method based on human visual characteristics also includes:

[0040] Step 5: Fine-tune the model: Input the long-wave infrared image and the ground-truth image into the network model for training, compare the obtained results with image evaluation indicators, and fine-tune the network model parameters;

[0041] Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colored image.

[0042] To solve the above technical problems, the present application further proposes an adaptive infrared image colorization system based on human visual characteristics, which is used to implement the above-mentioned adaptive infrared image colorization method based on human visual characteristics. The system includes:

[0043] An image acquisition module, used for acquiring an image to be colored;

[0044] The image processing module is used to adaptively process the acquired colorized images and divide them into training sets and test sets;

[0045] The model training module is used to input the processed training set into the designed network for training, obtain the optimal generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image;

[0046] The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process and compare the final generated image with the input infrared image and the reference color image;

[0047] The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is used as the final colorization result; if the preset quality requirements are not met, the quality improvement module is activated;

[0048] The quality improvement module is used to reacquire the dataset and continue training the model with the new dataset, repeating the quality assessment module until an image that meets the preset quality requirements is generated;

[0049] The hardware deployment module is used to deploy the model that meets the quality assessment module on hardware.

[0050] Compared with the prior art, this application has the following beneficial effects:

[0051] 1. This paper proposes a multi-scale optimal sliding generative adversarial network to colorize infrared images and select the optimal solution in continuous dynamic adversarial learning, thereby obtaining more realistic colorization results.

[0052] 2. The present invention proposes an optimal sliding strategy. By solving the optimal value of the sliding step size, it achieves the minimum error when constructing the minimum loss function for the network, and improves the network color translation accuracy without increasing the network complexity.

[0053] 3. The present invention proposes a generator network, designs a multi-scale feature dimensionality reduction module, a stacking module and a composite convolutional attention module, optimizes the generation process through different modules, and obtains a colorized image that is more in line with the visual characteristics of the human eye.

[0054] 4. The present invention proposes a composite loss function composed of adversarial loss, pixel loss, depth loss, perceptual similarity loss, contour loss, segmentation loss, sliding loss and total variation loss to constrain the image color generation results and improve the image colorization quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0056] Figure 1 This is a flowchart of the steps of an adaptive infrared image colorization method based on human visual characteristics of the present invention;

[0057] Figure 2 This is a working principle diagram of an adaptive infrared image colorization method based on human visual characteristics of the present invention;

[0058] Figure 3 This is a working dynamic principle diagram of the optimal sliding strategy, countermeasure strategy and fusion module of the present invention;

[0059] Figure 4 Schematic diagram of the specific composition of the edge module of the present invention;

[0060] Figure 5 A generator network diagram for the generative adversarial network of the present invention;

[0061] Figure 6 This is a schematic diagram of the specific composition of all stacking modules of the present invention;

[0062] Figure 7 This is a schematic diagram of the specific composition of all multi-scale feature dimensionality reduction modules of the present invention;

[0063] Figure 8 This is a schematic diagram of the specific composition of all downsampling enhancement modules of the present invention;

[0064] Figure 9 This is a schematic diagram of the specific composition of all expanded attention modules of the present invention;

[0065] Figure 10 This is a schematic diagram of the specific composition of all composite convolutional attention modules of the present invention;

[0066] Figure 11 This is a schematic diagram of the specific composition of all composite attention modules of the present invention;

[0067] Figure 12 This is a schematic diagram of the specific composition of all convolutional attention modules in the present invention;

[0068] Figure 13 This is a schematic diagram of the specific composition of all enhanced feature modules of the present invention;

[0069] Figure 14 A discriminator network diagram for generating an adversarial network for the present invention;

[0070] Figure 15 A schematic diagram comparing the effects of the method proposed in the present invention and the prior art;

[0071] Figure 16 The schematic diagram of the structure of an adaptive infrared image colorization system based on human visual characteristics is shown in the figure.

[0072] Figure 17 The figure is a schematic diagram of the structure of an adaptive infrared image colorization electronic device based on the visual characteristics of the human eye. DETAILED DESCRIPTION

[0073] The present invention proposes an adaptive infrared image colorization method and system based on human visual characteristics, aiming to design an adaptive infrared image colorization method and system based on human visual characteristics.

[0074] The adaptive infrared image colorization method based on human visual characteristics proposed by the present invention will be described in the following specific embodiments:

[0075] In the technical solution of this embodiment, Figure 1 As shown, an adaptive infrared image colorization method based on human visual characteristics includes:

[0076] Step 1: Prepare the dataset and adapt: Obtain the infrared image dataset, preprocess it, and then adapt it to output fixed-size training set images. The training set is divided into the KAIST dataset and the FLIR dataset.

[0077] Step 2: Build the network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator;

[0078] The first adversarial network, consisting of Generator 1, Generator 2, and the Discriminator, continuously obtains more realistic colorization results by performing adversarial learning between the generated colorized image and the true value image. The second adversarial network, consisting of the optimal sliding strategy, adversarial strategy, and fusion module in the generative network, continuously performs adversarial learning between Image 1 and Image 2, selects the optimal generator, and completes effective dynamic segmentation and image stitching and fusion operations.

[0079] The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is as follows:

[0080]

[0081] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step size for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. nRepresents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation,

[0082] Follow K n and The quantitative relationship between them leads to K n The value of is:

[0083]

[0084] Among them, the value of k is a positive integer, e n The expression can be replaced by:

[0085]

[0086] The adversarial strategy consists of a judgment statement, which is used to complete adversarial learning and selection between image 1 and image 2. The segmentation loss is calculated for image region II of input image 1 and image region II of image 2, respectively, to obtain segmentation loss G1(II) and segmentation loss G2(II), and the sizes of G1(II) and G2(II) are compared. The following five results are obtained:

[0087]

[0088] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0089] The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy:

[0090]

[0091] in, represents the output of the fusion module, Indicates concatenating and resizing the three inputs;

[0092] The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x-direction and y-direction of the colorized image and the true value image respectively, then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map;

[0093] Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training;

[0094] Step 4: Construct and minimize the composite loss function: The composite loss function is minimized, including adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss, and sliding loss.

[0095] Furthermore, in step 2, the generator 1 and the generator 2 have the same module structure, including an initial convolution, a multi-scale feature dimensionality reduction module, a stacking module, an expanded attention module, a skip connection module, a maximum pooling layer, an addition operation, a splicing operation, and a multiplication operation;

[0096] The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information;

[0097] The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampled information to obtain high-dimensional feature information for output;

[0098] The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features;

[0099] The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module to assign different weights to different regions of the image.

[0100] The maximum pooling layer reduces the input feature map size to half of its original size, facilitating further feature calculations;

[0101] The jump connection is used to fuse different information of the same size and dimension;

[0102] The splicing operation is used to perform channel fusion on different information of the same size but different dimensions to enhance the nonlinearity of features;

[0103] The addition operation is used to add and fuse different information of the same size and dimension;

[0104] The multiplication operation is used to multiply and fuse different information of the same size and dimension.

[0105] Furthermore, the composite convolutional attention module consists of a composite attention module, a convolutional feature extraction module and an enhanced feature module;

[0106] The composite attention module consists of a fully connected layer, maximum pooling, average pooling, convolution layer 1, convolution layer 2, convolution layer 3, S-type function, splicing operation and addition operation; wherein, the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and calculates the average value of each channel, the channel maximum pooling takes the maximum value of the different channel numbers of each pixel and outputs it, the channel average pooling takes the average value of the different channel numbers of each pixel and outputs it, the convolution kernel size used in convolution layer 1 and convolution layer 2 is 1×1, the step size is 1, the convolution kernel size used in convolution layer 3 is 7×7, the step size is 3,

[0107] The convolution feature extraction module consists of average pooling, convolution layer, sigmoid function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image and finds the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1;

[0108] The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection. The convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1. The convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1.

[0109] The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. Among them, the convolution blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a stride of 2. The convolution block in convolutional layer 5 uses a convolution kernel size of 4×4 and a stride of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

[0110] Furthermore, the adaptive infrared image colorization method based on human visual characteristics also includes:

[0111] Step 5: Fine-tune the model: Input the long-wave infrared image and the ground-truth image into the network model for training, compare the obtained results with image evaluation indicators, and fine-tune the network model parameters;

[0112] Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colored image.

[0113] Example 1:

[0114] An embodiment of the present invention proposes an adaptive infrared image colorization method based on human visual characteristics, such as Figure 1 As shown, the following steps are included:

[0115] Step 1. Prepare the dataset and adapt it: Both the KAIST dataset and the FLIR dataset are composed of continuous frames of different videos. The adjacent images are not very different. Therefore, a certain degree of cleaning is performed on both datasets. The cleaned images are then adaptively output to 320×256 images as the input of the entire network. Finally, 4755 images are obtained from the KAIST dataset as training dataset 1, and 3918 images are obtained from the FLIR dataset as training dataset 2.

[0116] Step 2: Build a network model. The working principle diagram of the network model is as follows: Figure 2 As shown, it specifically includes the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator;

[0117] like Figure 3 As shown in Figure ①, the n-th sliding process is shown. The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. The image cropped by straight line l1 is divided into image I and image II, and the image cropped by straight line l2 is divided into image I and image III. The intersection between image I and image II is defined as image II. In the optimal sliding strategy, straight lines l1 and l2 are dynamically learned to move, and their moving distance l s And the direction is controlled by the adversarial strategy; in order to prevent the straight lines l1 and l2 from exceeding the edge of the image, the size of the straight lines l1 and l2 is restricted, and the range of the two straight lines l1 is 0 to the pixel length of the image l x , the value range of the line l2 is 0 to the pixel length of the image l x In order to ensure that part I of image 1 and part II of image 2 have the same size and image information, the pixel difference between the fixed lines l1 and l2 is a. The value of constant a is set by the program at the beginning. The default specific quantitative relationship is: Where a is the fixed pixel difference between lines l1 and l2, l x is the pixel size of the input image, To round down, is the pixel length of the line l1 relative to the lower left corner of the image, is the pixel length of line l2 relative to the lower left corner of the image;

[0118] Since the last cropped image 1 and image 2 need to be input to generator 1 and generator 2 respectively to generate the colorization task, it is necessary to transform the size of the cropped image 1 and image 2 to the input image size of the generator. The input image size of the generator is the same as the output image size. The present invention focuses on the step size l of the movement. s The relationship between the output image size of the generator after mapping to the transformation is considered, and the sliding loss designed based on the mapping distance is the core loss of the network of the present invention. Therefore, accurate calculation and optimization of the mapping distance are necessary; for the step size l s The mapping distance formula is as follows:

[0119]

[0120] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step size for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, where the value is an integer, e n It represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation. Therefore, in order to obtain a more accurate mapping distance, the mapping loss e is required. n Should be small enough; focus on K n and The quantitative relationship between them leads to K n The value of is:

[0121]

[0122] Among them, the value of k is a positive integer, so e n The expression can be replaced by:

[0123]

[0124] At this time n Transformed into a binary equation, the analysis function graph first rises and then falls, so there is an optimal point. By running the program, it can be obtained in different There is a step size under the input The optimal solution of and The distance between them is fixed, so the straight line The optimal solution for the moving step size is

[0125] like Figure 3 As shown in Figure ③, the adversarial strategy consists of judgment statements to complete the adversarial learning and selection between image one and image two. The adversarial strategy selects the image information generated by the best generator by judging the fixed image loss at the intersection of image one and image two. The specific working principle is as follows: the segmentation loss is calculated for image region II of input image one and image region II of image two respectively, and the corresponding segmentation loss G1(II) and segmentation loss G2(II) are obtained respectively. The sizes of G1(II) and G2(II) are compared, and the following five structures appear:

[0126]

[0127] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0128] like Figure 3 As shown in Figure ②, the fusion module mainly completes the splicing of the segmented images through the splicing operation using the PIL function in the Python language library. Different output results are obtained for different situations of the adversarial strategy:

[0129]

[0130] in, represents the output of the fusion module, Indicates concatenating and resizing the three inputs;

[0131] like Figure 4 As shown in the figure, the edge module mainly uses the Sobel edge detection algorithm to detect and display the edge information of the image in different directions; the edge information of the colorized image and the true value image is collected in the x-direction and y-direction respectively, and then the feature fusion is performed through the addition operation to obtain the edge information map, and then the contour loss calculation is performed on the edge information map;

[0132] like Figure 5As shown, the generator consists of initial convolution, stacking module, multi-scale feature dimensionality reduction module 1, maximum pooling layer 1, dilated attention module 1, multi-scale feature dimensionality reduction module 2, maximum pooling layer 2, dilated attention module 2, multi-scale feature dimensionality reduction module 3, maximum pooling layer 3, dilated attention module 3, multi-scale feature dimensionality reduction module 4, maximum pooling layer 4, dilated attention module 4, jump connection, multiplication operation, splicing operation and addition operation; the initial convolution maps the input image channel to high-dimensionality to extract high-dimensional detail information. The convolution kernel size is 1×1 and the step size is 1, which increases the number of image channels. The stacking module consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11, convolution layer 12, convolution layer 13, splicing operation and addition operation. The convolution kernel size used in convolution layer 1, convolution layer 2 and convolution layer 13 is 1×1 and the step size is 1. The convolution kernel size used in convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11 and convolution layer 12 is 3×3 and the step size is 1. The composition network of the stacking module is as follows: Figure 6 As shown in the figure, the multi-scale feature dimensionality reduction module downsamples the input image in different ways, obtains the target image from multiple angles, and obtains a more accurate downsampling result. It consists of convolution layer 1, convolution layer 2, maximum pooling, downsampling enhancement module and splicing operation. The specific composition of each multi-scale feature dimensionality reduction module is shown in the figure. Figure 7As shown in Figure 1, convolution layer 1 downsamples the image to extract abstract information. The convolution kernel size is 3×3 and the step size is 2. Convolution layer 2 convolutionally fuses the convolution layer 1 and the maximum pooling result, and the output image size remains unchanged. The convolution kernel size is 3×3 and the step size is 1. The size of the pooling kernel of the maximum pooling is 2×2. The multi-scale feature dimensionality reduction module consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11, convolution layer 12, convolution layer 13, convolution layer 14, convolution layer 15, convolution layer 16, convolution layer 17, convolution layer 18, normalization operation, L-type function, R-type function, splicing operation, multiplication operation and addition operation. The convolution kernel size of convolution layer 1 and convolution layer 18 is 1×1 and the step size is 1. Convolution layer 2, convolution layer 3, The convolution kernel size of convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10 and convolution layer 14 is 3×3, and the step size is 1. Convolution layer 11, convolution layer 12, convolution layer 13, convolution layer 14, convolution layer 15, convolution layer 16 and convolution layer 17 are hollow convolutions with a convolution kernel size of 3×3, a step size of 1, an image padding of 2, and a hollow rate of 2. The maximum pooling layer converts low-dimensional information into high-dimensional information for high-dimensional information extraction and splicing, thereby obtaining more comprehensive feature information. Among them, the maximum pooling layer 1, maximum pooling layer 4 and maximum pooling layer 6 are maximum pooling layers with a pooling kernel of 2×2, the maximum pooling layer 2 and maximum pooling layer 5 are maximum pooling layers with a pooling kernel of 4×4, and the maximum pooling layer 3 is a maximum pooling layer with a pooling kernel of 8×8. The network composition of the multi-scale feature dimensionality reduction module is as follows: Figure 8 As shown in Figure 2; the expanded attention module consists of a convolutional layer block, normalization, L-type function and attention module. The network composition of the expanded attention module is as follows: Figure 9 As shown, the convolution layer block uses deconvolution, the deconvolution kernel size is 4×4, and the stride is 2.

[0133] The composite convolution attention module consists of a composite attention module, a convolution feature extraction module, and an enhanced feature module. The network composition of the composite convolution attention module is as follows: Figure 10 As shown in , the composite attention module consists of a fully connected layer, maximum pooling, average pooling, convolution layer 1, convolution layer 2, convolution layer 3, S-type function, splicing operation and addition operation. The fully connected layer compresses the input channel to 8 times the original, and then restores it to the number of input channels. The maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2. The average pooling calculates the input image and calculates the average value of each channel. The channel maximum pooling takes the maximum value of the different channel numbers of each pixel and outputs it. The channel average pooling takes the average value of the different channel numbers of each pixel and outputs it. The convolution kernel size used in convolution layer 1 and convolution layer 2 is 1×1, with a step size of 1. The convolution kernel size used in convolution layer 3 is 7×7, with a step size of 3. The network composition of the composite attention module is as follows: Figure 11 As shown in Figure 2, the convolutional feature extraction module consists of average pooling, convolution layer, S-type function, multiplication operation, addition operation, jump connection and splicing operation. Average pooling calculates the input image and finds the average value of each channel. The convolution layer uses a convolution kernel size of 1×1 and a step size of 1. The network composition of the convolutional feature extraction module is shown in Figure 2. Figure 12 As shown in Figure 2, the enhanced feature module consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, addition operation and skip connection. The convolution kernel size used in convolutional layer 1 is 3×3 and the step size is 1. The convolution kernel size used in convolutional layer 2 and convolutional layer 3 is 1×1 and the step size is 1. The network composition of the enhanced feature module is as follows: Figure 13 As shown in Figure 2; the discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L function and compound convolutional attention module. The convolution blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a step size of 2. The convolution block in convolutional layer 5 uses a convolution kernel size of 4×4 and a step size of 1. The compound convolutional attention module is the same as the compound convolutional attention module network in the generator. The composition network of the discriminator is shown in Figure 2. Figure 14 As shown;

[0134] In general, the infrared image colorization process is to determine the input infrared data set, screen and adapt the data set, and divide the input image into image 1 and image 2 through the optimal sliding strategy. Image 1 is divided into image I and part II of image 1, and both are input into generator 1. Image 2 is divided into part II of image 2 and image III, and both are input into generator 2. The segmentation loss of the intersecting area of image 1 and image 2, that is, part II of image 1 and part II of image 2, is compared through the adversarial strategy. The optimal generator result is selected, and the optimal sliding step size is obtained using the optimal sliding strategy to guide the next round of dynamic segmentation straight lines. The generated results of generators 1 and 2 are input into the fusion module for fusion and splicing to output the final colorized image; the infrared image is used as a reference and the colorized image and the true value image are used as discriminator inputs respectively to identify the difference between the generated colorized image and the true value image, and the difference is quantified in the range of 0-1 to complete the evaluation of the colorization degree of the image;

[0135] In order to ensure the robustness of the network and retain more network information, the present invention uses four activation functions, namely L-type function, R-type function, G-type function and S-type function. The last layer of the generator uses the L-type function, the last layer of the discriminator uses the S-type function, the multi-scale feature dimensionality reduction module uses the R-type function, and the expanded attention module uses the L-type function and the S-type function. At the same time, all activation functions in the generator and discriminator in the present invention are L-type functions; the L-type function, R-type function, G-type function and S-type function are defined as follows

[0136]

[0137] In order to enhance the generalization ability of the network and prevent data overfitting, normalization operations are generally used in this invention. In this invention, all normalization operations are performed using batch normalization. Normalization is used before various activation functions to make the output of the previous layer distributed with a mean of 0 and a variance of 1, that is, to normalize the input of the next layer, so that it can have a certain gradient when passing through the activation function, thereby avoiding the value being too large and entering the saturation area.

[0138] Step 3: Train the network model: input the training dataset 1 and the training dataset 2 in the dataset obtained by S1 into the network model constructed by S2 for training;

[0139] Step 4: Obtain the minimized loss function value and optimal evaluation index: By constructing a reasonable loss function, the model can be constrained to optimize in the direction of the designer's attention. Therefore, the smaller the loss function, the better the robustness of the model. Selecting an appropriate evaluation index can effectively reflect the quality of the model and the degree of image distortion, and measure the role of the colorization network.

[0140] In step 4, the network output and label loss function are calculated to achieve better fusion effect by minimizing the loss function; during the training process, the loss function uses adversarial loss Pixel loss Depth Loss Perceptual Similarity Loss Contour loss Segmentation loss Sliding loss and the total variational loss Combine them optimally with certain weights to obtain the minimized total loss function;

[0141] In order to encourage the network to output color results with more realistic details, an adversarial loss is used; the adversarial loss is used to learn the implicit relationship between the thermal infrared image and the colorized image, which is defined as:

[0142]

[0143] Where x is the input infrared image, y is the input ground truth image, G(·) is the output of the entire generative network, and D(·,·) are the two images input to the discriminator respectively;

[0144] In order to maintain the similarity of brightness and contrast between the colorized image generated by the generator and the real image, a pixel loss function is designed to standardize the generation of the colorized image. By designing and using the L1 norm to standardize the differences between pixels, a better colorization effect is obtained. The pixel loss is defined as:

[0145]

[0146] Among them, A represents image 1, B represents image 2, G1(·) represents the output result of generator 1, G2(·) represents the output result of generator 2, and W A is the width of the input image, H A is the height of the input image, W B is the width of input image 2, H B is the height of the input image 2, S(y) represents the ground truth image of the corresponding area of the ground truth image, and ‖·‖1 represents the L1 norm of the given output;

[0147] The depth loss can effectively minimize the brightness and contrast differences between the color image and the colorized image. If the adversarial network focuses too much on the depth loss, the brightness and contrast in the thermal infrared image will be lost. In order to prevent the generator from over-representing the pixel-to-pixel relationship, an appropriate weight of the depth loss will be added. The depth loss can be expressed as:

[0148]

[0149] Among them, φ k (·) represents the feature representation of the kth max pooling layer in the VGG-16 network, C A_k H A_k W A_k represents the size of the image-feature representation, C B_k H B_k W B_k represents the size of the second feature representation of the image, ‖·‖1 represents the L1 norm; where A and The quantitative relationship between them can be expressed as: Among them B and The quantitative relationship between them can be expressed as:

[0150] In order to compensate for the differences in brightness, contrast, and network quality of images, this paper uses perceptual similarity loss to penalize the generated images so that they reach the optimal value faster. The perceptual similarity loss is defined as follows:

[0151]

[0152] Among them, ssim(·,·) indicates the use of the perceptual similarity module function integrated by pytorch for calculation;

[0153] In order to highlight the contour features of the image, this paper proposes to use contour loss to reduce the boundary blurring phenomenon of contour information in image translation tasks, thereby increasing the leading role of the colorization task on other image tasks; the contour loss is defined as follows:

[0154]

[0155] Where W is the width of the input image, H is the height of the input image, represents the output of the edge module passing the image, ‖·‖1 represents the L1 norm of the given output, is the output of the fusion module, and the transformation relationship is: in Represents the output of the fusion module relative to image one and image two;

[0156] In order to limit the generation of segmented images with the same color and texture features as the true image, the segmentation loss is obtained by taking the colorization result of the segmented image and the corresponding position of the true image, which can convert the more realistic colorization result into a low-level segmentation loss; since the segmentation results of the straight lines l1 and l2 divide the image into four parts, but the image I divided by the straight line l1 and the image III divided by the straight line l2 do not have much meaning, the segmentation loss body calculates the segmentation loss of image one and image two. Since image one can be broadly divided into part I of image one and part II of image one, the segmentation losses are defined as and Since image 2 can be broadly divided into image 2 part II and image 2 part III, the segmentation loss is defined as and and total segmentation loss Defined as:

[0157]

[0158]

[0159] Where W is the width of the input image, H is the height of the input image, S.(y) represents the true value image of the corresponding area of the true value image, and λ A_I_1 ,λ A_I_2 ,λ A_I_3 ,λ A_II_1 ,λ A_II_2 ,λ A_II_3 ,λ B_II_1 ,λ B_II_2 ,λ B_II_3 ,λ B_III_1 ,λ B_III_2 and λ B_III_3are all constants, with a default value of 1, λ sl_1 ,λ sl_2 ,λ sl_3 and λ sl_4 They are all constants, and their default values are 1, 0.5, 1, and 0.5 respectively;

[0160] In order to utilize the characteristic information and promote the interpretation and translation of the color information of the real image, the present invention proposes a sliding loss, which generates a more colorful colorized image by increasing the weight of the sliding loss in the total loss; the sliding loss is defined as:

[0161]

[0162] Among them, l s Indicates the step length of linear movement, l w Indicates that the generator needs to input the image size length, l h Indicates that the generator needs to input image size width;

[0163] In order to achieve better model performance, the above eight losses are combined to design the total loss of the model, constrain the model as a whole, and the total loss function is designed as

[0164]

[0165]

[0166] Among them, λ adv ,λ pixel ,λ df ,λ ssim ,λ tv ,λ sl ,λ el and λ step They represent the weights that control the complete loss function. The weights are set based on preliminary experiments on the training dataset. The final weights are set to 0.03, 1, 1, 1.25, 1.25, 1, 1, and 5 respectively.

[0167] In step 4, appropriate evaluation metrics for the network include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), and Perceptual Image Similarity (LPIPS). PSNR is a fundamental metric in objective image quality evaluation. Its value directly reflects the pixel-level mean squared error (MSE) between the generated image and the original image. A larger PSNR value indicates less image distortion, better quality, and a more balanced color distribution. Structural Similarity (SSIM): Perceptual similarity is an objective evaluation metric that measures image similarity in terms of brightness, contrast, and network quality. It reflects the subjective quality of an image. A higher SSIM value indicates a higher degree of image restoration and better visual quality. Perceptual Image Similarity (LPIPS): The LPIPS metric aims to quantify the perceptual similarity between colorized images and colorized images. It is related to human visual perception. By training on manually annotated datasets, it learns feature extraction and distance metrics that better reflect human visual perception and better reflect the subjective perception of the degree of image perceptual information retention. A lower LPIPS value indicates a closer quality between the generated image and the ground-truth image, resulting in a better image quality. The definitions of PSNR, SSIM, and Perceptual Image Similarity are as follows:

[0168]

[0169] Among them, (2 n -1) 2 It represents the square of the maximum signal value, the number of bits of each sample value, and the maximum value of 8-bit representation is 255; MSE represents the mean of the squares of the differences between corresponding pixels of two images, and the expression is as follows: μ x and μ y Represent the mean and variance of images x and y respectively, and Represents the standard deviation of images x and y, σ xy represents the covariance of images x and y, C1 and C2 are constants; d is the distance between x and x0, and w1 is a trainable weight parameter;

[0170] The number of training runs was set to 200, with the learning rate set to 0.0001 for the first 100 training runs and gradually decreasing from 0.0001 to 0 for the next 100 training runs. The number of input images was fixed at 1 per run. The Adam optimizer was selected as the network parameter optimizer. Its advantages lie in its simplicity of implementation, high computational efficiency, low memory requirements, and the fact that parameter updates are not affected by gradient scaling, resulting in relatively stable parameters. When the discriminator's ability to determine whether a generated colorized image is a true color image is balanced with the generator's ability to generate colorized images, the network is considered to have been trained, and the discriminator output should be close to 0.5.

[0171] Step 5: Fine-tune the model: Use test dataset 1 and test dataset 2 to test and fine-tune the trained network. Use test dataset 1 and test dataset 2 as input, calculate the evaluation index value in step 4, and then compare it with the expected evaluation index value. Then fine-tune the model until the expected evaluation index value is achieved.

[0172] Step 6. Save the model: Encapsulate the model that meets the desired evaluation indicators and save all parameter information in the network; the network can be applied to infrared image input of any size and complete the task of converting the input infrared image into the generated colorized image;

[0173] Among them, the implementation of convolution, activation function, splicing operation, addition operation, multiplication operation and normalization are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0174] The present invention constructs an adaptive infrared image colorization method based on human visual characteristics, which can convert an input infrared image into a corresponding colorized image without going through other intermediate steps, avoiding the manual design of relevant colorization rules or other additional annotations; under the same experimental conditions, the feasibility and superiority of the method are further verified by calculating the relevant evaluation indicators of the colorized images obtained by the existing method; the relevant indicators of the existing technology and the method proposed by the present invention are compared. Figure 15 As shown;

[0175] Existing Method 1: A generative adversarial network for infrared image colorization consists of a generator and a discriminator. The generator uses three downsampling modules, nine residual modules, and three upsampling modules; the discriminator uses six convolutional layers and a sigmoid function to distinguish generated images from real images. This method was trained and tested using the training and test sets described in step 1 of the present invention, respectively.

[0176] Existing Method 2: A generative adversarial network for infrared image colorization consists of a generator and a discriminator. The generator first uses 3×3 max pooling to reduce the size of the input image. Then, it passes through four downsampling modules, nine residual modules, and four upsampling modules, and finally outputs the image using a sigmoid function. The discriminator uses six convolutional layers and a sigmoid function to distinguish the generated image from the real image. This method was trained and tested using the training and test sets described in step 1 of the present invention, respectively.

[0177] Existing Method 3: A generative adversarial network for infrared image colorization consists of a generator and a discriminator. The generator uses four downsampling modules and transfers features via residual connections. The model output is then processed through four attention upsampling modules. The resulting residuals are summed and averaged to produce the final output. The discriminator uses six convolutional layers and an R-type function to distinguish the generated image from the real image. This method was trained and tested using the training and test sets described in step 1 of the present invention, respectively.

[0178] Existing Method 4: A generative adversarial network for infrared image colorization consists of a generator and a discriminator. The generator uses four downsampling attention modules, each followed by a residual module for feature extraction, followed by four upsampling attention modules for feature expansion, ultimately generating a model output. The discriminator uses six convolutional layers and an R-type function to distinguish generated images from real images. This method was trained and tested using the training and test sets described in step 1 of the present invention, respectively.

[0179] Figure 15 The colorization effect evaluation of this method and the existing four algorithms on the KAIST dataset and FLIR dataset is shown in Figure 2. Figure 15 It can be seen that the method proposed in the present invention has a higher peak signal-to-noise ratio, higher structural similarity and lower perceived image similarity than the existing method. These indicators further illustrate that the method proposed in the present invention has better colorization quality.

[0180] Example 2:

[0181] An adaptive infrared image colorization system based on human visual characteristics, such as Figure 16 As shown, the system mainly includes an image acquisition module, an image processing module, a model training module, a feature visualization module, a quality assessment module, a quality improvement module and a hardware deployment module;

[0182] An image acquisition module, used for acquiring an image to be colored;

[0183] The image processing module is used to adaptively process the acquired colorized images and divide them into training sets and test sets;

[0184] The model training module is used to input the processed training set into the designed network for training, obtain the optimal generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image;

[0185] The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process and compare the final generated image with the input infrared image and the reference color image to further visually observe the image colorization effect;

[0186] The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is used as the final colorization result; if the preset quality requirements are not met, the quality improvement module is activated;

[0187] The quality improvement module is used to reacquire the dataset and continue training the model with the new dataset, repeating the quality assessment module until an image that meets the preset quality requirements is generated;

[0188] The hardware deployment module is used to deploy the model that meets the quality assessment module on hardware. Deploying the model on suitable hardware can help users complete the infrared image colorization process more conveniently.

[0189] Furthermore, the image to be colored in the image acquisition module is an infrared image, or a grayscale image, but a true value image is required;

[0190] Furthermore, the adaptive processing in the image processing module includes image cropping and image flipping, and the image ratio of the training set and the test set is finally 10:1;

[0191] Furthermore, the model training module fixes each image from different size inputs to a size of 320×256; trains for a total of 200 rounds with a batch size of 4; uses a learning rate training strategy, setting the learning rate to 0.002 in the first 100 rounds and gradually decreasing it from 0.002 to 0 in the next 100 rounds; the number of filters in the generator and discriminator is set to 64; and the present invention uses the Adam optimizer;

[0192] Furthermore, the visualization image during the training process in the feature visualization module is the attention heat map in the composite convolutional attention module. By observing the attention heat map, it is possible to clearly understand whether the focus area of the attention module is the target area; after the training is completed and the visualization image is saved, the model is predicted, and the predicted image is compared with the input infrared image and the true value image, so that the colorization effect can be intuitively felt;

[0193] Furthermore, the evaluation indicators of the quality assessment module are peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and perceived image similarity (LPIPS); the higher the value of the peak signal-to-noise ratio, the better; the higher the value of the structural similarity, the better; the higher the value of the perceived image similarity, the better;

[0194] Furthermore, the hardware deployment of the hardware deployment module needs to be deployed on suitable hardware that supports the algorithm model. The hardware should at least include memory, processor and communication interface. By storing the input model in the memory, the model can be read at any time. The processor is used to load the model and perform calculations. The communication interface is used to accept the input training data set and test data set, and output the output results for display or transmission to other terminal devices.

[0195] Example 3:

[0196] An adaptive infrared image colorization electronic device based on human visual characteristics, such as Figure 17 As shown, it mainly includes a memory, a communication interface and a bus; wherein the memory, the processor and the communication interface are used to execute each step of the adaptive infrared image colorization method based on human visual characteristics of embodiment 1 of the present invention;

[0197] The memory can be a ROM, a static storage device, a dynamic storage device or a RAM; the memory can store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to perform the various steps of the training method of the infrared image colorization network according to the embodiment of the present invention;

[0198] The processor can be a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute relevant programs to implement the functions required to be performed by the units in the infrared image colorization training system of the present invention, or to perform the infrared image colorization training method of the present invention;

[0199] The processor can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the infrared image colorization training method of the present invention can be completed by hardware integrated logic circuits or software instructions in the processor. The above-mentioned processor can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, capable of implementing or executing the infrared image colorization method and system, steps and logic block diagram of the present invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0200] The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or communication networks; for example, it is possible to obtain an image to be processed or an initial feature map of the image to be processed through the communication interface;

[0201] A bus may include a pathway that transfers information between various components of a system (eg, memory, processor, communication interface).

[0202] Example 4:

[0203] A computer-readable storage medium for adaptive infrared image colorization based on human visual characteristics, wherein the computer-readable storage medium can exist independently and not be incorporated into a device; the computer-readable storage medium stores one or more programs, which are used by one or more processors to execute the method provided in Example 1;

[0204] It should be noted that although Figure 17 The simplified diagram of the electronic device system shown only shows the memory, processor and communication interface. However, in the specific implementation process, those skilled in the art should understand that the system also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the system may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the system may also include only the devices necessary to implement the embodiments of the present invention, and does not necessarily include Figure 17 All devices shown in .

[0205] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An adaptive infrared image colorization method based on human visual characteristics, characterized in that: include: Step 1: Prepare the dataset and adapt: Obtain the infrared image dataset, preprocess the infrared image dataset, and output a fixed-size training set image through adaptation; It is divided into training set 1 KAIST dataset and training set 2 FLIR dataset; Step 2: Build the network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator. The first adversarial network consists of generator 1, generator 2 and discriminator; the second adversarial network consists of the optimal sliding strategy, adversarial strategy and fusion module. The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is: in, Indicates the step size of the nth calculation of the program, Indicates the l1 length of the program during the nth calculation, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step size for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. n Represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation, Follow K n and The quantitative relationship between them leads to K n The value of is: Among them, the value of k is a positive integer, e n Replace the expression with: The adversarial strategy consists of a judgment statement, which is used to complete adversarial learning and selection between image 1 and image 2. The segmentation loss is calculated for image region II of input image 1 and image region II of image 2, respectively, to obtain segmentation loss G1(II) and segmentation loss G2(II), and the sizes of G1(II) and G2(II) are compared. The following five results are obtained: in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image 2, l x is the pixel length; The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy: in, represents the output of the fusion module, Indicates concatenating and resizing the three inputs; The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x-direction and y-direction of the colorized image and the true value image respectively, then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map; Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training; Step 4: Construct and minimize the composite loss function: The composite loss function is minimized, including adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss, and sliding loss.

2. The adaptive infrared image colorization method based on human visual characteristics according to claim 1, characterized in that: In step 2, the generator 1 and the generator 2 have the same module structure, including initial convolution, multi-scale feature dimensionality reduction module, stacking module, dilated attention module, skip connection module, maximum pooling layer, addition operation, splicing operation and multiplication operation; The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information; The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampled information to obtain high-dimensional feature information for output; The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features; The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module to assign different weights to different regions of the image. The maximum pooling layer reduces the input feature map size to half of its original size, facilitating further feature calculations; The jump connection is used to fuse different information of the same size and dimension; The splicing operation is used to perform channel fusion on different information of the same size but different dimensions to enhance the nonlinearity of features; The addition operation is used to add and fuse different information of the same size and dimension; The multiplication operation is used to multiply and fuse different information of the same size and dimension.

3. The adaptive infrared image colorization method based on human visual characteristics according to claim 2, characterized in that: The composite convolutional attention module consists of a composite attention module, a convolutional feature extraction module and an enhanced feature module; The composite attention module consists of a fully connected layer, maximum pooling, average pooling, convolution layer 1, convolution layer 2, convolution layer 3, S-type function, splicing operation and addition operation; wherein, the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and calculates the average value of each channel, the channel maximum pooling takes the maximum value of the different channel numbers of each pixel and outputs it, the channel average pooling takes the average value of the different channel numbers of each pixel and outputs it, the convolution kernel size used in convolution layer 1 and convolution layer 2 is 1×1, the step size is 1, the convolution kernel size used in convolution layer 3 is 7×7, the step size is 3, The convolution feature extraction module consists of average pooling, convolution layer, sigmoid function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image and finds the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1; The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection. The convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1. The convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1. The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. Among them, the convolution blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a stride of 2. The convolution block in convolutional layer 5 uses a convolution kernel size of 4×4 and a stride of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

4. The adaptive infrared image colorization method based on human visual characteristics according to claim 1, characterized in that: Also includes: Step 5: Fine-tune the model: Input the long-wave infrared image and the ground-truth image into the network model for training, compare the obtained results with image evaluation indicators, and fine-tune the network model parameters; Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colored image.

5. An adaptive infrared image colorization system based on human visual characteristics, used to execute the adaptive infrared image colorization method based on human visual characteristics according to any one of claims 1 to 4, characterized in that: The system comprises: An image acquisition module, used for acquiring an image to be colored; The image processing module is used to adaptively process the acquired colorized image and divide it into a training set and a test set; The model training module is used to input the processed training set into the designed network for training, obtain the optimal generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image; The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process and compare the final generated image with the input infrared image and the reference color image; The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is used as the final colorization result; if the preset quality requirements are not met, the quality improvement module is activated; The quality improvement module is used to reacquire the dataset and continue training the model with the new dataset, repeating the quality assessment module until an image that meets the preset quality requirements is generated; The hardware deployment module is used to deploy the model that meets the quality assessment module on hardware.

Images