Self-adaptive infrared image colorization method and system based on human visual features

Abstract

The invention discloses a self-adaptive infrared image colorization method and system based on human visual features, and relates to the technical field of image processing, and the method comprises the following steps: preparing a data set, and carrying out the self-adaption; constructing a network model; training a network model; constructing a minimum composite loss function; finely adjusting the model; storing the model; and finally, solidifying and outputting the finally determined model parameters as a model file, and then directly calling the model to carry out colorization output in future testing and model hardware deployment to obtain a colorization image. According to the invention, through deep mining of human eye visual features, application of an overlapping principle in the human eye visual features, introduction of a generation strategy and design of a generator, an output infrared image colorization result is more real and accords with local and overall feelings of human eye visual observation.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an adaptive infrared image colorization method and system based on human visual characteristics. Background Art

[0002] Image colorization technology is an image processing technology that converts the collected images into more visually universal images. In the field of image colorization, infrared image colorization, as an emerging technology, is being widely used in many industries such as military, medicine, security and industry, and has important research value. However, there are still many challenges in converting single-channel infrared images into three-channel RGB images, including poor quality of generated images, serious loss of detail information, and difficulty in collecting paired data sets. With the continuous deepening of research on human visual characteristics, scholars in various fields have been inspired to propose cutting-edge technologies such as convolutional layers, pooling layers, normalization layers, and attention mechanisms that are widely used in the field of deep learning. Continuously exploring human body characteristics is still of great significance in today's era.

[0003] The Chinese patent publication number is "CN114067018B", and the name is "A method and system for colorizing infrared images based on dilated residual generative adversarial networks". This method first constructs four modules: shallow feature extraction module, deep feature extraction module, information fusion module, and upsampling module; then, by simulating the degradation of the data set used, the obtained high-resolution and low-resolution image pairs are used to train the entire convolutional neural network; then, by training the network model, the model is fine-tuned by minimizing the loss value; finally, the obtained model parameters are solidified to facilitate the direct call of parameters for image colorization next time. The colorization result obtained by this method has serious loss of picture details, which is quite different from the visual effect of the human eye, and has high requirements on the format and quality of the data set, and poor robustness.

[0004] In summary, how to propose an adaptive infrared image colorization method and system based on human visual characteristics to solve the above problems is an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The technical solution of the present invention to solve the above technical problems is to provide an adaptive infrared image colorization method based on human visual characteristics, comprising:

[0006] Step 1: Prepare the data set and adapt: ​​obtain the infrared image data set, preprocess the infrared image data set, and output the fixed-size training set image after adaptation; it is divided into training set 1 KAIST data set and training set 2 FLIR data set;

[0007] Step 2: Construct the network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator; the first adversarial network is an adversarial generative network composed of generator 1, generator 2 and discriminator, which continuously obtains more realistic colorization results by completing adversarial learning between the generated colorized image and the true value image; the second adversarial network is an adversarial network composed of the optimal sliding strategy, adversarial strategy and fusion module in the generative network, which continuously conducts adversarial learning between image 1 and image 2, selects the optimal generator, and completes effective dynamic segmentation and image splicing and fusion operations;

[0008] The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is as follows:

[0009]

[0010] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step length for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. n Represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation,

[0011] Follow K n and The quantitative relationship between them gives K n The value of is:

[0012]

[0013] Among them, k is a positive integer, e n Replace the expression with:

[0014]

[0015] The adversarial strategy consists of judgment statements, which are used to complete adversarial learning and selection between image one and image two; the segmentation loss is calculated for image region II of input image one and image region II of image two, respectively, and the segmentation loss G1(II) and the segmentation loss G2(II) are obtained respectively, and the sizes of G1(II) and G2(II) are compared, and there are the following five results:

[0016]

[0017] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0018] The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy:

[0019]

[0020] in, represents the output of the fusion module, Indicates concatenation and resizing of three inputs;

[0021] The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x direction and the y direction of the colorized image and the true value image respectively, and then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map;

[0022] Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training;

[0023] Step 4, construct and minimize the composite loss function: The minimized composite loss function includes adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss and sliding loss.

[0024] Furthermore, in step 2, the generator 1 and the generator 2 have the same module structure, including an initial convolution, a multi-scale feature dimension reduction module, a stacking module, an expanded attention module, a skip connection module, a maximum pooling layer, an addition operation, a concatenation operation, and a multiplication operation;

[0025] The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information;

[0026] The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampling information to obtain high-dimensional feature information for output;

[0027] The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features;

[0028] The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module, which gives different weights to different areas of the image;

[0029] The maximum pooling layer reduces the input feature map size to half of the original size, facilitating further feature calculations;

[0030] The jump connection is used to fuse different information of the same size and dimension;

[0031] The splicing operation is used to perform channel fusion of different information of the same size but different dimensions to enhance the nonlinearity of features;

[0032] The addition operation is used to add and fuse different information of the same size and dimension;

[0033] The multiplication operation is used to multiply and fuse different information of the same size and dimension.

[0034] Furthermore, the composite convolutional attention module is composed of a composite attention module, a convolutional feature extraction module and an enhanced feature module;

[0035] The composite attention module consists of a fully connected layer, a maximum pooling, an average pooling, a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, an S-type function, a splicing operation and an addition operation; wherein the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and finds the average value of each channel, the channel maximum pooling takes the maximum value of the number of different channels for each pixel and outputs it, the channel average pooling takes the average value of the number of different channels for each pixel and outputs it, the convolutional kernel size used in the convolutional layer 1 and the convolutional layer 2 is 1×1, the step size is 1, the convolutional kernel size used in the convolutional layer 3 is 7×7, the step size is 3,

[0036] The convolution feature extraction module is composed of average pooling, convolution layer, S-type function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image to find the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1;

[0037] The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection, wherein the convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1, and the convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1.

[0038] The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. The convolutional blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a step size of 2. The convolutional block in convolutional layer 5 uses a convolution kernel size of 4×4 and a step size of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

[0039] Furthermore, the adaptive infrared image colorization method based on human visual characteristics also includes:

[0040] Step 5, fine-tuning the model: input the long-wave infrared image and the true value image into the network model for training respectively, compare the obtained results with the image evaluation index, and fine-tune the network model parameters;

[0041] Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colorized image.

[0042] In order to solve the above technical problems, the present application also proposes an adaptive infrared image colorization system based on human eye visual features, which is used to execute the adaptive infrared image colorization method based on human eye visual features as described above, and the system comprises:

[0043] An image acquisition module, used for acquiring an image to be colored;

[0044] An image processing module is used to adaptively process the acquired colorized image and divide it into a training set and a test set;

[0045] The model training module is used to input the processed training set into the designed network for training, obtain the best generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image;

[0046] The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process and compare the final generated image with the input infrared image and the reference color image.

[0047] The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is used as the final colorization result; if the preset quality requirements are not met, the quality improvement module is started;

[0048] The quality improvement module is used to reacquire the data set and continue to train the model with the new data set, repeating the quality assessment module until an image that meets the preset quality requirements is generated;

[0049] The hardware deployment module is used to deploy the model that meets the quality assessment module on the hardware.

[0050] Compared with the prior art, this application has the following beneficial effects:

[0051] 1. The present invention proposes a multi-scale optimal sliding generative adversarial network, which colors infrared images and selects the optimal solution in continuous dynamic adversarial learning to obtain a more realistic colorization result.

[0052] 2. The present invention proposes an optimal sliding strategy. By solving the optimal value of the sliding step size, it achieves the minimum error when constructing the minimum loss function for the network, thereby improving the accuracy of network color translation without increasing the complexity of the network.

[0053] 3. The present invention proposes a generator network, designs a multi-scale feature dimensionality reduction module, a stacking module and a composite convolutional attention module, optimizes the generation process through different modules, and obtains a colorized image that is more in line with the visual characteristics of the human eye.

[0054] 4. The present invention proposes a composite loss function composed of adversarial loss, pixel loss, depth loss, perceptual similarity loss, contour loss, segmentation loss, sliding loss and total variation loss to constrain the image color generation result and improve the image colorization quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0056] Figure 1 A flowchart of the steps of an adaptive infrared image colorization method based on human visual characteristics of the present invention;

[0057] Figure 2 This is a working principle diagram of an adaptive infrared image colorization method based on human visual characteristics of the present invention;

[0058] Figure 3 It is a working dynamic principle diagram of the optimal sliding strategy, confrontation strategy and fusion module of the present invention;

[0059] Figure 4 It is a schematic diagram of the specific composition of the edge module of the present invention;

[0060] Figure 5 A generator network diagram for generating adversarial networks of the present invention;

[0061] Figure 6 A schematic diagram of the specific composition of all stacking modules of the present invention;

[0062] Figure 7 This is a schematic diagram of the specific composition of all multi-scale feature dimensionality reduction modules of the present invention;

[0063] Figure 8 This is a schematic diagram of the specific composition of all downsampling enhancement modules of the present invention;

[0064] Fig. 9 This is a schematic diagram of the specific composition of all the expanded attention modules of the present invention;

[0065] Fig.10 This is a schematic diagram of the specific composition of all the composite convolutional attention modules of the present invention;

[0066] Fig.11 A schematic diagram of the specific composition of all the composite attention modules of the present invention;

[0067] Fig.12 This is a schematic diagram of the specific composition of all convolutional attention modules of the present invention;

[0068] Fig.13 It is a schematic diagram of the specific composition of all enhanced feature modules of the present invention;

[0069] Fig.14 A discriminator network diagram for generating an adversarial network for the present invention;

[0070] Fig.15 A schematic diagram showing the effect comparison between the method proposed by the present invention and the prior art;

[0071] Fig.16 It is a structural schematic diagram of an adaptive infrared image colorization system based on human visual characteristics;

[0072] Fig.17 The figure is a schematic diagram of the structure of an adaptive infrared image colorization electronic device based on the visual characteristics of the human eye. DETAILED DESCRIPTION

[0073] The present invention proposes an adaptive infrared image colorization method and system based on human eye visual characteristics, aiming to design an adaptive infrared image colorization method and system based on human eye visual characteristics.

[0074] The adaptive infrared image colorization method based on human visual characteristics proposed by the present invention will be described in the following specific embodiments:

[0075] In the technical solution of this embodiment, Figure 1 As shown, an adaptive infrared image colorization method based on human visual characteristics includes:

[0076] Step 1: Prepare the data set and adapt: ​​obtain the infrared image data set, preprocess the infrared image data set, and output the fixed-size training set image after adaptation; it is divided into training set 1 KAIST data set and training set 2 FLIR data set;

[0077] Step 2: Build the network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator;

[0078] The first adversarial network is composed of a generative adversarial network consisting of generator 1, generator 2 and the discriminator. It continuously obtains more realistic colorization results by completing adversarial learning between the generated colorized image and the true value image. The second adversarial network is composed of an adversarial network consisting of the optimal sliding strategy, adversarial strategy and fusion module in the generative network. It continuously conducts adversarial learning between image 1 and image 2, selects the optimal generator, and completes effective dynamic segmentation and image splicing and fusion operations.

[0079] The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is as follows:

[0080]

[0081] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step length for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. nRepresents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation,

[0082] Follow K n and The quantitative relationship between them gives K n The value of is:

[0083]

[0084] Among them, k is a positive integer, e n The expression can be replaced by:

[0085]

[0086] The adversarial strategy consists of judgment statements, which are used to complete adversarial learning and selection between image one and image two; the segmentation loss is calculated for image region II of input image one and image region II of image two, respectively, and the segmentation loss G1(II) and the segmentation loss G2(II) are obtained respectively, and the sizes of G1(II) and G2(II) are compared, and there are the following five results:

[0087]

[0088] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0089] The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy:

[0090]

[0091] in, represents the output of the fusion module, Indicates concatenation and resizing of three inputs;

[0092] The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x direction and the y direction of the colorized image and the true value image respectively, and then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map;

[0093] Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training;

[0094] Step 4, construct and minimize the composite loss function: The minimized composite loss function includes adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss and sliding loss.

[0095] Furthermore, in step 2, the generator 1 and the generator 2 have the same module structure, including an initial convolution, a multi-scale feature dimension reduction module, a stacking module, an expanded attention module, a skip connection module, a maximum pooling layer, an addition operation, a concatenation operation, and a multiplication operation;

[0096] The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information;

[0097] The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampling information to obtain high-dimensional feature information for output;

[0098] The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features;

[0099] The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module, which gives different weights to different areas of the image;

[0100] The maximum pooling layer reduces the input feature map size to half of the original size, facilitating further feature calculations;

[0101] The jump connection is used to fuse different information of the same size and dimension;

[0102] The splicing operation is used to perform channel fusion of different information of the same size but different dimensions to enhance the nonlinearity of features;

[0103] The addition operation is used to add and fuse different information of the same size and dimension;

[0104] The multiplication operation is used to multiply and fuse different information of the same size and dimension.

[0105] Furthermore, the composite convolutional attention module is composed of a composite attention module, a convolutional feature extraction module and an enhanced feature module;

[0106] The composite attention module consists of a fully connected layer, a maximum pooling, an average pooling, a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, an S-type function, a splicing operation and an addition operation; wherein the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and finds the average value of each channel, the channel maximum pooling takes the maximum value of the number of different channels for each pixel and outputs it, the channel average pooling takes the average value of the number of different channels for each pixel and outputs it, the convolutional kernel size used in the convolutional layer 1 and the convolutional layer 2 is 1×1, the step size is 1, the convolutional kernel size used in the convolutional layer 3 is 7×7, the step size is 3,

[0107] The convolution feature extraction module is composed of average pooling, convolution layer, S-type function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image to find the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1;

[0108] The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection, wherein the convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1, and the convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1.

[0109] The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. The convolutional blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a step size of 2. The convolutional block in convolutional layer 5 uses a convolution kernel size of 4×4 and a step size of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

[0110] Furthermore, the adaptive infrared image colorization method based on human visual characteristics also includes:

[0111] Step 5, fine-tuning the model: input the long-wave infrared image and the true value image into the network model for training respectively, compare the obtained results with the image evaluation index, and fine-tune the network model parameters;

[0112] Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colorized image.

[0113] Embodiment 1:

[0114] An embodiment of the present invention proposes an adaptive infrared image colorization method based on human visual characteristics, such as Figure 1 As shown, the following steps are included:

[0115] Step 1, prepare the data set and adapt: ​​KAIST data set and FLIR data set are composed of continuous frames of different videos. The adjacent images are not much different. Therefore, the two data sets are cleaned to a certain extent, and the cleaned images are outputted as images resized to 320×256 using the adaptive method as the input of the entire network. Finally, KAIST data set obtained 4755 images as training data set 1, and FLIR data set obtained 3918 images as training data set 2;

[0116] Step 2: Build a network model. The working principle diagram of the network model is as follows: Figure 2 As shown, it specifically includes the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator;

[0117] like Figure 3 As shown in Figure ①, the nth sliding process is shown. The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. The image cut out by straight line l1 is divided into image I and image II, and the image cut out by straight line l2 is divided into image I and image III. The intersection between image I and image II is defined as image II. In the optimal sliding strategy, straight lines l1 and l2 are dynamically learned to move, and their moving distance l s and direction are controlled by the adversarial strategy; in order to prevent the straight lines l1 and l2 from exceeding the edge of the image, the size of the straight lines l1 and l2 is restricted, and the range of the two obtained values ​​of the straight line l1 is from 0 to the pixel length l of the image. x , the value range of line l2 is from 0 to the pixel length l of the image x In order to ensure that part I of image 1 and part II of image 2 have the same size and image information, the pixel difference between the fixed lines l1 and l2 is a. The value of constant a is set by the program. The default specific quantitative relationship is: Where a is the fixed pixel difference between lines l1 and l2, l x is the pixel size of the input image, To round down, is the pixel length of line l1 relative to the lower left corner of the image, is the pixel length of line l2 relative to the lower left corner of the image;

[0118] Since the last cropped image 1 and image 2 need to be input to generator 1 and generator 2 respectively to generate the colorization task, it is necessary to transform the size of the cropped image 1 and image 2 to the input image size of the generator, and the input image size of the generator is the same as the output image size; the present invention focuses on the moving step length l s The relationship between the size of the image output by the generator after mapping to the transformation is considered to be the core loss of the network of the present invention, so the accurate calculation and optimal selection of the mapping distance are necessary; for the step length l s The mapping distance formula is as follows:

[0119]

[0120] in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step length for the nth calculation The mapping distance on the generator input image size, K n represents the optimal mapping distance at the nth calculation, where the value is an integer, e n It represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation. Therefore, in order to obtain a more accurate mapping distance, the mapping loss e is required. n Should be small enough; focus on K n and The quantitative relationship between them gives K n The value of is:

[0121]

[0122] Among them, the value of k is a positive integer, so e n The expression can be replaced by:

[0123]

[0124] At this time n Transformed into a binary equation, the function graph first rises and then falls, so there is an optimal point. By running the program, it can be concluded that under different There is a step size under the input The optimal solution of and The distance between is fixed, so the straight line The optimal solution for the moving step size is

[0125] like Figure 3 As shown in Figure ③, the adversarial strategy consists of judgment statements to complete the adversarial learning and selection between image one and image two; the adversarial strategy selects the image information generated by the best generator by judging the fixed image loss of the intersection of image one and image two. The specific working principle is as follows: the segmentation loss is calculated for the image region II of the input image one and the image region II of the image two, and the corresponding segmentation loss G1(II) and segmentation loss G2(II) are obtained respectively, and the sizes of G1(II) and G2(II) are compared. The following five structures appear:

[0126]

[0127] in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two;

[0128] like Figure 3 As shown in Figure ②, the fusion module mainly completes the splicing of the segmented images through the splicing operation, using the PIL function in the Python language library. Different output results are obtained for different situations of the adversarial strategy:

[0129]

[0130] in, represents the output of the fusion module, Indicates concatenation and resizing of three inputs;

[0131] like Figure 4 As shown in the figure, the edge module mainly uses the Sobel edge detection algorithm to detect and display the edge information of the image in different directions; the edge information of the colorized image and the true value image is collected in the x direction and the y direction respectively, and then the feature fusion is performed through the addition operation to obtain the edge information map, and then the contour loss calculation is performed on the edge information map;

[0132] like Figure 5As shown in Figure 1, the generator consists of an initial convolution, a stacking module, a multi-scale feature dimensionality reduction module 1, a maximum pooling layer 1, an expanded attention module 1, a multi-scale feature dimensionality reduction module 2, a maximum pooling layer 2, an expanded attention module 2, a multi-scale feature dimensionality reduction module 3, a maximum pooling layer 3, an expanded attention module 3, a multi-scale feature dimensionality reduction module 4, a maximum pooling layer 4, an expanded attention module 4, a jump connection, a multiplication operation, a splicing operation, and an addition operation. The initial convolution maps the input image channel to a high dimension to extract high-dimensional detail information. The convolution kernel size is 1×1, the step size is 1, and the number of image channels is increased in dimension. ; The stacking module consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11, convolution layer 12, convolution layer 13, splicing operation and addition operation. The convolution kernel size used in convolution layer 1, convolution layer 2 and convolution layer 13 is 1×1, and the step size is 1. The convolution kernel size used in convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11 and convolution layer 12 is 3×3, and the step size is 1. The composition network of the stacking module is as follows: Figure 6 As shown in the figure, the multi-scale feature dimensionality reduction module downsamples the input image in different ways, obtains the target image from multiple angles, and thus obtains a more accurate downsampling result. It consists of convolution layer 1, convolution layer 2, maximum pooling, downsampling enhancement module, and splicing operation. The specific composition of each multi-scale feature dimensionality reduction module is shown in the figure. Figure 7As shown in the figure, convolution layer 1 downsamples the image to extract abstract information. The convolution kernel size is 3×3 and the step size is 2. Convolution layer 2 convolutionally fuses the convolution layer 1 and the maximum pooling result, and the output image size remains unchanged. The convolution kernel size is 3×3 and the step size is 1. The size of the pooling kernel of the maximum pooling is 2×2. The multi-scale feature dimensionality reduction module consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10, convolution layer 11, convolution layer 12, convolution layer 13, convolution layer 14, convolution layer 15, convolution layer 16, convolution layer 17, convolution layer 18, normalization operation, L-type function, R-type function, splicing operation, multiplication operation and addition operation. The convolution kernel size of convolution layer 1 and convolution layer 18 is 1×1 and the step size is 1. Convolution layer 2, convolution layer 3, The convolution kernel size of convolution layer 4, convolution layer 5, convolution layer 6, convolution layer 7, convolution layer 8, convolution layer 9, convolution layer 10 and convolution layer 14 is 3×3, and the step size is 1. Convolution layer 11, convolution layer 12, convolution layer 13, convolution layer 14, convolution layer 15, convolution layer 16 and convolution layer 17 are hollow convolutions with a convolution kernel size of 3×3, a step size of 1, an image padding of 2, and a hollow rate of 2. The maximum pooling layer converts low-dimensional information into high-dimensional information for high-dimensional information extraction and splicing, so as to obtain more comprehensive feature information. Among them, the maximum pooling layer 1, maximum pooling layer 4 and maximum pooling layer 6 are maximum pooling layers with a pooling kernel of 2×2, the maximum pooling layer 2 and the maximum pooling layer 5 are maximum pooling layers with a pooling kernel of 4×4, and the maximum pooling layer 3 is a maximum pooling layer with a pooling kernel of 8×8. The network composition of the multi-scale feature dimensionality reduction module is as follows: Figure 8 As shown in Figure 2, the dilated attention module consists of a convolutional layer block, normalization, an L-type function, and an attention module. The dilated attention module network consists of Fig. 9 As shown, the convolution layer block uses deconvolution, the deconvolution kernel size is 4×4, and the step size is 2.

[0133] The composite convolutional attention module consists of a composite attention module, a convolutional feature extraction module, and an enhanced feature module. The network composition of the composite convolutional attention module is as follows: Fig.10 As shown in Figure 1, the composite attention module consists of a fully connected layer, maximum pooling, average pooling, convolutional layer 1, convolutional layer 2, convolutional layer 3, S-type function, concatenation operation and addition operation. The fully connected layer compresses the input channel to 8 times the original, and then restores it to the number of input channels. The maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2. The average pooling calculates the input image and finds the average value of each channel. The channel maximum pooling takes the maximum value of the number of different channels for each pixel and outputs it. The channel average pooling takes the average value of the number of different channels for each pixel and outputs it. The convolution kernel size used in convolutional layer 1 and convolutional layer 2 is 1×1, with a step size of 1. The convolution kernel size used in convolutional layer 3 is 7×7, with a step size of 3. The network composition of the composite attention module is shown in Figure 1. Fig.11 As shown in Figure 1, the convolution feature extraction module consists of average pooling, convolution layer, S-type function, multiplication operation, addition operation, jump connection and splicing operation. Average pooling calculates the input image to find the average value of each channel. The convolution layer uses a convolution kernel size of 1×1 and a step size of 1. The network composition of the convolution feature extraction module is as follows: Fig.12 As shown in Figure 1, the enhanced feature module consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, addition operation and skip connection. The convolutional kernel size used in convolutional layer 1 is 3×3 and the step size is 1. The convolutional kernel size used in convolutional layer 2 and convolutional layer 3 is 1×1 and the step size is 1. The network composition of the enhanced feature module is as follows: Fig.13 As shown in ; the discriminator consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, convolution layer 6, normalization, L function and composite convolution attention module, where the convolution blocks in convolution layer 1, convolution layer 2, convolution layer 3 and convolution layer 4 use a convolution kernel size of 4×4 and a step size of 2, and the convolution block in convolution layer 5 uses a convolution kernel size of 4×4 and a step size of 1. The composite convolution attention module is the same as the composite convolution attention module network in the generator. The composition network of the discriminator is shown in Fig.14 As shown;

[0134] In general, the infrared image colorization process is to determine the input infrared data set, screen and adapt the data set, and divide the input image into image 1 and image 2 through the optimal sliding strategy, and divide image 1 into image I and part II of image 1, and input them into generator 1 together, and divide image 2 into part II of image 2 and image III, and input them into generator 2 together. The segmentation loss of the intersecting area of ​​image 1 and image 2, that is, part II of image 1 and part II of image 2, is compared through the adversarial strategy, and the optimal generator result is selected. The optimal sliding strategy is used to obtain the optimal sliding step size to guide the next round of dynamic segmentation straight lines, and the generated results of generator 1 and generator 2 are input into the fusion module for fusion and splicing output as the final colorized image; the infrared image is used as a reference and the colorized image and the true value image are used as the discriminator input respectively, the difference between the generated colorized image and the true value image is identified, and the difference is quantified in the range of 0-1 to complete the evaluation of the colorization degree of the image;

[0135] In order to ensure the robustness of the network and retain more network information, the present invention uses four activation functions, namely L-type function, R-type function, G-type function and S-type function. The last layer of the generator uses the L-type function, the last layer of the discriminator uses the S-type function, the multi-scale feature dimensionality reduction module uses the R-type function, and the expanded attention module uses the L-type function and the S-type function. At the same time, all activation functions in the generator and the discriminator in the present invention are L-type functions; the L-type function, R-type function, G-type function and S-type function are defined as follows

[0136]

[0137] In order to enhance the generalization ability of the network and prevent data overfitting, normalization operations are generally used in this invention. In this invention, all normalization operations are performed using batch normalization. Normalization is used before various activation functions to make the output of the previous layer distributed with a mean of 0 and a variance of 1, that is, to normalize the input of the next layer, so that it can have a certain gradient when passing through the activation function, thereby avoiding the value being too large and entering the saturation area.

[0138] Step 3, training the network model: input the training data set 1 and the training data set 2 in the data set obtained by S1 into the network model constructed by S2 for training;

[0139] Step 4: Get the minimized loss function value and the optimal evaluation index: By constructing a reasonable loss function, the model can be constrained to optimize in the direction of the designer's attention. Therefore, the smaller the loss function, the better the robustness of the model. Selecting appropriate evaluation indicators can effectively reflect the quality of the model and the degree of image distortion, and measure the role of the colorization network.

[0140] In step 4, the network output and label loss function are calculated to achieve better fusion effect by minimizing the loss function; in the training process, the loss function uses adversarial loss Pixel loss Depth Loss Perceptual Similarity Loss Contour loss Segmentation loss Sliding loss and the total variational loss Combine them optimally with certain weights to obtain the minimized total modulus loss function;

[0141] In order to encourage the network to output color results with more realistic details, an adversarial loss is used; the adversarial loss is used to learn the implicit relationship between the thermal infrared image and the colorized image, which is defined as:

[0142]

[0143] Where x is the input infrared image, y is the input true value image, G(·) is the output of the entire generation network, and D(·,·) are the two images input into the discriminator respectively;

[0144] In order to maintain the similarity of brightness and contrast between the colorized image generated by the generator and the real image, a pixel loss function is designed to standardize the generation of colorized images; by designing the L1 norm to standardize the difference between pixels, a better colorization effect can be obtained. The pixel loss is defined as:

[0145]

[0146] Where A represents image 1, B represents image 2, G1(·) represents the output of generator 1, G2(·) represents the output of generator 2, and W A is the width of the input image, H A is the height of the input image, W B is the width of input image 2, H B is the height of input image 2, S(y) represents the true image of the corresponding area of ​​the true image, and ‖·‖1 represents the L1 norm of the given output;

[0147] The depth loss can effectively minimize the brightness and contrast differences between the color image and the colorized image; if the adversarial network focuses too much on the depth loss, the brightness and contrast in the thermal infrared image will be lost; in order to prevent the generator from over-representing the pixel-to-pixel relationship, an appropriate weight of the depth loss will be added; the depth loss can be expressed as:

[0148]

[0149] Among them, φ k (·) represents the feature representation of the kth max pooling layer in the VGG-16 network, C A_k H A_k W A_k represents the size of the image-feature representation, C B_k H B_k W B_k represents the size of the second feature representation of the image, ‖·‖1 represents the L1 norm; where A and The quantitative relationship between them can be expressed as: Among them, B and The quantitative relationship between them can be expressed as:

[0150] In order to compensate for the differences in brightness, contrast and network of images, the present invention adopts perceptual similarity loss to penalize the generated images so that they can reach the optimal value faster; the perceptual similarity loss is defined as follows:

[0151]

[0152] Among them, ssim(·,·) indicates the use of the perceptual similarity module function integrated in pytorch for calculation;

[0153] In order to highlight the contour features of the image, the present invention intends to use contour loss to reduce the phenomenon of blurred boundaries of contour information in image translation tasks, thereby increasing the pre-effect of colorization tasks on other image tasks; the contour loss is defined as follows:

[0154]

[0155] Where W is the width of the input image, H is the height of the input image, represents the output of the image passing through the edge module, ‖·‖1 represents taking the L1 norm of the given output, is the output of the fusion module, and the transformation relationship is: in Represents the output of the fusion module relative to image one and image two;

[0156] In order to limit the generation of the segmented image to have the same color and texture features as the true image, the colorization result of the segmented image and the corresponding position of the true image are taken to obtain the segmentation loss, which can convert the more realistic colorization result into a low level of segmentation loss; since the segmentation results of the straight lines l1 and l2 divide the image into four parts, but the image I divided by the straight line l1 and the image III divided by the straight line l2 do not have much meaning, the segmentation loss body calculates the segmentation loss of image one and image two. Since image one can be broadly divided into part I of image one and part II of image one, the segmentation losses are defined as and Since image 2 can be broadly divided into part II of image 2 and part III of image 2, the segmentation losses are defined as and and the total segmentation loss Defined as:

[0157]

[0158]

[0159] Where W is the width of the input image, H is the height of the input image, S.(y) represents the true value image of the corresponding area of ​​the true value image, and λ A_I_1 , A_I_2 , A_I_3 , A_II_1 , A_II_2 , A_II_3 , B_II_1 , B_II_2 , B_II_3 , B_III_1 , B_III_2 and λ B_III_3are constants, with a default value of 1, sl_1 , sl_2 , sl_3 and λ sl_4 They are all constants, and their default values ​​are 1, 0.5, 1, and 0.5 respectively;

[0160] In order to utilize the characteristic information and promote the interpretation and translation of the color information of the real image, the present invention proposes a sliding loss, which generates a more colorful colorized image by increasing the weight of the sliding loss in the total loss; the loss of the sliding loss is defined as:

[0161]

[0162] Among them, l s Indicates the step length of linear movement, l w Indicates that the generator needs to input the image size length, l h Indicates that the generator needs to input the image size width;

[0163] In order to achieve better model performance, the above eight losses are combined to design the total loss of the model, constrain the model as a whole, and the total loss function is designed as

[0164]

[0165]

[0166] Among them, λ adv , pixel , df , ssim , tv , sl , el and λ step They represent the weights controlling the complete loss function. The weights are set based on preliminary experiments on the training data set. The final weights are set to 0.03, 1, 1, 1.25, 1.25, 1, 1, and 5 respectively.

[0167] In step 4, the appropriate evaluation indicators of the network are peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and perceptual image similarity (LPIPS); the peak signal-to-noise ratio is a basic indicator in the objective evaluation standard of image quality, and the size of this value directly reflects the pixel-level mean square error between the generated image and the original image. The larger the PSNR value, the smaller the image distortion, the better the quality, and the more reasonable the color distribution; structural similarity (SSIM): perceptual similarity is an objective evaluation indicator that measures image similarity from three aspects: brightness, contrast, and network. It reflects the subjective quality of the image. The higher the value, the higher the image restoration and the better the visual effect; perceptual image similarity (LPIPS): the LPIPS indicator aims to quantify the perceptual similarity between colorized images and color images; it is related to human visual perception. By training on manually annotated data sets, feature extraction and distance measurement that can better reflect human visual perception can be learned, and the subjective feeling of human beings on the degree of retention of image perception information can be better reflected. The lower the LPIPS value, the closer the quality of the generated image is to the true value image, and the better the effect; the peak signal-to-noise ratio, structural similarity and perceptual image similarity are defined as follows:

[0168]

[0169] Among them, (2 n -1) 2 It represents the square of the maximum value of the signal, the number of bits of each sample value, and the maximum value of 8-bit representation is 255; MSE represents the mean of the square of the difference between the corresponding pixels of the two images, and the expression is as follows: μ x and μ y Represent the mean and variance of images x and y respectively, and Represents the standard deviation of images x and y, σ xy represents the covariance of images x and y, C1 and C2 are constants; d is the distance between x and x0, and w1 is a trainable weight parameter;

[0170] The number of training times is set to 200, the learning rate of the first 100 training processes is set to 0.0001, and the learning rate of the next 100 training processes is gradually reduced from 0.0001 to 0; the number of input images is fixed to 1 image each time; the Adam optimizer is selected as the network parameter optimizer; its advantages are simple implementation, efficient calculation, low memory requirement, and parameter update is not affected by the scaling transformation of the gradient, making the parameters relatively stable; when the discriminator's ability to judge whether the generated colorized image is a real color image is balanced with the generator's ability to generate colorized images, the network is considered to have been basically trained, and the discriminator output should be close to 0.5 at this time;

[0171] Step 5, fine-tune the model: Use test data set 1 and test data set 2 to test and fine-tune the trained network; use test data set 1 and test data set 2 as input, calculate the evaluation index value in step 4, and then compare it with the expected evaluation index value, and then fine-tune the model until the expected evaluation index value is reached;

[0172] Step 6, save the model: encapsulate the model that reaches the expected evaluation index and save all parameter information in the network; the network can be applied to infrared image input of any size and complete the task of converting the input infrared image into the generated colorized image;

[0173] Among them, the implementation of convolution, activation function, splicing operation, addition operation, multiplication operation and normalization are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature.

[0174] The present invention constructs an adaptive infrared image colorization method based on human visual characteristics, which can realize the conversion of an input infrared image into a corresponding colorized image without going through other intermediate steps, thus avoiding the manual design of relevant colorization rules or other additional annotations; under the same experimental conditions, the feasibility and superiority of the method are further verified by calculating the relevant evaluation indexes of the colorized image obtained by the existing method; the relevant indexes of the existing technology and the method proposed by the present invention are compared. Fig.15 As shown;

[0175] Existing method 1: A generative adversarial network for infrared image colorization, consisting of a generator and a discriminator. The generator uses three downsampling modules, nine residual modules and three upsampling modules; the discriminator uses six convolutional layers and a sigmoid function to distinguish the generated image from the real image. The method is trained and tested using the training set and test set in step 1 of the present invention, respectively.

[0176] Existing method 2: A generative adversarial network for infrared image colorization, consisting of a generator and a discriminator. The generator first uses a 3×3 maximum pooling to reduce the size of the input image, then passes through four downsampling modules, nine residual modules and four upsampling modules, and finally outputs through a S-type function; the discriminator uses six convolutional layers and a S-type function to distinguish the generated image from the real image. The method is trained and tested using the training set and test set in step 1 of the present invention, respectively.

[0177] Existing method 3: A generative adversarial network for infrared image colorization, consisting of a generator and a discriminator. The generator uses four downsampling modules and transmits features through residual connections, and then outputs the model through four attention upsampling modules, and finally sums and averages the residual results to obtain the final output; the discriminator uses six convolutional layers and R-type functions to distinguish the generated image from the real image. The method is trained and tested using the training set and test set in step 1 of the present invention, respectively.

[0178] Existing method 4: A generative adversarial network for infrared image colorization, consisting of a generator and a discriminator. The generator uses four downsampling attention modules and a residual module after each downsampling attention module for feature extraction, and then uses four upsampling attention modules for feature expansion to finally obtain the model output; the discriminator uses six convolutional layers and an R-type function to distinguish the generated image from the real image. The method is trained and tested using the training set and test set in step 1 of the present invention, respectively.

[0179] Fig.15 The colorization effect of this method and the existing four algorithms on the KAIST dataset and FLIR dataset is evaluated; Fig.15 It can be seen that the method proposed in the present invention has a higher peak signal-to-noise ratio, higher structural similarity and lower perceived image similarity than the existing method. These indicators also further illustrate that the method proposed in the present invention has better colorization quality.

[0180] Embodiment 2:

[0181] An adaptive infrared image colorization system based on human visual characteristics, such as Fig.16 As shown, the system mainly includes an image acquisition module, an image processing module, a model training module, a feature visualization module, a quality assessment module, a quality improvement module and a hardware deployment module;

[0182] An image acquisition module, used for acquiring an image to be colored;

[0183] An image processing module is used to adaptively process the acquired colorized image and divide it into a training set and a test set;

[0184] The model training module is used to input the processed training set into the designed network for training, obtain the best generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image;

[0185] The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process, and compare the final generated image with the input infrared image and the reference color image to further visually observe the image colorization effect;

[0186] The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is used as the final colorization result; if the preset quality requirements are not met, the quality improvement module is started;

[0187] The quality improvement module is used to reacquire the data set and continue to train the model with the new data set, repeating the quality assessment module until an image that meets the preset quality requirements is generated;

[0188] The hardware deployment module is used to deploy the model that meets the quality assessment module on hardware. Deploying the model on suitable hardware can make it easier for users to complete the infrared image colorization process.

[0189] Furthermore, the image to be colored in the image acquisition module is an infrared image, or a grayscale image, but a true value image is required;

[0190] Furthermore, the adaptive processing in the image processing module includes image cropping and image flipping, and finally the image ratio of the training set and the test set is 10:1;

[0191] Furthermore, the model training module fixes each image from different size inputs to a size of 320×256; a total of 200 rounds of training are performed with a batch size of 4; a learning rate training strategy is used, in the first 100 rounds, the learning rate size is set to 0.002, and in the next 100 rounds, the learning rate is set to gradually decrease from 0.002 to 0; the number of filters in the generator and the discriminator is set to 64; the present invention uses the Adam optimizer;

[0192] Furthermore, the visualization image in the training process of the feature visualization module is the attention heat map in the composite convolution attention module. By observing the attention heat map, it is possible to clearly understand whether the focus area of ​​the attention module is the target area; after the visualization image at the end of training is saved after the training is completed, the model is predicted, and the predicted image is compared with the input infrared image and the true value image for output, so that the colorization effect can be intuitively felt;

[0193] Furthermore, the evaluation indicators of the quality assessment module are peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and perceived image similarity (LPIPS); the higher the value of the peak signal-to-noise ratio, the better; the higher the value of the structural similarity, the better; the higher the value of the perceived image similarity, the better;

[0194] Furthermore, the hardware deployment of the hardware deployment module needs to be deployed on suitable hardware that supports the algorithm model. The hardware should at least include memory, processor and communication interface. The input model can be stored in the memory to read the model at any time. The processor is used to load the model and perform calculations. The communication interface is used to accept input training data sets and test data sets, and output the output results for display or transmission to other terminal devices.

[0195] Embodiment 3:

[0196] An adaptive infrared image colorization electronic device based on human visual characteristics, such as Fig.17 As shown, it mainly includes a memory, a communication interface and a bus; wherein the memory, the processor and the communication interface are used to execute each step of the adaptive infrared image colorization method based on human eye visual characteristics of embodiment 1 of the present invention;

[0197] The memory can be a ROM, a static storage device, a dynamic storage device or a RAM; the memory can store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to execute the various steps of the training method of the infrared image colorization network of the embodiment of the present invention;

[0198] The processor can be a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute relevant programs to implement the functions that the units in the infrared image colorization training system of the present invention need to perform, or to perform the infrared image colorization training method of the present invention;

[0199] The processor can also be an integrated circuit chip with signal processing capability; in the implementation process, each step of the infrared image colorization training method of the present invention can be completed by hardware integrated logic circuits or software instructions in the processor; the above-mentioned processor can also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; can implement or execute the infrared image colorization method and system, steps and logic block diagram of the present invention; the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.;

[0200] The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or communication networks; for example, it is possible to obtain an image to be processed or an initial feature map of the image to be processed through the communication interface;

[0201] A bus may include a pathway that transfers information between various components of the system (eg, memory, processor, communication interface).

[0202] Embodiment 4:

[0203] A computer-readable storage medium for adaptive infrared image colorization based on human visual characteristics, the computer-readable storage medium can be a computer-readable storage medium that exists independently and is not assembled into a device; the computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method provided in embodiment 1;

[0204] It should be noted that although Fig.17 The simplified diagram of the electronic device system shown only shows the memory, processor and communication interface, but in the specific implementation process, those skilled in the art should understand that the system also includes other devices necessary for normal operation; at the same time, according to specific needs, those skilled in the art should understand that the system may also include hardware devices for implementing other additional functions; in addition, those skilled in the art should understand that the system may also include only the devices necessary for implementing the embodiments of the present invention, and does not necessarily include Fig.17 All devices shown in .

[0205] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An adaptive infrared image colorization method based on human visual characteristics, characterized in that: include: Step 1: Prepare the data set and adapt: ​​obtain the infrared image data set, preprocess the infrared image data set, and output the fixed-size training set image through adaptation; It is divided into training set 1 KAIST dataset and training set 2 FLIR dataset; Step 2: Build the network model: The network model consists of two adversarial networks, including the optimal sliding strategy, adversarial strategy, fusion module, generator 1, generator 2, contrast network, edge module and discriminator; the first adversarial network consists of generator 1, generator 2 and discriminator; the second adversarial network consists of the optimal sliding strategy, adversarial strategy and fusion module; The optimal sliding strategy divides the entire input infrared image into three parts through straight lines l1 and l2. Straight lines l1 and l2 are dynamically learned and moved in the optimal sliding strategy. s The mapping distance formula is: in, Indicates the step size of the program's nth calculation, Indicates the length of l1 during the nth calculation of the program, l w Indicates that the generator needs to input the image size, which is a constant and is set to 256. n Indicates the step length for the nth calculation The mapping distance on the generator input image size, K n Indicates the optimal mapping distance for the nth calculation, which is an integer. n Represents the mapping loss between the optimal mapping distance and the mapping distance relative to the nth calculation, Follow K n and The quantitative relationship between them gives K n The value of is: Among them, k is a positive integer, e n Replace the expression with: The adversarial strategy consists of judgment statements, which are used to complete adversarial learning and selection between image one and image two; the segmentation loss is calculated for image region II of input image one and image region II of image two, respectively, and the segmentation loss G1(II) and the segmentation loss G2(II) are obtained respectively, and the sizes of G1(II) and G2(II) are compared, and there are the following five results: in, represents the output of the adversarial strategy, represents the segmentation loss of image region II of image I, represents the segmentation loss of image region II of image two; The fusion module uses the PIL function in the Python language library to complete the splicing of the segmented images. Different output results are obtained for different situations of the adversarial strategy: in, represents the output of the fusion module, Indicates concatenation and resizing of three inputs; The edge module uses the Sobel edge detection algorithm to detect and display edge information in different directions of the image; collects edge information in the x direction and the y direction of the colorized image and the true value image respectively, and then performs feature fusion through addition operation to obtain an edge information map, and performs contour loss calculation on the edge information map; Step 3, training the network model: input the infrared image dataset in step 1 into the network model in step 2 for training; Step 4, construct and minimize the composite loss function: The minimized composite loss function includes adversarial loss, pixel loss, depth loss, contour loss, perceptual similarity loss, segmentation loss, total variation loss and sliding loss.

2. The adaptive infrared image colorization method based on human visual characteristics according to claim 1 is characterized in that: In step 2, the generator 1 and the generator 2 have the same module structure, including an initial convolution, a multi-scale feature dimensionality reduction module, a stacking module, an expanded attention module, a skip connection module, a maximum pooling layer, an addition operation, a splicing operation, and a multiplication operation; The initial convolution consists of a 1×1 convolution, which is used to map the input low-dimensional information to high-dimensional information; The multi-scale feature dimensionality reduction module splices and fuses the multi-scale down-sampling information to obtain high-dimensional feature information for output; The stacking module achieves the receptive field of large kernel convolution by stacking multiple 3×3 convolutions to extract image detail features; The dilated attention module consists of a convolutional layer block, normalization, an L-type function, and a composite convolutional attention module, which gives different weights to different areas of the image; The maximum pooling layer reduces the input feature map size to half of the original size, facilitating further feature calculations; The jump connection is used to fuse different information of the same size and dimension; The splicing operation is used to perform channel fusion of different information of the same size but different dimensions to enhance the nonlinearity of features; The addition operation is used to add and fuse different information of the same size and dimension; The multiplication operation is used to multiply and fuse different information of the same size and dimension.

3. The adaptive infrared image colorization method based on human visual characteristics according to claim 2 is characterized in that: The composite convolutional attention module consists of a composite attention module, a convolutional feature extraction module and an enhanced feature module; The composite attention module consists of a fully connected layer, a maximum pooling, an average pooling, a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, an S-type function, a splicing operation and an addition operation; wherein the fully connected layer compresses the input channel to 8 times of the original, and then restores it to the number of input channels, the maximum pooling uses a maximum pooling layer with a pooling kernel of 2×2, the average pooling calculates the input image and finds the average value of each channel, the channel maximum pooling takes the maximum value of the number of different channels for each pixel and outputs it, the channel average pooling takes the average value of the number of different channels for each pixel and outputs it, the convolutional kernel size used in the convolutional layer 1 and the convolutional layer 2 is 1×1, the step size is 1, the convolutional kernel size used in the convolutional layer 3 is 7×7, the step size is 3, The convolution feature extraction module is composed of average pooling, convolution layer, S-type function, multiplication operation, addition operation, skip connection and splicing operation, wherein the average pooling calculates the input image to find the average value of each channel, and the convolution layer uses a convolution kernel size of 1×1 and a step size of 1; The enhanced feature module consists of convolution layer 1, convolution layer 2, convolution layer 3, addition operation and skip connection, wherein the convolution kernel size used in convolution layer 1 is 3×3 and the step size is 1, and the convolution kernel size used in convolution layer 2 and convolution layer 3 is 1×1 and the step size is 1. The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, convolutional layer 6, normalization, L-function and compound convolutional attention module. The convolutional blocks in convolutional layer 1, convolutional layer 2, convolutional layer 3 and convolutional layer 4 use a convolution kernel size of 4×4 and a step size of 2. The convolutional block in convolutional layer 5 uses a convolution kernel size of 4×4 and a step size of 1. The compound convolutional attention module has the same network as the compound convolutional attention module in the generator.

4. The adaptive infrared image colorization method based on human visual characteristics according to claim 1, characterized in that: Also includes: Step 5, fine-tuning the model: input the long-wave infrared image and the true value image into the network model for training respectively, compare the obtained results with the image evaluation index, and fine-tune the network model parameters; Step 6, saving the model: The finalized network model parameters are solidified and output as a model file. During testing and network model hardware deployment, the network model is directly called for colorized output to obtain a colorized image.

5. An adaptive infrared image colorization system based on human visual features, used to execute the adaptive infrared image colorization method based on human visual features as claimed in any one of claims 1 to 4, characterized in that: The system comprises: An image acquisition module, used for acquiring an image to be colored; An image processing module is used to adaptively process the acquired colorized image and divide it into a training set and a test set; The model training module is used to input the processed training set into the designed network for training, obtain the best generation network parameters by constructing the optimal loss function, save the parameters, and use the test set to perform model colorization verification to generate a colorized image; The feature visualization module is used to visualize the intermediate feature maps generated during the model generation process and compare the final generated image with the input infrared image and the reference color image. The quality assessment module is used to assess whether the quality of the colorized image finally generated meets the preset quality requirements; if the preset quality requirements are met, the generated colorized image is taken as the final colorization result; if the preset quality requirements are not met, the quality improvement module is started; The quality improvement module is used to reacquire the data set and continue to train the model with the new data set, repeating the quality assessment module until an image that meets the preset quality requirements is generated; The hardware deployment module is used to deploy the model that meets the quality assessment module on the hardware.

Images