Image fusion network and image fusion method based on semantic perception

Abstract

The invention provides an image fusion network and an image fusion method based on semantic perception, and belongs to the technical field of image processing and image analysis. On the basis of a deep learning framework, semantic perception is introduced into image fusion, a network structure is established by using a neural network, and an infrared and visible light image fusion network model and a training process are provided. The problems that information contained in the input infrared and visible light images is not comprehensive and semantic information is lost in the infrared and visible light image fusion process are solved. According to the method, image fusion and semantic perception are combined, combined loss composed of semantic loss and content loss is introduced as guidance, effective information of the input infrared and visible light images is fully combined, and the generated image is prominent in detail, comprehensive in information and high in image quality; the method has good application potential and value in the fields of image processing, video monitoring, medical images, vehicle guidance and the like.

Description

Technical Field

[0001] The present invention relates to the technical fields of image fusion, semantic segmentation and artificial intelligence image processing, and in particular to an image fusion network and an image fusion method based on semantic perception. Background Art

[0002] With the widespread application of various acquisition systems, the importance of image fusion technology in image processing, machine vision and other fields has become increasingly prominent. Image fusion technology generates a high-quality image by combining information from different time points, different perspectives and multiple sensors. This fusion not only significantly improves the clarity and details of the image, but also effectively retains the key features in the image, ensuring the integrity and accuracy of the information. This method can provide a more comprehensive and realistic visual experience, which can significantly improve the practical value of the image in application scenarios such as security monitoring, medical imaging or remote sensing monitoring. In the field of image fusion, the fusion of infrared and visible light images is the most widely used fusion technology. Infrared images are effective in detecting the heat distribution of objects and environments under low light or dark conditions. The recorded information is relatively comprehensive, but it ignores texture and is easily affected by noise; visible light images can provide rich color and detail information, but have specific requirements for lighting conditions and occlusion environments. This complementary role can be used in the field of image fusion. Fully combining the advantages of the two images can significantly improve the integrity and quality of the image content. It has important application potential and value in the fields of machine vision and information, and has become a current research hotspot.

[0003] In the development of image fusion technology, related research and algorithms mainly focus on improving the visual quality of images and the fusion of basic information. Existing fusion technologies and methods, such as those based on wavelet transform, multi-resolution analysis, and principal component analysis, usually focus on how to optimize the fusion effect of images in the spatial or frequency domain. They have achieved certain success in enhancing image details, contrast, and overall visual effects, but there are significant deficiencies in improving the semantic information of images, especially when processing infrared and visible light images. The comprehensive improvement of semantic information within them is ignored, and how to retain and enhance the semantic information of images during the fusion process is not fully considered, which seriously hinders its practical application in object recognition, scene understanding, and semantic segmentation. How to solve the problem of missing semantic information in the process of infrared and visible light image fusion and generate fused images with prominent details and rich semantic information is still a very challenging task. Current related research is trying to use advanced deep learning technology and artificial intelligence methods to improve the performance of image fusion technology at the semantic information level, so as to effectively extract and fuse the semantic information of multimodal images.

[0004] In order to solve the problem of missing semantic information in the process of infrared and visible light image fusion and improve the integrity of the fused image in terms of content and the quality of fusion, the present invention introduces semantic perception into the process of infrared and visible light image fusion based on a deep learning framework, solves the problem of missing semantic information in the image fusion process and improves the quality of the fused image. At the same time, by fully fusing the complementary information of infrared and visible light images, semantic loss is introduced into the image fusion network to further optimize the fusion effect, especially the semantic information contained in the image. Compared with the traditional infrared and visible light image fusion method, the present invention can obtain a fused image with prominent details and more semantic information, with high image quality, complete visually perceived image semantic content and ideal effect. The present invention provides an image fusion network and method by embedding the semantic information of the image, using convolution to fully obtain the feature information of infrared and visible light images and deeply mining the prior knowledge in the image enhancement process, thereby obtaining a relatively ideal image fusion target. The image fusion network constructed by the present invention effectively solves the problem of semantic information loss or insufficiency in the fusion process and can fully fuse all effective information contained in the input infrared and visible light images. Based on the image fusion network provided by the present invention, the present invention also proposes an image fusion method based on semantic perception, which is a network model training method based on image semantic perception. By obtaining the image fusion model, semantic segmentation model and image fusion network model of the present invention, they are then used to perform the fusion processing of the infrared and visible light images of the present invention. The present invention solves the problem that the information contained in the input infrared and visible light images is incomplete and the semantic information is lost during the fusion process. It can fully combine all the effective feature information of the input image to generate a fused image with prominent picture details, comprehensive information and high image quality. It has great application potential and value in the fields of image processing, video surveillance, medical imaging, vehicle guidance, etc. Summary of the invention

[0005] The purpose of the present invention is to solve the problem of semantic information loss in the process of infrared and visible light image fusion, and to fully fuse all the effective information contained in them to improve the fusion quality of the image, so as to obtain a fused image with prominent details, comprehensive semantic information and high visual perception quality. The present invention provides an image fusion network based on semantic perception, and also provides an image fusion method based on semantic perception. The present invention can reduce the loss of semantic information in the image fusion process while fully fusing the effective information of the input images, and obtain a fused image with prominent details and comprehensive information corresponding to the input infrared and visible light images, which effectively solves the problem that the information contained in the input infrared and visible light images is incomplete and the semantic information is lost during the fusion process.

[0006] In order to achieve the above-mentioned purpose, the present invention proposes a technical solution, a semantic-aware image fusion network, characterized in that the network is constructed by introducing semantic perception into the fusion process of infrared and visible light images based on deep learning, and it can fuse all effective information of infrared and visible light images, and further optimize the semantic information of the fused image by introducing semantic loss therein, so as to obtain a fused image with prominent details and more semantic content; the network is composed of an image fusion module, a semantic segmentation module, a network optimization controller 1 and a network optimization controller 2 and contains three inputs, namely, "visible light image", "infrared image" and "label image", which are represented by input 1, input 2 and input 3 respectively; the image fusion module contains two inputs and one output, the two inputs are the input 1 and input 2, and the one output is output KK, which is a fused image obtained by the module by fusing its inputs, and is also an input of the semantic segmentation module and an input of the network optimization controller 2; the semantic segmentation module contains two inputs and two outputs, and the function realized is to perform semantic segmentation on the input fused image, the two inputs are input KK and input R, which are respectively related to the output KK and the implicit output of the network optimization controller 1, the two outputs are output V and output W, and the stored therein are respectively category prediction image data 1 and category prediction image data 2, both of which are inputs of the network optimization controller 1; the network optimization controller 1 contains three inputs and two outputs, the three inputs are input V, input W and input 3, the input V and input W correspond to the output V and output W respectively, the two outputs are output K and implicit output R, and the stored therein are respectively the semantic loss of the fused image and the parameter optimization update information for adjusting the parameters of the semantic segmentation module; the function of the network optimization controller 1 is to calculate the semantic loss of the fused image and optimize the network learnable parameters or model parameters of the semantic segmentation module through its implicit output; the network optimization controller 2 contains four inputs and one implicit output, the four inputs are the input KK and input K, which correspond to the output KK and output K respectively, as well as the input 1 and input 2; the function of the network optimization controller 2 is to calculate the loss of the fused image according to its input, and optimize the network learnable parameters or model parameters of the image fusion module using its implicit output;

[0007] The image fusion module is composed of a color space converter 1, an image encoder 1, an image decoder 1 and a color space converter 2 connected in series in sequence, and the functions implemented are "converting the image to be processed from RGB color space to YUV color space", "extracting deep features of the image", "fusing the extracted features to generate a fused image", and "converting the fused image from YUV color space to RGB color space" in sequence; the color space converter 1 has an input and an output, the one input is the input 1, and the output is the input of the image encoder I, referred to as input A; the image encoder 1 has two inputs and one output, the two inputs are the input A and the input 2, and the output is the input of the image decoder 1; the image decoder 1 has an input and an output, and the output is the input of the color space converter 2; the color space converter 2 has an input and an output, and the output is the output KK;

[0008] The color space converter 1 is composed of a flattening layer, an extraction layer, a calculation layer, and a reshaping layer connected in series in sequence, and the functions implemented by it are respectively "flattening its input into a two-dimensional array, in which each row represents a pixel point, and each column represents the red (R), green (G), and blue (B) channel values ​​of the corresponding pixel", "extracting the R, G, and B channel values ​​of the pixel from the two-dimensional array", "calculating the Y, U, and V channel values ​​from the extracted R, G, and B channel values", and "performing tensor connection on its input and adjusting the dimension of the tensor".

[0009] The image encoder 1 is composed of a convolution layer A, an LReLU activation function layer, a convolution layer AA, an LReLU activation function layer, a residual block 1, a residual block 2, a residual block 3, a residual block 4, a residual block 5, a residual block 6 and a splicing layer, and its functions are realized by the following process:

[0010] Process 1: Input the input A to the convolution layer A, LReLU activation function layer, residual block 1, residual block 3 and residual block 5 connected in series in sequence, and obtain the output BB1 after processing;

[0011] Process 2: The input 2 is processed by the concatenated convolution layer AA, the LReLU activation function layer, the residual block 2, the residual block 4, and the residual block 6 to obtain the output BB2;

[0012] Process 3: The obtained output BB1 and output BB2 are input to the splicing layer, and its output is the output of the image encoder 1; the function of the convolution layer A and the convolution layer AA is to extract the feature map of the input, and the role of the LReLU activation function layer is to enhance the representation ability of the neural network and solve the gradient disappearance problem; the function of the splicing layer is to splice the input feature map in the channel dimension; the residual block 1 is composed of a convolution layer B1, an LReLU activation function layer, a splicing layer 1, a convolution layer B2, an LReLU activation function layer, a splicing layer 2, a convolution layer B3, a calculation layer, a gradient calculation layer, a convolution layer B4 and an LReLU activation function layer, which contains one input and one output, divided into The convolution layer B1 comprises an input and an output, the input is the input C, and the output is used as the input of the LReLU activation function layer after the convolution layer B1; the LReLU activation function layer after the convolution layer B1 comprises an input and an output, and the output is an input of the convolution layer 1; the convolution layer 1 comprises two inputs and one output, the two inputs are the input C and the output of the LReLU activation function layer after the convolution layer B1, and the one output is used as the input of the convolution layer B2; the convolution layer B2 comprises an input and an output, and the output is used as the LReLU activation function layer after the convolution layer B2 The LReLU activation function layer after the convolution layer B2 contains an input and an output, and the output is an input of the splicing layer 2; the splicing layer 2 contains two inputs and one output, and its other input is the output of the splicing layer 1, and the output is used as the input of the convolution layer B3; the convolution layer B3 includes an input and an output, and the output is used as an input of the operation layer; the gradient operation layer includes an input and an output, and the input is the input C, and the output is used as the input of the convolution layer B4; the convolution layer B4 includes an input and an output, and the output is used as another input of the operation layer, and the operation layer includes two An input and an output, the output is processed by an LReLU activation function layer to obtain the output C; the convolution layers B1, B2, B3 and B4 are used to extract features from the input feature map, the LReLU activation function layer is used to enhance the representation ability of the neural network and solve the gradient disappearance problem, the function of the splicing layer is to splice the input feature map in the channel dimension, the function of the gradient operation layer is to extract the edge information of the feature map, and the function of the operation layer is to add the input feature map element by element; the residual block 2, residual block 3, residual block 4, residual block 5, residual block 6 and the residual block 1 have the same structure, and are all used to further extract fine-grained information and high-level semantic features of the feature map;

[0013] The image decoder 1 is composed of a convolution layer C1, a normalization layer, an LReLU activation function layer, a convolution layer C2, a normalization layer, an LReLU activation function layer, a convolution layer C3, a normalization layer, an LReLU activation function layer, a convolution layer C4, a normalization layer, an LReLU activation function layer, a convolution layer C5, a normalization layer, and a Tanh activation function layer connected in series in sequence; the convolution layers C1, C2, C3, C4 and C5 have the same structure and function, and are used to extract features from the input feature map; the function of the normalization layer is to standardize all pixels of the feature map to make the network more stable and efficient; the function of the LReLU activation function layer is to introduce nonlinear transformation while avoiding the gradient disappearance problem; the function of the Tanh activation function layer is to map the output feature map to the range of [-1, 1];

[0014] The color space converter 2 has the same structure as the color space converter 1, and is composed of a flattening layer, an extraction layer, a calculation layer, and a reshaping layer connected in series in sequence, and its functions are respectively "flattening the image into a two-dimensional array, each row of the two-dimensional array represents a pixel point, and each column represents the value of the Y, U, and V channels", "extracting the Y, U, and V channel values ​​from the two-dimensional array respectively", "calculating the R, G, and B channel values ​​through the extracted Y, U, and V channel values", and "connecting tensors and adjusting the tensor dimensions".

[0015] The semantic segmentation module is composed of an image encoder 2 and an image decoder 2, and its functions are respectively "compressing and extracting the image features of its input" and "generating the classification of each pixel according to the spatial resolution of its input recovery data, thereby realizing image segmentation"; the image encoder 2 contains one input and three outputs, the input is the input KK, and the three outputs are output X, output Y and output Z, and their physical meanings are respectively "feature map containing basic edge and texture information", "feature map containing more complex local structure and texture information" and "feature map containing high-level semantic information and global context", which are all used as inputs of the image decoder 2; the image decoder 2 includes three inputs and two outputs, the two outputs are the output V and output W, and the category prediction image data 1 and the category prediction image data 2 are stored therein, which are both used as inputs of the network optimization controller 1;

[0016] The image encoder 2 is composed of a convolution layer D1, a normalization layer, a ReLU activation function layer, a pooling layer, two series-connected residual blocks A, two series-connected residual blocks B, two series-connected residual blocks C and two series-connected residual blocks D, which are sequentially connected in series; the two series-connected residual blocks B include an input and an output, the input is the output of the two series-connected residual blocks A, and the one output is the output X, which is simultaneously used as the input of the image decoder 2 and the two series-connected residual blocks C; the two series-connected residual blocks C include an input and an output, the input is the output X of the two series-connected residual blocks B, and the output is the output Y, which is simultaneously used as the image decoder 2 and the two series The input of the residual block D connected in series; the two residual blocks D connected in series include an input and an output Z, the input is the output Y of the two residual blocks C connected in series, and the one output is the output Z, which serves as the input of the convolution layer E1 of the image decoder 2; the functions of the convolution layer D1, the pooling layer, the two residual blocks A connected in series, the two residual blocks B connected in series, the two residual blocks C connected in series, and the two residual blocks D connected in series are to extract the image data features of different levels of their inputs; the function of the normalization layer is to standardize all pixels of the input feature map; the function of the ReLU activation function layer is to introduce nonlinear transformation while avoiding the gradient disappearance problem; the residual block A of the image encoder 2 is composed of A convolution layer D2, a normalization layer, a ReLU activation function layer, a convolution layer D3, a normalization layer, a calculation layer and a ReLU activation function layer are connected in series; the convolution layer D2, the normalization layer, the RELU activation function layer, the convolution layer D3, and the normalization layer are connected in series in sequence and each of them contains an input and an output, and the output of the normalization layer after the convolution layer D3 is used as an input of the connection layer; the calculation layer includes two inputs and one output, and the two inputs are respectively the input of the residual block A and the output of the normalization layer after the convolution layer D3, and the one output is used as the input of the ReLU activation function layer after the calculation layer; the ReLU activation function layer after the calculation layer contains an input and an output, and the The output of the residual block A is the output of the residual block A; the function of the convolution layers D2 and D3 is to extract the features of the input feature map; the function of the normalization layer is to standardize all pixel data of the feature map input thereto; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the operation layer is to perform a sum operation on its input; the structures of the residual block B, residual block C, and residual block D are exactly the same and each contains one input and one output, and each is composed of a convolution layer D4, a normalization layer, a ReLU activation function layer, a convolution layer D5, a normalization layer, a convolution layer D6, a normalization layer, an operation layer, and a ReLU activation function layer, and the functions thereof are realized by the following process:

[0017] Process 1: The input is processed by the convolution layer D4, normalization layer, ReLU activation function layer, convolution layer D5, and normalization layer in series to obtain the output NN1;

[0018] Process 2: The input is processed by the convolutional layer D6 and the normalization layer in series to obtain the output NN2;

[0019] Process 3: The obtained output NN1 and output NN2 are input to the operation layer, and the output of the operation layer is obtained after processing, and then the output of the operation layer is input to the ReLU activation function layer to obtain the output of the residual block; the convolution layer D4, the convolution layer D5, and the convolution layer D6 are used to extract the features of the input feature map; the function of the normalization layer is to standardize all pixels of the feature map output by the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the operation layer is to perform element-wise summation operation on its input;

[0020] The image decoder 2 is composed of a convolution layer E1, a normalization layer, a ReLU activation function layer, a bilateral attention decoder 1, an output block 1, a convolution layer E2, a normalization layer, a ReLU activation function layer, a bilateral attention decoder 2 and an output block 2; the convolution layer E1, the normalization layer, and the ReLU activation function layer are connected in series in sequence, and each of them contains an input and an output; the input of the convolution layer E1 is the input Z, which corresponds to the output Z, and the output H of the ReLU activation function layer is used as an input of the bilateral attention decoder 1; the bilateral attention decoder 1 includes two inputs and one output, and the two inputs are respectively Input H and the input Y, the input H corresponds to the output H of the ReLU activation function layer after the convolution layer E1, the input Y corresponds to the output Y of the residual block C, and the one output is simultaneously used as the input of the convolution layer E2 and the output block 1; the output block 1 includes an input and an output, the one input is the output of the bilateral attention decoder 1, and the one output is the output V of the semantic segmentation module; the convolution layer E2, the normalization layer, and the ReLU activation function layer are connected in series in sequence and each of them contains an input and an output, and the input of the convolution layer E2 is the bilateral attention decoder 1 The output of the ReLU activation function layer, the output HH of the ReLU activation function layer is an input of the bilateral attention decoder 2; the bilateral attention decoder 2 includes two inputs and one output, the two inputs are input HH and the input X, which correspond to the output HH of the ReLU activation function layer after the convolution layer E2 and the output X, respectively, and the one output is used as the input of the output block 2; the output block 2 includes one input and one output, the one input is the output of the bilateral attention decoder 2, and the one output is the output W of the semantic segmentation module; the functions of the convolution layer E1 and the convolution layer E2 To extract the features of the input feature map; the function of the normalization layer is to standardize all pixels of the feature map output by the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the functions of the output block 1 and the output block 2 are to convert the final feature map of the network into the classification result of each pixel; the bilateral attention decoder 1 and the bilateral attention decoder 2 have the same structure and function, which are both composed of a spatial attention branch block, a channel attention branch block, a convolution layer E3, an upsampling layer and a pooling fusion block, and the low-level feature map and the high-level feature map input are fused through the following process:

[0021] Process 1: The input Y or X is processed by the spatial attention branch and convolution layer E3 in series to obtain the output B;

[0022] Process 2: The input H or HH is processed by the channel attention branch layer and the upsampling layer connected in series to obtain the output D;

[0023] Process 3: The obtained output B and output D are input to the pooling fusion module, and the output of the bilateral attention decoder is obtained after processing; the function of the spatial attention branch block is to refine the input low-level feature map feature map and extract more spatial information; the function of the channel attention branch block is to refine the input high-level feature map and extract more semantic information; the function of the convolution layer E3 is to extract the features of the input low-level feature map; the function of the upsampling layer is to restore the resolution of the input high-level feature map; the function of the pooling fusion block is to fuse the input high- and low-level feature maps to generate a fusion feature map that accurately combines semantic information and spatial information; the spatial attention branch block is composed of the convolution layers AA1, The network is composed of a normalization layer, a ReLU activation function layer, a maximum pooling layer, a mean pooling layer, a splicing layer, a convolution layer AA2, a normalization layer, a Sigmoid activation function layer and an operation layer; the convolution layer AA1, the normalization layer, and the ReLU activation function layer are connected in series in sequence and each of them has an input and an output; the input of the convolution layer AA1 is the input of the spatial attention branch block, and the output of the ReLU activation function layer is also used as the input of the maximum pooling layer and the mean pooling layer; the mean pooling layer has an input and an output, the one input is the output of the ReLU activation function layer, and the one output is the input of the splicing layer; the mean pooling layer has an input and an output, The one input is the output of the ReLU activation function layer, and the one output is another input of the splicing layer; the splicing layer contains two inputs and one output, and the one output is used as the input of the convolution layer AA2; the convolution layer AA2, the normalization layer, and the Sigmoid activation function layer are connected in series in sequence and each contains one input and one output, the input of the convolution layer AA2 is the output of the splicing layer, and the output of the Sigmoid activation function layer is an input of the operation layer; the operation layer includes two inputs and one output, and the other input corresponds to the input Y or the input X, and the one output is the output of the spatial attention branch block; the convolution layer AA1 , the function of the convolution layer AA2 is to extract features from the input feature map; the function of the maximum pooling layer is to extract the most significant features of the input feature map; the function of the mean pooling layer is to extract the average representation of the global features of the input feature map; the function of the splicing layer is to splice its input in the channel dimension; the function of the normalization layer is to standardize all pixels of the output feature map of the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the Sigmoid activation function layer is to generate a spatial attention map containing attention weights by introducing nonlinear transformation; the function of the operation layer is to perform multiplication operation on its input;The channel attention branch block is composed of a global average pooling layer, a convolution layer AA3, a normalization layer, a Sigmoid activation function layer and a calculation layer. The global average pooling layer, the convolution layer AA3, the normalization layer and the Sigmoid activation function layer are connected in series in sequence and each of them contains an input and an output. The input of the global average pooling layer is the input of the channel attention branch block, and the output of the Sigmoid activation function layer is an input of the calculation layer; the calculation layer contains two inputs and one output, and the two inputs are respectively the output of the Sigmoid activation function layer and the input of the channel attention branch block, and the one output is the output of the channel attention branch block; The function of the global average pooling layer is to compress the input feature map information into a global feature description; the function of the Sigmoid activation function layer is to generate a channel attention map containing attention weights by introducing nonlinear transformation; the function of the operation layer is to perform multiplication operation on its input and adjust the input feature map; the pooling fusion block is composed of an average pooling layer, a splicing layer, a convolution layer AA4, a normalization layer and a ReLU activation function layer connected in series in sequence, the average pooling layer contains one input and one output, the input is input D, which corresponds to the output D, and the one output is used as an input of the splicing layer; the splicing layer contains two inputs and one output, the The two inputs are the output of the average pooling layer and the input B, and the input B corresponds to the output B of the convolution layer AA; the convolution layer AA4, the normalization layer and the ReLU activation function layer are connected in series in sequence and each of them contains an input and an output, the input of the convolution layer AA4 is the output of the splicing layer, and the output of the ReLU activation function layer is the output of the pooling fusion block; the function of the average pooling layer is to smooth the input feature map; the function of the splicing layer is to splice its input in the channel dimension; the function of the convolution layer AA4 is to reconstruct the input feature map; the function of the normalization layer is to standardize all pixels of the output feature map of the input convolution layer processing; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the output block 1 and the output block 2 have the same structure, both of which are composed of a convolution layer AA5, a normalization layer, a ReLU activation function layer, a convolution layer AA6 and an upsampling layer connected in series in sequence, and each of them contains an input and an output; the function of the convolution layer AA5 is to extract the features of the input feature map; the function of the normalization layer is to standardize all pixels of the input convolution layer output feature map; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the convolution layer AA6 is to extract the features of the input feature map and generate a category prediction map;The function of the upsampling layer is to align the feature map output by the upsampling layer with the input image data in space;

[0024] The network optimization controller 1 is composed of a semantic loss function and an optimizer connected in series; the semantic loss function has three inputs and one output, and the one output is output K, which serves as the input of the optimizer and the output K of the network optimization controller 1; the optimizer has one input and one output, and the output is the implicit output R of the network optimization controller 1; the semantic loss function L semantic for

[0025] L semantic =L main +λL aux

[0026]

[0027] Among them, L main is the main semantic loss function, which is the cross entropy of the main segmentation result "output W" of the semantic segmentation module and the semantic segmentation label "input 3", L aux is the auxiliary semantic loss function, which is the cross entropy of the auxiliary segmentation result "output V" of the semantic segmentation module and the semantic segmentation label "input 3", λ is the weight constant for balancing the main semantic loss and auxiliary semantics, L S ,I s ,I Sa are the input 3, the "output W" and "output V" of the semantic segmentation module, H, W and C are the height, width and number of channels of the input data respectively; the output K is the semantic loss function L semantic The output of reflects the richness of the semantic information contained in the fused image; the optimizer optimizes the learnable parameters of the semantic segmentation module including the network structure, weights and bias parameters according to its input and finally obtains the network model, and the optimization rule adopted is to make the gradients of all neurons of the semantic segmentation module converge as a whole;

[0028] The network optimization controller 2 is composed of a joint loss function and an optimizer in series, wherein the joint loss function has four inputs and one output, and the optimizer has one input and one output; the joint loss function L joint It is obtained through the following process:

[0029] Process 1: Constructing content loss function L content ,

[0030]

[0031] Among them, If is the output KK of the image fusion module, I ir ,I vi are the input 1 and input 2, H and W are the height and width of the image, respectively, ||I f -max(I ir ,I vi )||1 is the norm operation, representing the intensity distribution loss of the fused image. It is the Sobel gradient operator that operates and represents the fine-grained texture information of the image. is a norm operation, characterizing the texture distribution loss of the fused image, and α is a weight constant for balancing the intensity distribution loss and the texture distribution loss;

[0032] Process 2: Constructing the joint loss function L joint ,

[0033] L loint =L content +βL semantic ,

[0034] Among them, L content , L semantic are the content loss and semantic loss respectively, β=γ×(m-1) is the weight parameter for measuring the contribution of semantic loss, m is the number of iterative training of the image fusion network, γ is the adjustment coefficient for adjusting the contribution of the semantic loss, which is an integer; the optimizer optimizes the learnable parameters of the image fusion module including network structure, weights, biases and other parameters according to its input and finally obtains the network model, and the optimization rule adopted is to make the gradients of all neurons of the image fusion module descend and converge as a whole.

[0035] Furthermore, the image fusion network based on semantic perception is characterized in that the network is built based on the PyTorch deep learning framework; the weight parameters of the loss function of the network optimization controller 1 and the network optimization controller 2 are weight parameters λ∈[0.00001,1.0], ɑ∈[0.0001,1000], γ∈[1,10000], preferably, λ=0.1, ɑ=10, γ=1; the optimizer of the network optimization controller 1 is an SGD optimizer, which is implemented using the Torch.Optim.SGD algorithm, and its input parameters include batch size, momentum, weight decay, initial Learning rate, learning rate scheduling attenuation index, preferably, the batch size is 16, the momentum is 0.9, the weight decay is 0.0005, the initial learning rate is 0.01, and the learning rate scheduling attenuation index is 0.9, which optimizes the semantic segmentation module guided by semantic loss; the optimizer of the network optimization controller 2 is an Adam optimizer, which is implemented using the Torch.Optim.Adam algorithm, and its input parameters include batch size, first-order moment estimation decay rate β1, second-order moment estimation decay rate β2, constant ε, weight decay and initial learning rate, preferably, the batch size is 8, β1=0.9, β2=0.99, and constant ε=1e -8 , the weight decay is 0.0002, the initial learning rate is 0.001, and the image fusion module is optimized with the joint loss as the guide;

[0036] The dimension splicing operation of the splicing layer of the image fusion module adopts the Torch.cat method of PyTorch to splice along the second dimension, that is, the column dimension. The flattening layer is implemented by the Torch.transpose and Torch.reshape algorithms, and the reshaping layer is implemented by the Torch.unsqueeze, Torch.cat and Torch.reshape algorithms. The color space converter 1, color space The calculation layer of the interconverter 2 uses the relations Y=0.299×R+0.5287×G+0.114×B, U=-0.147×R-0.289×G+0.436×B, V=0.615×R-0.515×G-0.1×B and R=Y+1.403×(V-0.5), G=Y-0.344×(U-0.5)-0.714×(V-0.5), B=Y+1.773×(U-0.5) for color conversion;

[0037] The summation operation layer and the multiplication operation layer are implemented by Torch.add algorithm and Torch.mul algorithm respectively; the pooling layer, average pooling layer, global average pooling layer, maximum pooling layer and mean pooling layer are implemented by Torch.nn.maxpool2d, Torch.nn.functional.avg_pool2d, Torch.nn.avg_pool2d, Torch.max algorithm and Torch.mean algorithm respectively; the upsampling layer is implemented by Torch.nn.functional.interpolate algorithm, and the sampling mode is "bilinear";

[0038] The convolutional layers include convolutional layer A, convolutional layer AA, convolutional layer B1, convolutional layer B2, convolutional layer B3, convolutional layer B4, convolutional layer C1, convolutional layer C2, convolutional layer C3, convolutional layer C4, convolutional layer D1, convolutional layer D2, convolutional layer D3, convolutional layer D4, convolutional layer D5, convolutional layer D6, convolutional layer E1, convolutional layer E2, convolutional layer E3, convolutional layer AA1, convolutional layer AA2, convolutional layer AA3, convolutional layer AA4, convolutional layer AA5, and convolutional layer AA6. The initial value of the weight of the convolution kernel is a normal distribution with a mean of 0, and the initial value is The Torch.nn.init.kaiming_normal algorithm is used for generation; the convolution operation of the convolution layer is implemented by the Torch.nn.Conv2d algorithm; the construction parameters of the convolution layer A, convolution layer AA, convolution layer B1, convolution layer B2, convolution layer C1, convolution layer C2, and convolution layer C3 of the image fusion module are: the convolution kernel size is 3*3, the convolution kernel step size is 1, and the convolution kernel padding is 1; the construction parameters of the convolution layer B3, convolution layer B4, and convolution layer C4 of the image fusion module are: the convolution kernel size is The convolution kernel size is 1*1, the convolution kernel step size is 1, and the convolution kernel padding is 0; the construction parameters of the convolution layer D2, convolution layer D3, convolution layer D4, convolution layer D5, convolution layer E1, convolution layer E2, convolution layer E3, convolution layer AA2, and convolution layer AA5 are: the convolution kernel size is 3*3, the convolution kernel step size is 1, and the convolution kernel padding is 1; the construction parameters of the convolution layer D6, convolution layer AA1, convolution layer AA3, convolution layer AA4, and convolution layer AA6 of the image fusion module are: the convolution kernel size is 1*1, the convolution kernel step size is 1, and the convolution kernel padding is 0 ; The construction parameters of the convolution layer D1 are: the convolution kernel size is 7*7, the convolution kernel step size is 2, and the convolution kernel padding is 3; the normalization layer is implemented using the Torch.nn.BatchNorm2d algorithm, the Leakyrelu activation function layer is implemented using the Torch.nn.leaky_relu algorithm, the Sigmoid activation function layer is implemented using the Torch.nn.Sigmoid algorithm, and the Tanh activation function layer is implemented using the Torch.nn.Tanh algorithm.

[0039] The present invention also provides a semantically-aware image fusion method, which is characterized in that the method is based on the image fusion network provided by the present invention and comprises the following steps:

[0040] Step 1: Construct an image training dataset A for training. The dataset is composed of a series of image data pairs consisting of infrared images, visible light images, and label images. The infrared images and visible light images are registered images collected for the same scene, and the label images are corresponding semantic label images representing the category to which their content objects belong. Each pixel in the label images is assigned a category label, which is a semantic label used to represent the category information of the pixel. The size of the image is MM×NN pixels.

[0041] Step 2: Perform random cropping on all image data pairs in the image training dataset A obtained in step 1. The size of the cropped image is smaller than the size of the original image, which is MM1×NN1 pixels. Perform tensor normalization on the obtained image, convert the image into a Pytorch tensor form and normalize its elements to [0, 1] to obtain the image training dataset.

[0042] Step 3: The image training data set obtained in step 2 is called the image data training set B, and any set of image data pairs is taken out therefrom, and the infrared image and the visible light image therein are respectively input as the input 1 and the input 2 to the image fusion module described in claims 1-2 to obtain its output, namely the output KK; then the obtained output KK is input to the semantic segmentation module described in claims 1-2 to obtain its output, namely the output V and the output W; then the obtained output V and the output W and the label image in the image data pair are input as the input 3 to the network controller 1 described in claims 1-2 to obtain its output, namely the output K;

[0043] Step 4: Update the output K described in step 3 according to the formula β = γ × (m-1), that is, the semantic loss L described in claims 1-2 semantic The weight parameter β;

[0044] Step 5: The output KK obtained in step 3 is used as the input KK described in claim 1-2, and the output K is used as the input K weight parameter described in claim 1-2, multiplied together with the input 1 and input 2 described in step 3, and inputted into the network optimization controller 2 described in claim 1-2, and the image fusion module described in claim 1-2 is optimized after being processed by the controller and according to its output, so as to realize the training of a group of image data pairs of the image training data set B described in step 2;

[0045] Step 6: Continuously repeat steps 3-5 to implement training of all image data pairs of the image training data set B in step 2, and at this time, one training of the image fusion module in claim 1 is completed, and the training result is stored in the image fusion module;

[0046] Step 7: Repeat step 6 continuously, train the image fusion module P times, save the training results in the image fusion module, and obtain the network training model of the image fusion module, which is called image fusion model BE;

[0047] Step 8: Repeat step 2 to obtain an image training data set, input all the data in the data set into the image fusion model BE obtained in step 7, and obtain the fused image KK outputted by the model, i.e., output KK; then form a data pair with the corresponding label image, infrared image, and visible light image to form an image training data set C;

[0048] Step 9: Take any set of data pairs of "output KK, label image, infrared image, visible light image" from the image training data set C obtained in step 8, and then synchronously perform random image processing operations of "image horizontal flip", "image vertical flip", and "size adjustment processing", and resize the processed image. The pixel size of the adjusted image is smaller than the size of the original image, and then the adjusted image is tensor normalized to convert it into a tensor mode and normalize its elements to [0, 1], and the processed image data are obtained, which are output KKc, output BQc, output HWc, and output KJc, respectively, which correspond to the input KK, input 3, input 1, and input 2 described in claims 1-2, respectively;

[0049] Step 10: Input the "output KKc" obtained in step 9 into the semantic segmentation module described in claims 1-2 to obtain its output, namely the output V and output W, which respectively represent the "category prediction image data 1" and "category prediction image data 2" contained in its input, which respectively correspond to the input V and input W described in claims 1-2;

[0050] Step 11: Input the output V and output W obtained in step 10 and the output BQc obtained in step 9 to the network optimization controller 1 described in claims 1-2 to obtain the semantic loss of its output, that is, the output K or input K described in claims 1-2, and optimize the semantic segmentation module described in claims 1-2 according to its implicit output R. At the same time, input the semantic loss and the output KKc obtained in step 9 as well as the output HWc and the output KJc to the network optimization controller 2 and further optimize the learnable parameters of the image fusion module according to its output, so as to realize the training of a group of image data pairs of the image data training set C described in step 8, and the training results are stored in the semantic segmentation module and the image fusion module;

[0051] Step 12: Repeat steps 9-11 continuously to implement the training of all image data pairs in the image data training set C described in step 8. At this time, the training of the semantic segmentation module described in claims 1-2 is completed once, and the training results are stored in the semantic segmentation module;

[0052] Step 13: Repeat step 12 continuously, train the semantic segmentation module Q times, save the training results in the semantic segmentation module, and obtain a network training model of the semantic segmentation module, which is called the semantic segmentation model LE; at this time, the image fusion network of claim 1-2 is trained once, and the training results are saved in the image fusion network;

[0053] Step 14: Continuously repeat step 3 and steps 5-13, and train the image fusion network described in claims 1-2 M times to obtain a network training model of the image fusion network described in claims 1-2, which is called image fusion network model AA;

[0054] Step 15: Input the infrared image and visible light image to be processed as the input 1 and input 2 described in claims 1-2 to the image fusion module of the image fusion network model AA obtained in step 14, so that it can output a fused image with prominent details and rich semantic information after image fusion processing of the input.

[0055] Furthermore, the image fusion method based on semantic perception is characterized in that the format of the image in the image training data set A described in step 1 is "JPG" or "PNG", the size of the image is MM×NN pixels, MM=640, NN=480, the number of image data pairs contained is W=W0∈[500,10000], the number of category labels is K=K0∈[2,10000], which are cars, people, bicycles, curves, parking areas, guardrails, tones and backgrounds, etc., preferably K=9; the "image horizontal flip", "image vertical flip", "size adjustment processing" and "tensor normalization processing" described in step 9 respectively adopt "RandomHorizontalFlip", "RandomVerticalFlip", "Resize" and "Transforms" module of the "Torchvision" library of the PyTorch deep learning framework. "ToTensor" method; the resizing process described in step 9 is to adjust the size of the image to no more than MM×NN pixels; the random cropping process described in step 2 is implemented by the randint method of the random library of Python language, and the "tensor normalization process" is implemented by the Torchvision.transforms.toTensor method of PyTorch, and the MM1=340, NN1=550; the number of training times of the image fusion module described in step 7 P=P0∈[2700,100000]; the number of training times of the semantic segmentation module described in step 13 Q=Q0∈[20000,200000]; the number of training times of the image fusion network described in step 14 M=M0∈[1,100000]; the weight parameters of the loss function of the network optimization controller 2 described in step 5 and the network optimization controller 1 described in step 11 are α=10, λ=0.1, and γ=1.

[0056] The semantic-aware image fusion network and image fusion method provided by the present invention also have the following characteristics:

[0057] 1. The image fusion network described in the present invention can be constructed using Python and PyTorch framework.

[0058] 2. Step 10 involved in the technical solution of the present invention adopts a data enhancement strategy, which can be implemented by using the "RandomHorizontalFlip", "RandomVerticalFlip" and "Resize" methods of the Transforms module of the Torchvision library of the PyTorch framework. The first two can enrich the input data so that the network can obtain more image feature information during training, while the latter is to make the size of the input image consistent.

[0059] 3. The function of the color space converter 1 described in the present invention is to convert the input visible light image from RGB color space to YUV color space, and the function of the color space converter 2 is to convert its input image from YUV color space to RGB color space. Such processing can effectively process the brightness and color information of the image during image fusion, and can better fuse the heat source information of the infrared image and the texture information of the visible light image.

[0060] 4. The image fusion module provided by the present invention can effectively fuse infrared and visible light images. Compared with other image fusion methods, the image fusion network of the present method adopts a lightweight network structure, and uses multiple convolutional layers and multiple residual blocks to extract deep features and high-level semantic features with rich fine-grained detail information from infrared and visible light images. The extracted feature information is reconstructed through multiple convolutions by multiple convolutional layers and multiple activation function layers to generate the final fused image.

[0061] 5. The function of the semantic segmentation module provided by the present invention is to perform semantic segmentation on the input fused image, and use multiple convolutional layers, instance normalization layers, LeakyReLU activation function layers, and multiple residual blocks to extract image structure features and reduce data dimension on the input data to obtain its key information and features. The extracted feature information is received and processed by multiple convolutional layers, activation function layers, operation layers, and sampling layers, and then the damaged boundary and detail information are restored by gradually adding the underlying feature map to the output, and the final category prediction data is output.

[0062] 6. Although the network optimization controller 1 and the network optimization controller 2 provided by the present invention both contain a loss function and an optimizer in form, the loss functions they use are different; the function of the network optimizer 1 is to calculate the semantic loss and reduce the value of the semantic loss function through multiple trainings, and output its semantic loss while optimizing the semantic segmentation module; one input of the network optimization controller 2 is the semantic loss calculated by the network optimizer 1, which serves as a part of the joint loss of the network optimization controller 2 and reduces the value of the joint loss function through multiple trainings, continuously optimizing the image fusion module, and ultimately making the obtained fused image contain richer content details and comprehensive semantic information, and the quality of the fused image is high.

[0063] 7. The image fusion method provided by the present invention is based on the image fusion network. It performs semantic segmentation on the fused image through a semantic segmentation module. The semantic loss of the obtained image is used to guide the optimization of the semantic segmentation module and the image fusion module. This can not only meet the visual quality and statistical evaluation indicators of the image, but also fully enhance the semantic information of the fused image, so that the obtained semantic information is complete and comprehensive.

[0064] 8. The fused image generated by the method provided by the present invention has rich content details and semantic information. The semantic information contained in the fused image can better meet the needs of advanced tasks such as target detection, and has great application potential and value in many fields such as image processing, image content analysis, computer vision, and machine vision.

[0065] 9. L of the present invention semantic It not only participates in the optimization training of the semantic segmentation module, but also participates in the optimization training of the image fusion module as the input of the network optimization controller 2, thereby enhancing the semantic information of the obtained fused image, and the semantic information is complete and comprehensive. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 The structure of the image fusion network described in the present invention and its relationship diagram.

[0067] Figure 2 The structure and relationship diagram of the image fusion module described in the present invention.

[0068] Figure 3 The structure and relationship diagram of the color space converter 1 and the color space converter 2 of the image fusion module of the present invention.

[0069] Figure 4 1 is a diagram showing the structure and relationship of the image encoder 1 described in the present invention.

[0070] Figure 5 1 is a diagram showing the structure and relationship of residual blocks 1-6 of the image encoder 1 of the present invention.

[0071] Figure 6 The structure and relationship diagram of the image decoder 1 of the image fusion module of the present invention.

[0072] Figure 7 The structure and relationship diagram of the semantic segmentation module described in the present invention.

[0073] Figure 8 The structure and relationship diagram of the image encoder 2 of the semantic segmentation module described in the present invention.

[0074] Fig. 9 1 is a diagram showing the structure of the residual block A of the image encoder 2 of the present invention and its relationship diagram.

[0075] Fig.10 1 is a diagram showing the structure of residual blocks B, C, and D of the image encoder 2 of the present invention and their relationship.

[0076] Fig.11 The structure and relationship diagram of the image decoder 2 of the semantic segmentation module described in the present invention.

[0077] Fig.12 The structure and relationship diagram of the bilateral attention decoders 1 and 2 of the image decoder 2 described in the present invention.

[0078] Fig.13 The structure and relationship diagram of the spatial attention branch blocks of the bilateral attention decoders 1 and 2 described in the present invention.

[0079] Fig.14 The structure and relationship diagram of the channel attention branch blocks of the bilateral attention decoders 1 and 2 described in the present invention.

[0080] Fig.15 The structure of the pooling fusion block of the bilateral attention decoder 1 and 2 described in the present invention and its relationship diagram.

[0081] Fig.16 1 is a diagram showing the structure and relationship of output blocks 1 and 2 of the image decoder 2 of the present invention.

[0082] Fig.17 It is a structure diagram and relationship diagram of the network optimization controller 1 described in the present invention.

[0083] Fig.18 It is the structure and relationship diagram of the network optimization controller 2 described in the present invention.

[0084] Fig.19 This is a training control flow chart of the image fusion network described in the present invention.

[0085] Fig. 20 This is the infrared image inputted in the first embodiment of the present invention.

[0086] Fig.21 This is the visible light image inputted in the first embodiment of the present invention.

[0087] Fig. 22 This is a fused image output by the image fusion network model of the first embodiment of the present invention. DETAILED DESCRIPTION

[0088] The following is a further description of the specific implementation of the present invention in conjunction with the accompanying drawings and examples. The following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0089] Figure 1-18 The structure and structural relationship diagram of the semantic-aware image fusion network of the present invention are formed. Fig.19The training control flow chart of the image fusion network of the present invention includes the network training process and the construction of the network model; combined with the technical solution of the image fusion network and the technical solution of the image fusion method provided by the present invention, it is possible to construct the construction and application of related neural networks and network models for related fields and specific problems, solve the problem that the information contained in the input infrared and visible light images is incomplete and the semantic information is lost during the fusion process of infrared and visible light images, generate a fused image with prominent details and more semantic information, and the image information is complete and of high quality. The following embodiments provided by the present invention are based on Figure 1-19 The structure and control flow chart of the present invention and the technical solution provided by the present invention are implemented, and the specific embodiments are as follows:

[0090] Embodiment 1:

[0091] First, an image training data set A is constructed, which contains W=1000 sets of image data pairs, each set of data pairs contains infrared images, visible light images and label images, the first two of which are registered image pairs, and the image format of the latter is PNG format, and the image size is MM×NN pixels, MM=480, NN=640; the number of iterative training times in this embodiment is M=10, P=2700, Q=20000; the number of category labels of the label image is 9, including cars, people, bicycles, curves, parking areas, guardrails, tones and backgrounds. Then, the Randint method in the Random library of the Python language is used to randomly crop all infrared images, visible light images and label images in the data set A, and the resulting image size is MM1*NN1, MM1=340, NN1=550. The “ToTensor” method of the “Transforms” module of the “Torchvision” library of the PyTorch framework is used to perform tensor normalization on the cropped infrared, visible light images, and label images, converting them into Pytorch tensor form and normalizing their elements to [0, 1] to obtain an image training dataset B, in which the infrared images and visible light images of the data pairs are one-to-one corresponding.

[0092] Using the image fusion network of the present invention, see Figure 1-6 , build the network structure of the image fusion module of this embodiment, and its network structure and construction parameters are the same as those described in the technical solution of the present invention. Use the image fusion method provided by the present invention to build the image fusion model BE of this embodiment, and the training control flow chart is shown in Fig.19Any set of infrared images and their corresponding visible light images in the data set B are input as input 1 and input 2 to the image fusion module of the present invention, which processes the input images to obtain a fused image, namely output KK; then the semantic loss is updated according to the semantic loss weight parameter β=γ×(m-1) and the semantic loss function, namely output K of the present invention. Then, the image fusion network of the present invention is used, see Figure 1-18 , build a network optimization controller 2, whose network structure and construction parameters are the same as those described in the technical solution of the present invention, the relevant parameters of the loss function α=10, γ=1.0, and the optimizer therein adopts the Train.AdamOptimizer algorithm, the batch size parameter of the algorithm is 8, the decay rate parameter β1 of the first-order moment estimation is 0.9, the decay rate parameter β2 of the second-order moment estimation is 0.99, and the constant ε is 1e -8 , the weight decay parameter is 0.0002, and the initial learning rate parameter is 0.001, which optimizes the image fusion module with the joint loss as the guide. The output KK of this embodiment and the updated semantic loss are used as the input of the network optimization controller 2, which optimizes the image fusion module according to the input to realize the training of a set of image data pairs of the data set B. Fig.19 The training control flow chart shown in FIG. 1 trains all the group images of the data set B to complete one training of the image fusion module. Fig.19 The training process shown performs P training on the image fusion module to obtain the image fusion model BE of this embodiment.

[0093] The above random cropping operation is then performed on all image pairs of the data set A of this embodiment, and the resulting image size is M1*N1, M1=300, N1=480; the cropped infrared, visible light images and label images are subjected to tensor normalization processing by using the "ToTensor" method of the "Transforms" module of the "Torchvision" library of the PyTorch framework, and are converted into Pytorch tensor form and their elements are normalized to [0, 1] to obtain an image training data set CC, in which the infrared and visible light images are one-to-one corresponding, and each pair of infrared and visible light images corresponds to a label image; using the semantic segmentation module of the present invention, see Figure 1-18 , build its network structure, and its network structure and construction parameters are the same as those described in the technical solution of the present invention. The semantic segmentation model LE of this embodiment is constructed by using the image fusion method provided by the present invention. The training control process is shown in Fig.19. Input any group of infrared images and the corresponding visible light images in the data set CC into the obtained image fusion model BE to obtain the output fusion image KK, i.e., output KK; use the PyTorch framework "Torchvision.Transforms" module "RandomHorizontalFlip", "RandomVerticalFlip", and "Resize" methods to synchronously perform random image processing operations such as "image horizontal flip", "image vertical flip", and "size adjustment processing" on the output KK and the corresponding label image to obtain the "input KK" of the semantic segmentation module and the "input 3" of the network optimization controller 1. Then input the obtained "input KK" into the semantic segmentation module of the present invention, see Figure 1-18 , the input KK is processed by the semantic segmentation module in turn to obtain "category prediction image data 1" and "category prediction image data 2"; using the image fusion network provided by the present invention, see Figure 1-18 , build the network optimizer 1 of this embodiment, its network structure and construction parameters are the same as those described in the technical solution part of the present invention, the relevant parameter λ of the loss function is 0.1, and the optimizer is implemented by Torch.optim.SGD algorithm, the batch size parameter of the algorithm is 16, the momentum parameter is 0.9, the weight decay parameter is 0.0005, the initial learning rate is 0.01, and the decay exponent of the learning rate scheduling is 0.9, and the semantic loss is used as a guide to optimize the semantic segmentation model. The "input 3" and the "category prediction image data 1" and "category prediction image data 2" obtained by the semantic segmentation module of this embodiment are used as inputs of the network optimization controller 1, which is transmitted to the network optimization controller 2 according to the input and output semantic loss and optimizes the semantic segmentation module at the same time, and the semantic loss, output KK and the corresponding infrared image and visible light image are input to the network optimization controller 2 and the learnable parameters of the image fusion module are further optimized according to its output, so as to realize the training of a group of image data pairs of the data set CC, and the training results are stored in the semantic segmentation module and the image fusion module. All image pairs of the dataset CC are trained to complete one training of the semantic segmentation module of the present invention. Fig.19 The training process shown in the figure performs Q training on the semantic segmentation module of the present invention to obtain the semantic segmentation model LE of the present embodiment;

[0094] The image fusion network of the present invention is composed of the image fusion module, the semantic segmentation module, the network optimization controller 1 and the network optimization controller 2. Fig.19 The training process shown in FIG. 1 performs M iterations of training on the image fusion network to obtain its network model AA. Finally, the infrared image and visible light image to be processed are Fig. 20 and Fig.21 , the image fusion module of the image fusion network model AA obtained by inputting into this embodiment can output the final fused image therefrom, as shown in Fig. 22 , the obtained fused image has prominent details, comprehensive and complete semantic information, and high image quality.

[0095] Embodiment 2:

[0096] First, construct an image training dataset A, which contains W = W0 ∈ [500, 10000] groups of image data pairs. Each data pair contains an infrared image, a visible light image, and a label image. The first two of them are registered image pairs, and the image format of the latter is JPG format, and the size of the image is MM × NN pixels; the number of iteration training times M = M0 ∈ [1, 100000], P = P0 ∈ [2700, 100000], Q = Q0 ∈ [20000, 200000] in this embodiment; the number K of class labels of the label image is 9 kinds, which are cars, people, bicycles, curves, parking areas, guardrails, hues, and backgrounds, etc. Then, use the Randint method in the Random library of the Python language to randomly crop all the infrared images, visible light images, and label images in the dataset A, and the obtained image size is MM1 * NN1, where MM1 < MM and NN1 < NN. Use the "ToTensor" method in the "Transforms" module of the "Torchvision" library of the PyTorch framework to perform tensor normalization processing on the cropped infrared, visible light images, and label images, convert them into Pytorch tensor form, and normalize their elements to [0, 1] to obtain the image training dataset B, in which the infrared images and visible light images of the data pairs correspond one by one.

[0097] Use the image fusion network of the present invention, as shown in Figure 1-6 , build the network structure of the image fusion module of this embodiment, and its network structure and construction parameters are the same as those described in the technical solution part of the present invention. Use the image fusion method provided by the present invention to construct the image fusion model BE of this embodiment, and the training control flow chart is shown in Fig.19 . Take any group of infrared images and the corresponding visible light images in the dataset B as input 1 and input 2 and input them into the image fusion module of the present invention, and it processes the input images to obtain the fused image, that is, output KK; then update the semantic loss according to the semantic loss weight parameter β = γ × (m - 1) and the semantic loss function, that is, the output K described in the present invention. Then use the image fusion network of the present invention, as shown in Figure 1-18, a network optimization controller 2 is built. Its network structure and construction parameters are the same as those described in the technical solution part of the present invention. The relevant parameters of the loss function are α = α0 ∈ [0.0001, 1000] and γ = γ0 ∈ [1, 10000]. The optimizer inside it adopts the Train.AdamOptimizer algorithm. The batch size parameter of the algorithm is 8, β1 is 0.9, β2 is 0.99, and the constant ε is 1e -8 , the weight decay parameter is 0.0002, and the initial learning rate parameter is 0.001. It optimizes the image fusion module under the guidance of the joint loss. The output KK of this embodiment and the updated semantic loss are used as the input of the network optimization controller 2, and it optimizes the image fusion module according to the input to realize the training of a set of image data pairs of the data set B. According to Fig.19 's process control, all groups of images in the data set B are trained to complete one training of the image fusion module. According to Fig.19 The training process shown is used to train the image fusion module P times to obtain the image fusion model BE of this embodiment.

[0098] Then, the above random cropping operation is performed on all image pairs of the data set A of this embodiment. The obtained image size is M1*N1, where M1 < MM and N1 < NN; the cropped infrared, visible light images and label images are tensor-normalized using the "ToTenBor" method in the "Transforms" module of the "Torchvision" library in the PyTorch framework to convert them into the Pytorch tensor form and normalize their elements to [0, 1] to obtain the image training data set CC. The infrared and visible light images inside it are in one-to-one correspondence, and each pair of infrared and visible light images corresponds to a label image; using the semantic segmentation module of the present invention, see Figure 1-18 , build its network structure. Its network structure and construction parameters are the same as those described in the technical solution part of the present invention. The semantic segmentation model LE of this embodiment is constructed using the image fusion method provided by the present invention. The training control process is shown in Fig.19. Input any group of infrared images and the corresponding visible light images in the data set CC into the obtained image fusion model BE to obtain the output fusion image KK, i.e., output KK; use the PyTorch framework "Torchvision.Transforms" module "RandomHorizontalFlip", "RandomVerticalFlip", and "Resize" methods to synchronously perform random image processing operations such as "image horizontal flip", "image vertical flip", and "size adjustment processing" on the output KK and the corresponding label image to obtain the "input KK" of the semantic segmentation module and the "input 3" of the network optimization controller 1. Then input the obtained "input KK" into the semantic segmentation module of the present invention, see Figure 1-18 , the input KK is processed by the semantic segmentation module in turn to obtain "category prediction image data 1" and "category prediction image data 2"; using the image fusion network provided by the present invention, see Figure 1-18 , build the network optimizer 1 of this embodiment, whose network structure and construction parameters are the same as those described in the technical solution of the present invention, the relevant parameters of the loss function λ∈[0.00001,1.0], and the optimizer is implemented by Torch.optim.SGD algorithm, the batch size parameter of the algorithm is 16, the momentum parameter is 0.9, the weight decay parameter is 0.0005, the initial learning rate is 0.01, and the decay exponent of the learning rate scheduling is 0.9, and the semantic loss is used as a guide to optimize the semantic segmentation model. The "input 3" and the "category prediction image data 1" and "category prediction image data 2" obtained by the semantic segmentation module of this embodiment are used as the input of the network optimization controller 1, which is transmitted to the network optimization controller 2 according to the input and output semantic loss and optimizes the semantic segmentation module at the same time, and the semantic loss, output KK and the corresponding infrared image and visible light image are input to the network optimization controller 2 and the learnable parameters of the image fusion module are further optimized according to its output, so as to realize the training of a group of image data pairs of the data set CC, and the training results are stored in the semantic segmentation module and the image fusion module. All image pairs of the dataset CC are trained to complete one training of the semantic segmentation module of the present invention. Fig.19 The training process shown in the figure performs Q training on the semantic segmentation module of the present invention to obtain the semantic segmentation model LE of the present embodiment;

[0099] The image fusion network of the present invention is composed of the image fusion module, the semantic segmentation module, the network optimization controller 1 and the network optimization controller 2. Fig.19The training process shown performs M iterations of training on the image fusion network to obtain its network model AA. Finally, the infrared image and visible light image to be processed are input into the image fusion module of the image fusion network model AA obtained in this embodiment, and the final fused image can be output therefrom. The obtained fused image has prominent details, comprehensive and complete semantic information, and high image quality.

[0100] The above embodiments only illustrate and explain the implementation methods of the present invention. Any modifications and variations made according to the scope of the claims of the present invention belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A semantic-aware image fusion network, characterized in that: The network is constructed by introducing semantic perception into the fusion process of infrared and visible light images based on deep learning. It can fuse all effective information of infrared and visible light images, and further optimize the semantic information of the fused image by introducing semantic loss therein, so as to obtain a fused image with prominent details and more semantic content; the network is composed of an image fusion module, a semantic segmentation module, a network optimization controller 1 and a network optimization controller 2 and contains three inputs, namely "visible light image", "infrared image" and "label image", which are represented by input 1, input 2 and input 3 respectively; the image fusion module contains two inputs and one output, the two inputs are the input 1 and input 2, and the one output is output KK, which is the fused image obtained by the module by fusing its inputs, and is also the input of the semantic segmentation module and an input of the network optimization controller 2; the semantic segmentation module contains two inputs and two outputs, and the function realized is to perform semantic segmentation on the input fused image, the two inputs are input KK and input R, which correspond to the output KK and the implicit output of the network optimization controller 1 respectively. , the two outputs are output V and output W, which store category prediction image data 1 and category prediction image data 2, respectively, which are inputs of the network optimization controller 1; the network optimization controller 1 contains three inputs and two outputs, the three inputs are input V, input W and input 3, respectively, the input V and input W correspond to the output V and output W, respectively, the two outputs are output K and implicit output R, which store the semantic loss of the fused image and parameter optimization update information for adjusting the parameters of the semantic segmentation module, respectively; the function of the network optimization controller 1 is to calculate the semantic loss of the fused image and optimize the network learnable parameters or model parameters of the semantic segmentation module through its implicit output; the network optimization controller 2 contains four inputs and one implicit output, the four inputs are input KK and input K, which correspond to the output KK and output K, respectively, as well as input 1 and input 2; the function of the network optimization controller 2 is to calculate the loss of the fused image according to its input, and optimize the network learnable parameters or model parameters of the image fusion module using its implicit output; The image fusion module is composed of a color space converter 1, an image encoder 1, an image decoder 1 and a color space converter 2 connected in series in sequence, and the functions implemented by the module are "converting the image to be processed from the RGB color space to the YUV color space", "extracting the deep features of the image", "fusing the extracted features to generate a fused image", and "converting the fused image from the YUV color space to the RGB color space" in sequence; the color space converter 1 has an input and an output, the one input is the input 1, and the output is the input of the image encoder 1, which is called input A; The image encoder 1 has two inputs and one output, the two inputs are the input A and the input 2, and the output is used as the input of the image decoder 1; the image decoder 1 has one input and one output, and the output is used as the input of the color space converter 2; the color space converter 2 has one input and one output, and the output is the output KK; The color space converter 1 is composed of a flattening layer, an extraction layer, a calculation layer, and a reshaping layer connected in series in sequence, and the functions implemented by it are respectively "flattening its input into a two-dimensional array, in which each row represents a pixel point, and each column represents the red (R), green (G), and blue (B) channel values ​​of the corresponding pixel", "extracting the R, G, and B channel values ​​of the pixel from the two-dimensional array", "calculating the Y, U, and V channel values ​​from the extracted R, G, and B channel values", and "performing tensor connection on its input and adjusting the dimension of the tensor". The image encoder 1 is composed of a convolution layer A, an LReLU activation function layer, a convolution layer AA, an LReLU activation function layer, a residual block 1, a residual block 2, a residual block 3, a residual block 4, a residual block 5, a residual block 6 and a splicing layer, and its functions are realized by the following process: Process 1: Input the input A to the convolution layer A, LReLU activation function layer, residual block 1, residual block 3 and residual block 5 connected in series in sequence, and obtain the output BB1 after processing; Process 2: The input 2 is processed by the concatenated convolution layer AA, the LReLU activation function layer, the residual block 2, the residual block 4, and the residual block 6 to obtain the output BB2; Process 3: The obtained output BB1 and output BB2 are input to the splicing layer, and its output is the output of the image encoder 1; the function of the convolution layer A and the convolution layer AA is to extract the feature map of the input, and the role of the LReLU activation function layer is to enhance the representation ability of the neural network and solve the gradient disappearance problem; the function of the splicing layer is to splice the input feature map in the channel dimension; the residual block 1 is composed of a convolution layer B1, an LReLU activation function layer, a splicing layer 1, a convolution layer B2, an LReLU activation function layer, a splicing layer 2, a convolution layer B3, a calculation layer, a gradient calculation layer, a convolution layer B4 and an LReLU activation function layer, which contains one input and one output, divided into The convolution layer B1 comprises an input and an output, the input is the input C, and the output is used as the input of the LReLU activation function layer after the convolution layer B1; the LReLU activation function layer after the convolution layer B1 comprises an input and an output, and the output is an input of the convolution layer 1; the convolution layer 1 comprises two inputs and one output, the two inputs are the input C and the output of the LReLU activation function layer after the convolution layer B1, and the one output is used as the input of the convolution layer B2; the convolution layer B2 comprises an input and an output, and the output is used as the LReLU activation function layer after the convolution layer B2 The LReLU activation function layer after the convolution layer B2 contains an input and an output, and the output is an input of the splicing layer 2; the splicing layer 2 contains two inputs and one output, and its other input is the output of the splicing layer 1, and the output is used as the input of the convolution layer B3; the convolution layer B3 includes an input and an output, and the output is used as an input of the operation layer; the gradient operation layer includes an input and an output, and the input is the input C, and the output is used as the input of the convolution layer B4; the convolution layer B4 includes an input and an output, and the output is used as another input of the operation layer, and the operation layer includes two An input and an output, the output is processed by an LReLU activation function layer to obtain the output C; the convolution layers B1, B2, B3 and B4 are used to extract features from the input feature map, the LReLU activation function layer is used to enhance the representation ability of the neural network and solve the gradient disappearance problem, the function of the splicing layer is to splice the input feature map in the channel dimension, the function of the gradient operation layer is to extract the edge information of the feature map, and the function of the operation layer is to add the input feature map element by element; the residual block 2, residual block 3, residual block 4, residual block 5, residual block 6 and the residual block 1 have the same structure, and are all used to further extract fine-grained information and high-level semantic features of the feature map; The image decoder 1 is composed of a convolution layer C1, a normalization layer, an LReLU activation function layer, a convolution layer C2, a normalization layer, an LReLU activation function layer, a convolution layer C3, a normalization layer, an LReLU activation function layer, a convolution layer C4, a normalization layer, an LReLU activation function layer, a convolution layer C5, a normalization layer, and a Tanh activation function layer connected in series in sequence; the convolution layers C1, C2, C3, C4 and C5 have the same structure and function, and are used to extract features from the input feature map; the function of the normalization layer is to standardize all pixels of the feature map to make the network more stable and efficient; the function of the LReLU activation function layer is to introduce nonlinear transformation while avoiding the gradient disappearance problem; the function of the Tanh activation function layer is to map the output feature map to the range of [-1, 1]; The color space converter 2 has the same structure as the color space converter 1, and is composed of a flattening layer, an extraction layer, a calculation layer, and a reshaping layer connected in series in sequence, and its functions are respectively "flattening the image into a two-dimensional array, each row of the two-dimensional array represents a pixel point, and each column represents the value of the Y, U, and V channels", "extracting the Y, U, and V channel values ​​from the two-dimensional array respectively", "calculating the R, G, and B channel values ​​through the extracted Y, U, and V channel values", and "connecting tensors and adjusting the tensor dimensions". The semantic segmentation module is composed of an image encoder 2 and an image decoder 2, and its functions are respectively "compressing and extracting the image features of its input" and "generating the classification of each pixel according to the spatial resolution of its input recovery data, thereby realizing image segmentation"; the image encoder 2 contains one input and three outputs, the input is the input KK, and the three outputs are output X, output Y and output Z, and their physical meanings are respectively "feature map containing basic edge and texture information", "feature map containing more complex local structure and texture information" and "feature map containing high-level semantic information and global context", which are all used as inputs of the image decoder 2; the image decoder 2 includes three inputs and two outputs, the two outputs are the output V and output W, and the category prediction image data 1 and the category prediction image data 2 are stored therein, which are both used as inputs of the network optimization controller 1; The image encoder 2 is composed of a convolution layer D1, a normalization layer, a ReLU activation function layer, a pooling layer, two series-connected residual blocks A, two series-connected residual blocks B, two series-connected residual blocks C and two series-connected residual blocks D in series in sequence; the two series-connected residual blocks B include an input and an output, the input is the output of the two series-connected residual blocks A, and the one output is the output X, which is simultaneously used as the input of the image decoder 2 and the two series-connected residual blocks C; the two series-connected residual blocks C include an input and an output, the The input is the output X of the two series-connected residual blocks B, and the output is the output Y, which is simultaneously used as the input of the image decoder 2 and the two series-connected residual blocks D; the two series-connected residual blocks D include an input and an output Z, the input is the output Y of the two series-connected residual blocks C, and the one output is the output Z, which is used as the input of the convolution layer E1 of the image decoder 2; the convolution layer D1, the pooling layer, the two series-connected residual blocks A, the two series-connected residual blocks B, the two series-connected residual blocks C, and the two series-connected residual blocks D work as follows: The function of the normalization layer is to standardize all pixels of the input feature map; the function of the ReLU activation function layer is to introduce nonlinear transformation and avoid the gradient disappearance problem; the residual block A of the image encoder 2 is composed of a convolution layer D2, a normalization layer, a ReLU activation function layer, a convolution layer D3, a normalization layer, a calculation layer and a ReLU activation function layer connected in series; the convolution layer D2, the normalization layer, the RELU activation function layer, the convolution layer D3, the normalization layer are connected in series in sequence and each of them contains a An input and an output, the output of the normalized layer after the convolution layer D3 is used as an input of the connection layer; the operation layer includes two inputs and one output, the two inputs are the input of the residual block A and the output of the normalized layer after the convolution layer D3, and the one output is used as the input of the ReLU activation function layer after the operation layer; the ReLU activation function layer after the operation layer contains one input and one output, and the one output is the output of the residual block A; the function of the convolution layers D2 and D3 is to extract the features of the input feature map; The function of the normalization layer is to standardize all pixel data of the feature map input thereto; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient vanishing problem; the function of the operation layer is to perform a sum operation on its input; the structures of the residual block B, residual block C, and residual block D are exactly the same and each contains one input and one output, and each is composed of a convolution layer D4, a normalization layer, a ReLU activation function layer, a convolution layer D5, a normalization layer, a convolution layer D6, a normalization layer, an operation layer, and a ReLU activation function layer, and its functions are realized by the following process: Process 1: The input is processed by the convolution layer D4, normalization layer, ReLU activation function layer, convolution layer D5, and normalization layer in series to obtain the output NN1; Process 2: The input is processed by the convolutional layer D6 and the normalization layer in series to obtain the output NN2; Process 3: The obtained output NN1 and output NN2 are input to the operation layer, and the output of the operation layer is obtained after processing, and then the output of the operation layer is input to the ReLU activation function layer to obtain the output of the residual block; the convolution layer D4, the convolution layer D5, and the convolution layer D6 are used to extract the features of the input feature map; The function of the normalization layer is to standardize all pixels of the feature map output by the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the operation layer is to perform element-wise summation operation on its input; The image decoder 2 is composed of a convolution layer E1, a normalization layer, a ReLU activation function layer, a bilateral attention decoder 1, an output block 1, a convolution layer E2, a normalization layer, a ReLU activation function layer, a bilateral attention decoder 2 and an output block 2; the convolution layer E1, the normalization layer, and the ReLU activation function layer are connected in series in sequence, and each of them contains an input and an output; the input of the convolution layer E1 is the input Z, which corresponds to the output Z, and the output H of the ReLU activation function layer is used as an input of the bilateral attention decoder 1; the bilateral attention decoder 1 includes two inputs and one output, and the two inputs are respectively Input H and the input Y, the input H corresponds to the output H of the ReLU activation function layer after the convolution layer E1, the input Y corresponds to the output Y of the residual block C, and the one output is simultaneously used as the input of the convolution layer E2 and the output block 1; the output block 1 includes an input and an output, the one input is the output of the bilateral attention decoder 1, and the one output is the output V of the semantic segmentation module; the convolution layer E2, the normalization layer, and the ReLU activation function layer are connected in series in sequence and each of them contains an input and an output, and the input of the convolution layer E2 is the bilateral attention decoder 1 The output of the ReLU activation function layer, the output HH of the ReLU activation function layer is an input of the bilateral attention decoder 2; the bilateral attention decoder 2 includes two inputs and one output, the two inputs are input HH and the input X, which correspond to the output HH of the ReLU activation function layer after the convolution layer E2 and the output X, respectively, and the one output is used as the input of the output block 2; the output block 2 includes one input and one output, the one input is the output of the bilateral attention decoder 2, and the one output is the output W of the semantic segmentation module; the functions of the convolution layer E1 and the convolution layer E2 To extract the features of the input feature map; the function of the normalization layer is to standardize all pixels of the feature map output by the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the functions of the output block 1 and the output block 2 are to convert the final feature map of the network into the classification result of each pixel; the bilateral attention decoder 1 and the bilateral attention decoder 2 have the same structure and function, which are both composed of a spatial attention branch block, a channel attention branch block, a convolution layer E3, an upsampling layer and a pooling fusion block, and the low-level feature map and the high-level feature map input are fused through the following process: Process 1: The input Y or X is processed by the spatial attention branch and convolution layer E3 in series to obtain the output B; Process 2: The input H or HH is processed by the channel attention branch layer and the upsampling layer connected in series to obtain the output D; Process 3: The obtained output B and output D are input to the pooling fusion module, and the output of the bilateral attention decoder is obtained after processing; the function of the spatial attention branch block is to refine the input low-level feature map feature map and extract more spatial information; the function of the channel attention branch block is to refine the input high-level feature map and extract more semantic information; the function of the convolution layer E3 is to extract the features of the input low-level feature map; the function of the upsampling layer is to restore the resolution of the input high-level feature map; the function of the pooling fusion block is to fuse the input high- and low-level feature maps to generate a fusion feature map that accurately combines semantic information and spatial information; the spatial attention branch block is composed of the convolution layers AA1, The network is composed of a normalization layer, a ReLU activation function layer, a maximum pooling layer, a mean pooling layer, a splicing layer, a convolution layer AA2, a normalization layer, a Sigmoid activation function layer and an operation layer; the convolution layer AA1, the normalization layer, and the ReLU activation function layer are connected in series in sequence and each of them has an input and an output; the input of the convolution layer AA1 is the input of the spatial attention branch block, and the output of the ReLU activation function layer is also used as the input of the maximum pooling layer and the mean pooling layer; the mean pooling layer has an input and an output, the one input is the output of the ReLU activation function layer, and the one output is the input of the splicing layer; the mean pooling layer has an input and an output, The one input is the output of the ReLU activation function layer, and the one output is another input of the splicing layer; the splicing layer contains two inputs and one output, and the one output is used as the input of the convolution layer AA2; the convolution layer AA2, the normalization layer, and the Sigmoid activation function layer are connected in series in sequence and each contains one input and one output, the input of the convolution layer AA2 is the output of the splicing layer, and the output of the Sigmoid activation function layer is an input of the operation layer; the operation layer includes two inputs and one output, and the other input corresponds to the input Y or the input X, and the one output is the output of the spatial attention branch block; the convolution layer AA1 , the function of the convolution layer AA2 is to extract features from the input feature map; the function of the maximum pooling layer is to extract the most significant features of the input feature map; the function of the mean pooling layer is to extract the average representation of the global features of the input feature map; the function of the splicing layer is to splice its input in the channel dimension; the function of the normalization layer is to standardize all pixels of the output feature map of the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the Sigmoid activation function layer is to generate a spatial attention map containing attention weights by introducing nonlinear transformation; the function of the operation layer is to perform multiplication operation on its input;The channel attention branch block is composed of a global average pooling layer, a convolution layer AA3, a normalization layer, a Sigmoid activation function layer and an operation layer. The global average pooling layer, the convolution layer AA3, the normalization layer and the Sigmoid activation function layer are connected in series in sequence and each of them contains an input and an output. The input of the global average pooling layer is the input of the channel attention branch block, and the output of the Sigmoid activation function layer is an input of the operation layer; the operation layer contains two inputs and one output, and the two inputs are the outputs of the Sigmoid activation function layer and the input of the channel attention branch block, respectively, and the one output is the output of the channel attention branch block; the function of the global average pooling layer is to convert the input feature map The information is compressed into a global feature description; the function of the Sigmoid activation function layer is to generate a channel attention map containing attention weights by introducing nonlinear transformations; the function of the operation layer is to perform multiplication operations on its input and adjust the input feature map; the pooling fusion block is composed of an average pooling layer, a splicing layer, a convolution layer AA4, a normalization layer and a ReLU activation function layer connected in series in sequence, the average pooling layer contains one input and one output, the input is input D, which corresponds to the output D, and the one output is used as an input of the splicing layer; the splicing layer contains two inputs and one output, the two inputs are the output of the average pooling layer and input B, and the input B corresponds to the output B of the convolution layer AA; The convolution layer AA4, normalization layer and ReLU activation function layer are connected in series in sequence and each of them contains an input and an output. The input of the convolution layer AA4 is the output of the splicing layer, and the output of the ReLU activation function layer is the output of the pooling fusion block; the function of the average pooling layer is to smooth the input feature map; the function of the splicing layer is to splice its input in the channel dimension; the function of the convolution layer AA4 is to reconstruct the input feature map; the function of the normalization layer is to standardize all pixels of the output feature map of the input convolution layer; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the The output block 1 and the output block 2 have the same structure, and are both composed of a convolution layer AA5, a normalization layer, a ReLU activation function layer, a convolution layer AA6 and an upsampling layer connected in series in sequence, and each of them contains an input and an output; the function of the convolution layer AA5 is to extract the features of the input feature map; the function of the normalization layer is to standardize all pixels of the input convolution layer output feature map; the function of the ReLU activation function layer is to introduce nonlinear transformation to avoid the gradient disappearance problem; the function of the convolution layer AA6 is to extract the features of the input feature map and generate a category prediction map; the function of the upsampling layer is to align the feature map output by it with the input image data in space; The network optimization controller 1 is composed of a semantic loss function and an optimizer connected in series; the semantic loss function has three inputs and one output, and the one output is output K, which serves as the input of the optimizer and the output K of the network optimization controller 1; the optimizer has one input and one output, and the output is the implicit output R of the network optimization controller 1; the semantic loss function L semantic for THE semantic =L main +λL aux Among them, L main is the main semantic loss function, which is the cross entropy of the main segmentation result "output W" of the semantic segmentation module and the semantic segmentation label "input 3", L aux is the auxiliary semantic loss function, which is the cross entropy of the auxiliary segmentation result "output V" of the semantic segmentation module and the semantic segmentation label "input 3", λ is the weight constant for balancing the main semantic loss and auxiliary semantics, L S ,I S ,I Sa are the input 3, the "output W" and "output V" of the semantic segmentation module, H, W and C are the height, width and number of channels of the input data respectively; the output K is the semantic loss function L semantic The output of reflects the richness of the semantic information contained in the fused image; the optimizer optimizes the learnable parameters of the semantic segmentation module including the network structure, weights and bias parameters according to its input and finally obtains the network model, and the optimization rule adopted is to make the gradients of all neurons of the semantic segmentation module converge as a whole; The network optimization controller 2 is composed of a joint loss function and an optimizer in series, wherein the joint loss function has four inputs and one output, and the optimizer has one input and one output; the joint loss function L joint It is obtained through the following process: Process 1: Constructing content loss function L content , Among them, I f is the output KK of the image fusion module, I ir ,I vi are the input 1 and input 2, H and W are the height and width of the image, respectively, ||I f -max(I ir , I vi )||1 is the norm operation, representing the intensity distribution loss of the fused image. It is the Sobel gradient operator that operates and represents the fine-grained texture information of the image. is a norm operation, characterizing the texture distribution loss of the fused image, and α is a weight constant for balancing the intensity distribution loss and the texture distribution loss; Process 2: Constructing the joint loss function L joint , L joint =L content +βL semantic , Among them, L content , L semantic are the content loss and semantic loss respectively, β=γ×(m-1) is the weight parameter for measuring the contribution of semantic loss, m is the number of iterative training of the image fusion network, γ is the adjustment coefficient for adjusting the contribution of the semantic loss, which is an integer; the optimizer optimizes the learnable parameters of the image fusion module including network structure, weights, biases and other parameters according to its input and finally obtains the network model, and the optimization rule adopted is to make the gradients of all neurons of the image fusion module descend and converge as a whole.

2. The semantic-aware image fusion network according to claim 1, characterized in that: The network is built based on the PyTorch deep learning framework; the weight parameters of the loss function of the network optimization controller 1 and the network optimization controller 2 are λ∈[0.00001, 1.0], α∈[0.0001, 1000], γ∈[1, 10000], preferably, λ=0.1, ɑ=10, γ=1; the optimizer of the network optimization controller 1 is the SGD optimizer, which is implemented using the Torch.Optim.SGD algorithm, and its input parameters include batch size, momentum, weight decay, initial learning rate, and learning rate scheduling decay exponent, preferably , the batch size is 16, the momentum is 0.9, the weight decay is 0.0005, the initial learning rate is 0.01, and the learning rate scheduling decay exponent is 0.

9. It optimizes the semantic segmentation module guided by semantic loss; the optimizer of the network optimization controller 2 is the Adam optimizer, which is implemented using the Torch.Optim.Adam algorithm, and its input parameters include batch size, first-order moment estimation decay rate β1, second-order moment estimation decay rate β2, constant ε, weight decay and initial learning rate. Preferably, the batch size is 8, β1=0.9, β2=0.99, and the constant ε=1e -8 , the weight decay is 0.0002, the initial learning rate is 0.001, and the image fusion module is optimized with the joint loss as the guide; The dimension splicing operation of the splicing layer of the image fusion module adopts the Torch.cat method of PyTorch to splice along the second dimension, that is, the column dimension. The flattening layer is implemented by the Torch.transpose and Torch.reshape algorithms, and the reshaping layer is implemented by the Torch.unsqueeze, Torch.cat and Torch.reshape algorithms. The color space converter 1, color space The calculation layer of the interconverter 2 uses the relations Y=0.299×R+0.5287×G+0.114×B, U=-0.147×R-0.289×G+0.436×B, V=0.615×R-0.515×G-0.1×B and R=Y+1.403×(V-0.5), G=Y-0.344×(U-0.5)-0.714×(V-0.5), B=Y+1.773×(U-0.5) for color conversion; The summation operation layer and the multiplication operation layer are implemented by Torch.add algorithm and Torch.mul algorithm respectively; the pooling layer, average pooling layer, global average pooling layer, maximum pooling layer and mean pooling layer are implemented by Torch.nn.maxpool2d, Torch.nn.functional.avg_pool2d, Torch.nn.avg_pool2d, Torch.max algorithm and Torch.mean algorithm respectively; the upsampling layer is implemented by Torch.nn.functional.interpolate algorithm, and the sampling mode is "bilinear"; The convolutional layers include convolutional layer A, convolutional layer AA, convolutional layer B1, convolutional layer B2, convolutional layer B3, convolutional layer B4, convolutional layer C1, convolutional layer C2, convolutional layer C3, convolutional layer C4, convolutional layer D1, convolutional layer D2, convolutional layer D3, convolutional layer D4, convolutional layer D5, convolutional layer D6, convolutional layer E1, convolutional layer E2, convolutional layer E3, convolutional layer AA1, convolutional layer AA2, convolutional layer AA3, convolutional layer AA4, convolutional layer AA5, and convolutional layer AA6. The initial value of the weight of the convolution kernel is a normal distribution with a mean of 0, and the generation of the initial value is The Torch.nn.init.kaiming_normal_ algorithm is used for the synthesis; the convolution operation of the convolution layer is implemented by the Torch.nn.Conv2d algorithm; the construction parameters of the convolution layer A, convolution layer AA, convolution layer B1, convolution layer B2, convolution layer C1, convolution layer C2, and convolution layer C3 of the image fusion module are: the convolution kernel size is 3*3, the convolution kernel step size is 1, and the convolution kernel padding is 1; the construction parameters of the convolution layer B3, convolution layer B4, and convolution layer C4 of the image fusion module are: the convolution kernel size is The convolution kernel size is 1*1, the convolution kernel step size is 1, and the convolution kernel padding is 0; the construction parameters of the convolution layer D2, convolution layer D3, convolution layer D4, convolution layer D5, convolution layer E1, convolution layer E2, convolution layer E3, convolution layer AA2, and convolution layer AA5 are: the convolution kernel size is 3*3, the convolution kernel step size is 1, and the convolution kernel padding is 1; the construction parameters of the convolution layer D6, convolution layer AA1, convolution layer AA3, convolution layer AA4, and convolution layer AA6 of the image fusion module are: the convolution kernel size is 1*1, the convolution kernel step size is 1, and the convolution kernel padding is 0 ; The construction parameters of the convolution layer D1 are: the convolution kernel size is 7*7, the convolution kernel step size is 2, and the convolution kernel padding is 3; the normalization layer is implemented using the Torch.nn.BatchNorm2d algorithm, the Leakyrelu activation function layer is implemented using the Torch.nn.leaky_relu algorithm, the Sigmoid activation function layer is implemented using the Torch.nn.Sigmoid algorithm, and the Tanh activation function layer is implemented using the Torch.nn.Tanh algorithm.

3. A semantic-aware image fusion method, characterized in that: The method is based on the image fusion network described in claim 1 or claim 2, and comprises the following steps: Step 1: Construct an image training dataset A for training. The dataset is composed of a series of image data pairs consisting of infrared images, visible light images, and label images. The infrared images and visible light images are registered images collected for the same scene, and the label images are corresponding semantic label images representing the category to which their content objects belong. Each pixel in the label images is assigned a category label, which is a semantic label used to represent the category information of the pixel. The size of the image is MM×NN pixels. Step 2: Perform random cropping on all image data pairs in the image training dataset A obtained in step 1. The size of the cropped image is smaller than the size of the original image, which is MM1×NN1 pixels. Perform tensor normalization on the obtained image, convert the image into a Pytorch tensor form and normalize its elements to [0, 1] to obtain the image training dataset. Step 3: The image training data set obtained in step 2 is called the image data training set B, and any set of image data pairs is taken out therefrom, and the visible light image and the infrared image therein are respectively input as the input 1 and the input 2 to the image fusion module described in claims 1-2 to obtain its output, namely the output KK; then the obtained output KK is input to the semantic segmentation module described in claims 1-2 to obtain its output, namely the output V and the output W; then the obtained output V and the output W and the label image in the image data pair are input as the input 3 to the network controller 1 described in claims 1-2 to obtain its output, namely the output K; Step 4: Update the output K described in step 3 according to the formula β = γ × (m-1), that is, the semantic loss L described in claims 1-2 semantic The weight parameter β; Step 5: The output KK obtained in step 3 is used as the input KK described in claims 1-2, the output K is used as the input K described in claims 1-2, and the input 1 and input 2 described in step 3 are input to the network optimization controller 2 described in claims 1-2, and the image fusion module described in claims 1-2 is optimized after being processed by the controller and according to its output, so as to realize the training of a group of image data pairs of the image training data set B described in step 2; Step 6: Continuously repeat steps 3-5 to implement training of all image data pairs of the image training data set B in step 2, and at this time, one training of the image fusion module in claim 1 is completed, and the training result is stored in the image fusion module; Step 7: Repeat step 6 continuously, train the image fusion module P times, save the training results in the image fusion module, and obtain the network training model of the image fusion module, which is called image fusion model BE; Step 8: Repeat step 2 to obtain an image training data set, input all the data in the data set into the image fusion model BE obtained in step 7, and obtain the fused image KK outputted by the model, i.e., output KK; then form a data pair with the corresponding label image, infrared image, and visible light image to form an image training data set C; Step 9: Take any set of data pairs of "output KK, label image, infrared image, visible light image" from the image training data set C obtained in step 8, and then synchronously perform random image processing operations of "image horizontal flip", "image vertical flip", and "size adjustment processing", and resize the processed image. The pixel size of the adjusted image is smaller than the size of the original image, and then the adjusted image is tensor normalized to convert it into a tensor mode and normalize its elements to [0, 1], and the processed image data are obtained, which are output KKc, output BQc, output HWc, and output KJc, respectively, which correspond to the input KK, input 3, input 1, and input 2 described in claims 1-2, respectively; Step 10: Input the "output KKc" obtained in step 9 to the semantic segmentation module described in claims 1-2 to obtain its output, namely the output V and output W, which respectively represent the "category prediction image data 1" and "category prediction image data 2" contained in its input, which respectively correspond to the input V and input W described in claims 1-2; Step 11: Input the output V and output W obtained in step 10 and the output BQc obtained in step 9 to the network optimization controller 1 described in claims 1-2 to obtain the semantic loss of its output, that is, the output K or input K described in claims 1-2, and optimize the semantic segmentation module described in claims 1-2 according to its implicit output R. At the same time, input the semantic loss and the output KKc obtained in step 9 as well as the output HWc and the output KJc to the network optimization controller 2 and further optimize the learnable parameters of the image fusion module according to its output, so as to realize the training of a group of image data pairs of the image data training set C described in step 8, and the training results are stored in the semantic segmentation module and the image fusion module; Step 12: Repeat steps 9-11 continuously to implement the training of all image data pairs in the image data training set C described in step 8. At this time, the training of the semantic segmentation module described in claims 1-2 is completed once, and the training results are stored in the semantic segmentation module; Step 13: Repeat step 12 continuously, train the semantic segmentation module Q times, save the training results in the semantic segmentation module, and obtain a network training model of the semantic segmentation module, which is called the semantic segmentation model LE; at this time, the image fusion network of claim 1-2 is trained once, and the training results are saved in the image fusion network; Step 14: Continuously repeat step 3 and steps 5-13, and train the image fusion network described in claims 1-2 M times to obtain a network training model of the image fusion network described in claims 1-2, which is called image fusion network model AA; Step 15: Input the infrared image and visible light image to be processed as the input 1 and input 2 described in claims 1-2 to the image fusion module of the image fusion network model AA obtained in step 14, so that it can output a fused image with prominent details and rich semantic information after image fusion processing of the input.

4. The image fusion method based on semantic perception according to claim 3, characterized in that: The format of the images in the image training data set A described in step 1 is "JPG" or "PNG", the size of the image is MM×NN pixels, MM=480, NN=640, the number of image data pairs contained is W=W0∈[500,10000], and the number of category labels is K=K0∈[2,10000], which are cars, people, bicycles, curves, parking areas, guardrails, tones and backgrounds, etc., preferably K=9; the "image horizontal flip", "image vertical flip", "size adjustment processing" and "tensor normalization processing" described in step 9 are respectively implemented by the "RandomHorizontalFlip", "RandomVerticalFlip", "Resize" and "ToTensor" methods of the "Transforms" module of the "Torchvision" library of the PyTorch deep learning framework; The resizing process described in step 9 is to adjust the size of the image to no more than MM×NN pixels; the random cropping process described in step 2 is implemented by the randint method of the random library of the Python language, and the "tensor normalization process" is implemented by the Torchvision.transforms.toTensor method of PyTorch, and the MM1=340, NN1=550; the number of training times P of the image fusion module described in step 7 is P=P0∈[2700,100000]; the number of training times Q of the semantic segmentation module described in step 13 is Q=Q0∈[20000,200000]; the number of training times M of the image fusion network described in step 14 is M=M0∈[1,100000]; the weight parameters α=10, λ=0.1, γ=1 of the loss function of the network optimization controller 2 described in step 5 and the network optimization controller 1 described in step 11.

Images