Multi-task image processing method, system and equipment

Abstract

The invention discloses a multi-task image processing method, system and device, and relates to the technical field of image processing, and the method comprises the following steps: constructing a multi-task network model; inputting the image data set into a multi-task network model to respectively obtain an infrared enhanced image, an infrared super-resolution image, an infrared and visible light image fusion image and an infrared image colorization image; and calculating a loss function between different images output by the multi-task network model and the infrared image and the visible light image in the input image data set, and carrying out loop training on the multi-task network model. According to the technical scheme, through unified training of the multi-task model, local details are effectively extracted, global consistency is kept, the processing efficiency and the image quality are greatly improved, redundant steps in the processing process are reduced, and the robustness and adaptability of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-task image processing method, system and device. Background Art

[0002] Multi-task image processing technology has been widely studied and applied in recent years. Its core idea is to process multiple related tasks simultaneously by sharing the feature extraction layer in the network model, thereby improving computational efficiency and reducing resource consumption. However, traditional multi-task processing methods usually face competition and conflict problems between tasks, that is, different tasks may have different requirements for feature extraction, resulting in shared features that are difficult to take into account the needs of all tasks, thus affecting the overall processing effect. In addition, there is also a global consistency problem in feature extraction in multi-task processing, that is, the feature expression of each task needs to maintain a certain consistency in the global scope to ensure information complementarity and collaborative processing between tasks.

[0003] Current research focuses on how to better achieve the distinction of fine-grained features and maintain global consistency in feature extraction. Some methods try to regulate the mutual influence between different tasks by introducing specific loss functions. Based on this, this application proposes a multi-task image processing method that combines fine-grained feature extraction and global consistency loss, which can share efficient and accurate feature representations between tasks and ensure the consistency of global information, thereby improving the comprehensive performance of multi-task processing, and is suitable for multiple fields such as target detection, image classification, and image segmentation. Summary of the invention

[0004] The technical solution of the present invention to solve the above technical problem is to provide a multi-task image processing method, comprising the following steps:

[0005] Construct a multi-task network model, including an encoder and an image enhancement decoder, an image super-resolution decoder, an image fusion decoder, and an image colorization enhancement decoder; the encoder includes a multi-granularity heterogeneous convolution module and an adaptive fine structure enhancement module;

[0006] The image dataset is input into the multi-task network model to obtain infrared enhanced images, infrared super-resolution images, infrared and visible light image fusion images, and infrared image colorization images.

[0007] Calculate the loss function between different images output by the multi-task network model and the infrared image and visible light image in the input image dataset, perform cyclic training on the multi-task network model, and calculate the loss function value through multiple training iterations until the number of training times reaches the set threshold or the value of the loss function reaches the set range;

[0008] Among them, the combination of detail-aware global consistency loss, L1 loss and structural similarity loss is the loss function, and the definition of detail-aware global consistency loss, L1 loss and structural similarity loss function is:

[0009]

[0010] in, represents the detail-aware global consistency loss function, represents the L1 loss function, represents the structural similarity loss function, represents the local detail preservation loss, represents the global consistency loss, represents the detail consistency loss, I out Represents the output images of different decoders, I gt represents the original input image, || ||1 represents the calculation of L1 norm;

[0011] The local detail preservation loss, global consistency loss and detail consistency loss functions are defined as:

[0012]

[0013] in, represents the Sobel gradient operator, H represents the histogram function of the image, N is the number of gray levels of the histogram, j represents the gray level, and patch i represents the i-th local region of the image, M is the number of local regions, and Autocorr represents the autocorrelation function;

[0014] The total loss function is defined as:

[0015]

[0016] Among them, α, β, and γ are weight parameters designed to balance the contribution of each loss term; the weight update method is:

[0017] Calculate the loss change rate,

[0018]

[0019] in, Represents the derivative of each loss with respect to time (number of training steps), represents the calculation of derivatives, and t represents time (number of training steps);

[0020] Adaptive weight adjustment, adjust the weight based on the loss change rate, the weight update formula is as follows:

[0021]

[0022] Among them, η is the learning rate hyperparameter, which is used to control the step size of weight adjustment. is the normalization factor of the total loss change rate, which is the sum of the change rates of all loss items. t+1 represents the next moment.

[0023] Normalize weights, normalize the weights after each update:

[0024]

[0025] Furthermore, the multi-granularity heterogeneous convolution module is composed of convolution block 1-1, convolution block 1-2, convolution block 1-3, convolution block 2-1, convolution block 2-2, convolution block 2-3, convolution block 3-1, convolution block 3-2, and convolution block 3-3. The convolution kernel size of convolution block 1-1, convolution block 2-1, and convolution block 3-1 is n1×n1, and HardSwish activation function is used. The convolution kernel size of convolution block 1-2, convolution block 2-2, and convolution block 3-2 is n2×n2, and S-type activation function is used. The convolution kernel size of convolution block 1-3, convolution block 2-3, and convolution block 3-3 is n3×n3, and S-type activation function is used.

[0026] The adaptive fine structure enhancement module consists of detail enhancement block 1, detail enhancement block 2, detail enhancement block 3 and detail enhancement block 4. Detail enhancement block 1 and detail enhancement block 2 receive feature information output by the multi-granularity heterogeneous convolution module, cross-splice the outputs of detail enhancement block 1 and detail enhancement block 2, and output them to detail enhancement block 3 and detail enhancement block 4. The size of the feature map is consistent with the size of the input image.

[0027] Furthermore, the image enhancement decoder consists of 2 convolutional layers, 2 activation functions and 1 Gabor filter;

[0028] The image super-resolution decoder consists of 2 convolutional layers, 3 activation functions, pixel reorganization and 1 Gabor filter;

[0029] The image fusion decoder consists of 3 convolutional layers and 3 activation functions;

[0030] The image colorization decoder consists of 6 convolutional layers, 4 dilated convolutional layers, 8 activation functions and 5 splicing blocks.

[0031] Furthermore, the infrared images and visible light images in the image dataset come from five datasets: CVC, MSRS, M3FD, KAIST and Urban100, and 15,000 pairs of high-quality infrared images and visible light images are selected for multi-task network training.

[0032] In order to solve the above technical problems, the present application also proposes a multi-task image processing system for implementing the above multi-task image processing method, comprising:

[0033] An image acquisition module, used to acquire training data; the training data are infrared images and visible light images;

[0034] A feature extraction module is used to perform feature extraction on each image in the training data using an encoder to obtain a feature map set for each image; the feature map set includes multiple feature maps of different sizes;

[0035] An infrared image enhancement module, used for enhancing a feature map of a specified size in a feature map set of each of the images to obtain an infrared enhanced image;

[0036] An infrared image super-resolution module, used to perform super-resolution on a feature map of a specified size in a feature map set of each image to obtain an infrared super-resolution image;

[0037] An infrared and visible light image fusion module, used for performing infrared and visible light image fusion on a feature map of a specified size in a feature map set of each image to obtain an infrared and visible light image fusion image;

[0038] An infrared image colorization module is used to perform infrared image colorization on a feature map of a specified size in a feature map set of each image to obtain a colored infrared image;

[0039] The model training module is used to perform multi-task cyclic supervised model training on the infrared enhanced image, infrared super-resolution image, infrared and visible light image fusion image and infrared image colorization image obtained by each module to obtain a multi-task image processing model.

[0040] To solve the above technical problems, the present application also proposes a multi-task image processing device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task image processing method as described above.

[0041] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0042] 1. Multi-task collaborative processing to improve system efficiency: The multi-task image processing method proposed in the present invention can simultaneously perform multiple processing tasks such as image enhancement, image fusion, image super-resolution and image colorization. Under the framework of multi-task processing, the fine-grained image features extracted by the encoder are shared between tasks, which reduces repeated calculations and significantly improves the efficiency of the overall processing system. At the same time, the global consistency loss function ensures the collaborative optimization between tasks, and ensures the consistency of the visual quality of the processed image and the linkage between tasks.

[0043] 2. Fine-grained feature extraction and enhanced detail performance: By extracting fine-grained features in the image through the encoder, the present invention can capture rich detail information in infrared and visible light images. Whether in image enhancement or super-resolution tasks, the accurate extraction of local details can significantly improve the detail fidelity of the output image, which is suitable for high-demand image processing scenarios, such as low-light and noisy environments.

[0044] 3. Global consistency optimization to ensure visual quality: This invention effectively solves the contradiction between local feature optimization and global consistency that may occur in the multi-task processing process through the independently designed global consistency loss function. Through the combination of detail-aware global consistency loss, L1 loss, and structural similarity loss, it not only ensures the accurate preservation of image details, but also maintains the visual quality and consistency of the overall image, so that high-quality processing results can be obtained in various tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0046] Figure 1 is a flowchart of the steps of the multi-task image processing method of the present invention;

[0047] Figure 2 It is a structural diagram of the multi-task network model of the present invention;

[0048] Figure 3 is a structural diagram of the encoder network of the present invention;

[0049] Figure 4 It is a structural diagram of each convolution block in the convolution block 1-1 to the convolution block 3-3 in the multi-granularity heterogeneous convolution module of the present invention;

[0050] Figure 5A structural diagram of a detail enhancement block in an encoder network of the present invention;

[0051] Figure 6 It is a structural diagram of the multi-layer perception module in the detail enhancement block of the present invention;

[0052] Figure 7 It is a structural diagram of the multi-granularity attention mechanism of the present invention;

[0053] Figure 8 is a structural diagram of an image enhancement decoder of the present invention;

[0054] Fig. 9 is a structural diagram of an image super-resolution decoder of the present invention;

[0055] Fig.10 is a structural diagram of the image fusion decoder of the present invention;

[0056] Fig.11 is a structural diagram of the image colorization decoder of the present invention;

[0057] Fig.12 A schematic diagram of the structure of a multi-task image processing system of the present invention;

[0058] Fig.13 The figure is a schematic structural diagram of a multi-task image processing device of the present invention. DETAILED DESCRIPTION

[0059] The present invention proposes a multi-task image processing method, system and device, aiming to design a multi-task image processing method that can simultaneously perform fine-grained feature extraction and global consistency optimization.

[0060] The multi-task image processing method proposed by the present invention will be described in a specific embodiment below:

[0061] Embodiment 1:

[0062] like Figure 1 , Figure 2 As shown, a multi-task image processing method comprises the following steps:

[0063] S10: Construct a multi-task network model, including an encoder and an image enhancement decoder, an image super-resolution decoder, an image fusion decoder, and an image colorization enhancement decoder; the encoder includes a multi-granularity heterogeneous convolution module and an adaptive fine structure enhancement module;

[0064] S20: Input the image data set into the multi-task network model to obtain infrared enhanced images, infrared super-resolution images, infrared and visible light image fusion images, and infrared image colorization images respectively;

[0065] S30: calculating the loss function between different images output by the multi-task network model and the infrared image and the visible light image in the input image data set, performing cyclic training on the multi-task network model, and calculating the loss function value through multiple training iterations until the number of training times reaches a set threshold or the value of the loss function reaches a set range;

[0066] Among them, the combination of detail-aware global consistency loss, L1 loss and structural similarity loss is the loss function, and the definition of detail-aware global consistency loss, L1 loss and structural similarity loss function is:

[0067]

[0068] in, represents the detail-aware global consistency loss function, represents the L1 loss function, represents the structural similarity loss function, represents the local detail preservation loss, represents the global consistency loss, represents the detail consistency loss, I out Represents the output images of different decoders, I gt represents the original input image, || ||1 represents the calculation of L1 norm;

[0069] The local detail preservation loss, global consistency loss and detail consistency loss functions are defined as:

[0070]

[0071] in, represents the Sobel gradient operator, H represents the histogram function of the image, N is the number of gray levels of the histogram, j represents the gray level, and patch i represents the i-th local region of the image, M is the number of local regions, and Autocorr represents the autocorrelation function;

[0072] The total loss function is defined as:

[0073]

[0074] Among them, α, β, and γ are weight parameters designed to balance the contribution of each loss term. An innovative weight update method is designed, and an adaptive optimization method is used to dynamically adjust α, β, and γ according to the importance of each loss term during the training process.

[0075] Specifically, an adaptive adjustment strategy based on the gradient change rate of each loss item is used, and the design scheme is:

[0076] Calculate the loss change rate,

[0077]

[0078] in, Represents the derivative of each loss with respect to time (number of training steps), represents the calculation of derivatives, and t represents time (number of training steps);

[0079] Adaptive weight adjustment adjusts weights based on the loss change rate. The weight update follows the following strategy: if the change rate of a loss item is large, it means that the loss item has a greater impact on the optimization of the model and should be given a higher weight; on the contrary, if the change rate is small, the weight should be reduced. The weight update formula is as follows:

[0080]

[0081]

[0082] Among them, η is the learning rate hyperparameter, which is used to control the step size of weight adjustment. is the normalization factor of the total loss change rate, which is the sum of the change rates of all loss items. t+1 represents the next moment (the number of training steps plus one).

[0083] Normalize the weights. In order to ensure the stability of the weights and that the sum is 1, the weights need to be normalized after each update:

[0084]

[0085] Furthermore, the encoder module consists of a multi-granularity heterogeneous convolution module and an adaptive fine structure enhancement module. The specific composition structure is as follows: Figure 3 As shown; the multi-granularity heterogeneous convolution module is composed of convolution block 1-1, convolution block 1-2, convolution block 1-3, convolution block 2-1, convolution block 2-2, convolution block 2-3, convolution block 3-1, convolution block 3-2, and convolution block 3-3. The convolution kernel size of convolution block 1-1, convolution block 2-1, and convolution block 3-1 is n1×n1, and HardSwish activation function is used. The convolution kernel size of convolution block 1-2, convolution block 2-2, and convolution block 3-2 is n2×n2, and S-type activation function is used. The convolution kernel size of convolution block 1-3, convolution block 2-3, and convolution block 3-3 is n3×n3, and S-type activation function is used.

[0086] The adaptive fine structure enhancement module consists of detail enhancement block 1, detail enhancement block 2, detail enhancement block 3 and detail enhancement block 4. Detail enhancement block 1 and detail enhancement block 2 receive feature information output by the multi-granularity heterogeneous convolution module, cross-splice the outputs of detail enhancement block 1 and detail enhancement block 2, and output them to detail enhancement block 3 and detail enhancement block 4. The size of the feature map is consistent with the size of the input image.

[0087] Specifically, each convolutional block consists of two convolutional layers, two activation functions, a skip connection, and a splicing block. The specific structure is as follows: Figure 4 As shown; in the multi-granularity heterogeneous convolution module, the convolution kernel size of convolution block 1-1, convolution block 2-1 and convolution block 3-1 is 17×17, using HardSwish activation function, the convolution kernel size of convolution block 1-2, convolution block 2-2 and convolution block 3-2 is 13×13, using S-type activation function, the convolution kernel size of convolution block 1-3, convolution block 2-3 and convolution block 3-3 is 9×9, using S-type activation function; the definitions of HardSwish activation function and S-type activation function are as follows:

[0088]

[0089] Specifically, the structure of the detail enhancement block is as follows: Figure 5 As shown in Figure 1, it includes a multi-layer perception module. The multi-layer perception module consists of 5 convolutional layers, 6 activation functions, and 1 splicing block. The convolution kernel size of convolutional layer 1, convolutional layer 2, and convolutional layer 3 is 5×5, and a linear rectification activation function is used. The specific composition structure is as follows: Figure 6 As shown in the figure, the adaptive fine structure enhancement module consists of detail enhancement block 1, detail enhancement block 2, detail enhancement block 3 and detail enhancement block 4. Each detail enhancement block consists of layer normalization, multi-granularity attention mechanism and multi-layer perception module. The multi-granularity attention mechanism consists of 1×1 convolution layer, query matrix (Q), key matrix (K), value matrix (V) and splicing block. The specific composition structure is shown in the figure. Figure 7 As shown; the definitions of layer normalization (LayerNorm) and linear rectification activation function are as follows:

[0090]

[0091] Where x is the input feature vector, μ is the mean of the feature vector, σ is the standard deviation of the feature vector, γ and β are the learned scaling and translation coefficients, and ∈ is a very small number for stable calculation.

[0092] Furthermore, if Figure 8As shown, the image enhancement decoder is composed of 2 convolutional layers, 2 activation functions and 1 Gabor filter. The visible light image is input into the Gabor filter to obtain high-frequency features and low-frequency features. The high-frequency features and the features output by the encoder are added as the input of the convolutional layer 1, and then pass through the activation function 1. The output of the activation function 1 and the low-frequency features are added as the input of the convolutional layer 2, and then pass through the activation function 2. Finally, the infrared enhanced image is obtained; the convolution kernel size of the two convolutional layers is n3×n3, the two activation functions use the S-type activation function, the inclination angle of the kernel function of the Gabor filter is θ1, and 4 filters of different sizes are set. Each image obtains 4 different high-frequency information. The 4 high-frequency information are subtracted from the original image to obtain 4 different low-frequency information;

[0093] Specifically, the convolution kernel size of the two convolutional layers is 3×3, the two activation functions use the S-type activation function, the Gabor filter kernel function has a tilt angle of 180°, and four filters of different sizes, each with a size of 7×7, 11×11, 15×15, and 17×17.

[0094] like Fig. 9 As shown, the image super-resolution decoder consists of 2 convolutional layers, 3 activation functions, pixel reorganization and 1 Gabor filter. First, the visible light image is input into the Gabor filter to obtain high-frequency features, and the high-frequency features and the features output by the encoder are added as the input of convolutional layer 1, and then pass through activation function 1. Then, the output of activation function 1 and the high-frequency features are added as the input of convolutional layer 2, and then pass through activation function 2. Again, the output of activation function 2 is used as the input of pixel reorganization, and then passes through activation function 3. Finally, an infrared super-resolution image is obtained; the convolution kernel size of the two convolutional layers is n4×n4, the three activation functions use S-type activation functions, the kernel function tilt angle of the Gabor filter is θ2, and 4 filters of different sizes are set, and each image obtains 4 different high-frequency information;

[0095] Specifically, the convolution kernel size of the two convolutional layers is 7×7, the three activation functions use the S-type activation function, the Gabor filter kernel function has a tilt angle of 90°, and there are four filters of different sizes, each with a size of 9×9, 13×13, 17×17, and 21×21. The pixel reorganization convolution kernel size is 7×7, the step size is 2, the padding method is SAME, and the number of output channels is 1.

[0096] like Fig.10As shown in the figure, the image fusion decoder consists of 3 convolutional layers and 3 activation functions; first, the features output by the encoder are used as the input of convolutional layer 1, and then pass through activation function 1; secondly, the output of activation function 1 is used as the input of convolutional layer 2, and then passes through activation function 2; then, the output of activation function 2 is used as the input of convolutional layer 3, and then passes through activation function 3; finally, the infrared and visible light fusion image is obtained; the convolution kernel size of the 3 convolutional layers is n5×n5, and the 3 activation functions use S-type activation function;

[0097] Specifically, the convolution kernel size of the three convolutional layers is 3×3, and the three activation functions use the S-type activation function.

[0098] like Fig.11 As shown in the figure, the image colorization decoder consists of 6 convolutional layers, 4 dilated convolutional layers, 8 activation functions and 5 splicing blocks. First, the output of the encoder is input to convolutional layer 1 and dilated convolutional layer 1 respectively, and then passes through activation functions 1 and 2 respectively. The visible light image is input to convolutional layer 2 and dilated convolutional layer 2 respectively, and then passes through activation functions 3 and 4 respectively. Secondly, the outputs of activation functions 1 and 2 are used as the input of splicing block 1, the output of splicing block 1 is used as the input of convolutional layer 3 and dilated convolutional layer 3 respectively, and then passes through activation functions 5 and 6 respectively. The outputs of activation functions 3 and 4 are used as the input of splicing block 2, and the output of splicing block 2 is used as the input of convolutional layer 4 and dilated convolutional layer 4 respectively, and then passes through activation functions 7 and 8 respectively. At the same time, the output of splicing block 1 is multiplied by the output of splicing block 2, and the multiplication result is input to the convolutional layer 1. Convolution layer 5, then, the outputs of activation functions 5 and 6 are used as the input of splicing block 3, the outputs of activation functions 7 and 8 are used as the input of splicing block 4, the outputs of splicing block 3, convolution layer 5 and splicing block 4 are used as the input of splicing block 5, and the output of splicing block 5 is used as the input of convolution layer 6. Finally, the infrared colorized image is obtained from the output of convolution layer 6; the convolution kernel size of convolution layer 1, convolution layer 2, convolution layer 3 and convolution layer 4 is n6×n6, the convolution kernel size of expanded convolution layer 1, expanded convolution layer 2, expanded convolution layer 3 and expanded convolution layer 4 is n7×n7, the expansion rate is m, activation functions 1 to 8 use linear rectification activation functions, and the convolution kernel size of convolution layer 5 and convolution layer 6 is n8×n8.

[0099] Specifically, the convolution kernel size of convolution layer 1, convolution layer 2, convolution layer 3 and convolution layer 4 is 3×3, the convolution kernel size of dilated convolution layer 1, dilated convolution layer 2, dilated convolution layer 3 and dilated convolution layer 4 is 3×3, the dilation rate is 4, activation functions 1 to 8 use linear rectification activation function, and the convolution kernel size of convolution layer 5 and convolution layer 6 is 5×5;

[0100] Furthermore, the infrared images and visible light images in the image dataset come from five datasets: CVC, MSRS, M3FD, KAIST and Urban100, and 15,000 pairs of high-quality infrared images and visible light images are selected for multi-task network training.

[0101] The multi-task image processing method constructed in this application can directly generate enhanced, super-resolution, fused and colorized output images from input infrared images and visible light images, avoiding the complexity of manually designing and adjusting each task step in traditional image processing methods. Through the unified training of multi-task models, this method not only effectively extracts local details, but also maintains global consistency, greatly improving processing efficiency and image quality, reducing redundant steps in the processing process, and improving the robustness and adaptability of the system.

[0102] The feasibility and superiority of this method are further verified by calculating the relevant indexes of the image obtained by the existing method. The existing method is: Chinese patent publication number: "CN112132258B", name: "A multi-task image processing method based on deformable convolution", and the relevant index comparison between the existing technology and the method proposed in this invention is shown in Table 1:

[0103] Table 1 Comparison of relevant indicators of the prior art and the method proposed in the present invention (“↑” means the higher the better, “↓” means the lower the better):

[0104]

[0105] It can be seen from the table that the method proposed in the present invention has higher image spatial frequency, edge strength, information entropy, visual information fidelity, structural similarity, peak signal-to-noise ratio, learnable perceptual image similarity and Fréchet initial distance. These indicators also further illustrate that the method proposed in the present invention has richer image details, maintains the visual quality and consistency of the overall image, and obtains high-quality image processing results in various tasks.

[0106] Embodiment 2:

[0107] A multi-task image processing system, such as Fig.12 As shown, the method for implementing the multi-task image processing method as described in Example 1 includes:

[0108] An image acquisition module, used to acquire training data; the training data are infrared images and visible light images;

[0109] A feature extraction module is used to perform feature extraction on each image in the training data using an encoder to obtain a feature map set for each image; the feature map set includes multiple feature maps of different sizes;

[0110] An infrared image enhancement module, used for enhancing a feature map of a specified size in a feature map set of each of the images to obtain an infrared enhanced image;

[0111] An infrared image super-resolution module, used to perform super-resolution on a feature map of a specified size in a feature map set of each image to obtain an infrared super-resolution image;

[0112] An infrared and visible light image fusion module, used for performing infrared and visible light image fusion on a feature map of a specified size in a feature map set of each image to obtain an infrared and visible light image fusion image;

[0113] An infrared image colorization module is used to perform infrared image colorization on a feature map of a specified size in a feature map set of each image to obtain a colored infrared image;

[0114] The model training module is used to perform multi-task cyclic supervised model training on the infrared enhanced image, infrared super-resolution image, infrared and visible light image fusion image and infrared image colorization image obtained by each module to obtain a multi-task image processing model.

[0115] Specifically, it also includes a storage medium for storing a computer program. Running the computer program can execute the multi-task image processing method provided in Example 1.

[0116] Embodiment 3:

[0117] A multi-tasking image processing device, such as Fig.13 As shown, it includes: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task image processing method as described in Example 1.

[0118] Specifically, the memory may be a ROM, a static storage device, a dynamic storage device or a RAM; the memory may store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to execute the various steps of the training method for the infrared and visible light image fusion network of the embodiment of the present invention;

[0119] The processor may be a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute relevant programs to implement the functions required to be performed by the units in the infrared and visible light image fusion training system of the present invention, or to perform the infrared and visible light image fusion training method of the present invention;

[0120] The processor may also be an integrated circuit chip with signal processing capability; in the implementation process, each step of the infrared and visible light image fusion training method of the present invention may be completed by hardware integrated logic circuits or software instructions in the processor; the above-mentioned processor may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; the infrared and visible light image fusion method, steps and logic block diagram of the present invention may be implemented or executed; the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.; the steps of the infrared and visible light image fusion method of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor; the software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.; the storage medium is located in a memory, the processor reads the information in the memory, and in combination with its hardware completes the functions required to be executed by the units included in the infrared and visible light image fusion training system of the present invention, or executes the infrared and visible light image fusion training method of the present invention;

[0121] The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or a communication network; for example, an image to be processed or an initial feature map of the image to be processed can be obtained through the communication interface;

[0122] A bus may include a pathway that transfers information between various components of the system (eg, memory, processor, communication interface).

[0123] Embodiment 4:

[0124] A computer-readable storage medium for multi-task image processing, which may be the computer-readable storage medium included in the system described in the above-mentioned embodiment; or it may be a computer-readable storage medium that exists independently and is not assembled into a device; the computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present invention.

[0125] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multi-task image processing method, characterized in that: The following steps are involved: Construct a multi-task network model, including an encoder and an image enhancement decoder, an image super-resolution decoder, an image fusion decoder, and an image colorization enhancement decoder; the encoder includes a multi-granularity heterogeneous convolution module and an adaptive fine structure enhancement module; The image dataset is input into the multi-task network model to obtain infrared enhanced images, infrared super-resolution images, infrared and visible light image fusion images, and infrared image colorization images. Calculate the loss function between different images output by the multi-task network model and the infrared image and visible light image in the input image dataset, perform cyclic training on the multi-task network model, and calculate the loss function value through multiple training iterations until the number of training times reaches the set threshold or the value of the loss function reaches the set range; Among them, the combination of detail-aware global consistency loss, L1 loss and structural similarity loss is the loss function, and the definition of detail-aware global consistency loss, L1 loss and structural similarity loss function is: in, represents the detail-aware global consistency loss function, represents the L1 loss function, represents the structural similarity loss function, represents the local detail preservation loss, represents the global consistency loss, represents the detail consistency loss, I out Represents the output images of different decoders, I gt represents the original input image, ||||1 represents the calculation of L1 norm, and SSIM is the calculation of the structural similarity between two images; The local detail preservation loss, global consistency loss and detail consistency loss functions are defined as: in, represents the Sobel gradient operator, H represents the histogram function of the image, N is the number of gray levels of the histogram, j represents the gray level, and patch i represents the i-th local region of the image, M is the number of local regions, and Autocorr represents the autocorrelation function; The total loss function is defined as: Among them, α, β, and γ are weight parameters designed to balance the contribution of each loss term; the weight update method is: Calculate the loss change rate, in, represents the derivative of each loss with respect to time, represents the calculation derivative, t represents time; Adaptive weight adjustment, adjust the weight based on the loss change rate, the weight update formula is as follows: Among them, η is the learning rate hyperparameter, which is used to control the step size of weight adjustment. is the normalization factor of the total loss change rate, which is the sum of the change rates of all loss items. t+1 represents the next moment. Normalize weights, normalize the weights after each update:

2. The multi-task image processing method according to claim 1, characterized in that: The multi-granularity heterogeneous convolution module is composed of convolution block 1-1, convolution block 1-2, convolution block 1-3, convolution block 2-1, convolution block 2-2, convolution block 2-3, convolution block 3-1, convolution block 3-2, and convolution block 3-3. The convolution kernel size of convolution block 1-1, convolution block 2-1, and convolution block 3-1 is n1×n1, and HardSwish activation function is used. The convolution kernel size of convolution block 1-2, convolution block 2-2, and convolution block 3-2 is n2×n2, and S-type activation function is used. The convolution kernel size of convolution block 1-3, convolution block 2-3, and convolution block 3-3 is n3×n3, and S-type activation function is used. The adaptive fine structure enhancement module consists of detail enhancement block 1, detail enhancement block 2, detail enhancement block 3 and detail enhancement block 4. Detail enhancement block 1 and detail enhancement block 2 receive feature information output by the multi-granularity heterogeneous convolution module, cross-splice the outputs of detail enhancement block 1 and detail enhancement block 2, and output them to detail enhancement block 3 and detail enhancement block 4. The size of the feature map is consistent with the size of the input image.

3. The multi-task image processing method according to claim 1, characterized in that: The image enhancement decoder consists of 2 convolutional layers, 2 activation functions and 1 Gabor filter. The image super-resolution decoder consists of 2 convolutional layers, 3 activation functions, pixel reorganization and 1 Gabor filter. The image fusion decoder consists of 3 convolutional layers and 3 activation functions; The image colorization decoder consists of 6 convolutional layers, 4 dilated convolutional layers, 8 activation functions and 5 splicing blocks.

4. The multi-task image processing method according to claim 1, characterized in that: The infrared images and visible light images in the image dataset come from five datasets: CVC, MSRS, M3FD, KAIST and Urban100. 15,000 pairs of high-quality infrared images and visible light images are selected for multi-task network training.

5. A multi-task image processing system, used to implement the multi-task image processing method according to any one of claims 1 to 4, characterized in that: include: An image acquisition module, used to acquire training data; The training data are infrared images and visible light images; A feature extraction module is used to perform feature extraction on each image in the training data using an encoder to obtain a feature map set for each image; the feature map set includes multiple feature maps of different sizes; An infrared image enhancement module, used for enhancing a feature map of a specified size in a feature map set of each of the images to obtain an infrared enhanced image; An infrared image super-resolution module, used to perform super-resolution on a feature map of a specified size in a feature map set of each image to obtain an infrared super-resolution image; An infrared and visible light image fusion module, used for performing infrared and visible light image fusion on a feature map of a specified size in a feature map set of each image to obtain an infrared and visible light image fusion image; An infrared image colorization module is used to perform infrared image colorization on a feature map of a specified size in a feature map set of each image to obtain a colored infrared image; The model training module is used to perform multi-task cyclic supervised model training on the infrared enhanced image, infrared super-resolution image, infrared and visible light image fusion image and infrared image colorization image obtained by each module to obtain a multi-task image processing model.

6. A multi-task image processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task image processing method according to any one of claims 1 to 4.

Images