Image colorization method, system and device based on dual-branch interactive fusion

Abstract

The present invention discloses an image colorization method, system and device based on dual-branch interactive fusion; it relates to the field of image processing technology, and the steps are: preprocessing the collected original data; using a local feature extraction branch and a global feature extraction branch to extract local spatial information and global semantic information from the processed original data; using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch; using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module; designing a loss function to minimize the loss function of the network output image and the label; the application proposes a local feature extraction branch and a global feature extraction branch, and combines the interactive fusion module and the context aggregation module to enhance the aggregation ability of multi-scale context semantic features, and optimize the boundary clarity and detail expression of the colorization result.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image colorization method, system and device based on dual-branch interactive fusion. Background Art

[0002] Infrared images can capture thermal radiation information that is imperceptible to the human eye, making them particularly suitable for target detection and recognition in complex environments such as low light, smoke, and inclement weather. However, infrared images lack rich color, detail, and texture information, resulting in poor visual expressiveness and difficulty conveying intuitive physical properties and scene information. To address this issue, infrared image colorization enhances the visual expressiveness and detail resolution of infrared images by introducing color information, making the images more realistic and easier to understand. This technology has shown broad potential in a variety of application areas, including military reconnaissance, autonomous driving, and security monitoring.

[0003] In recent years, infrared image colorization methods based on convolutional neural networks have been widely studied, especially those using UNet as the backbone network to learn the mapping relationship between pixels. Although these methods perform well in extracting local features, they have obvious limitations in capturing global features and long-range dependencies. The introduction of the Vision Transformer provides a new approach to the infrared image colorization task, effectively learning long-range dependencies in images and compensating for the shortcomings of convolution operations in capturing global information. However, although these colorization methods demonstrate excellent performance in global modeling, the computational complexity of the self-attention mechanism grows quadratically with the increase in image size, resulting in high consumption of computing resources.

[0004] Currently, state-space models have attracted the attention of many researchers. They establish long-range dependencies through efficient selective scanning mechanisms and show linear complexity related to the input size. However, directly applying state-space models leads to limited colorization effects due to the inability to capture local fine-grained features. Summary of the Invention

[0005] The technical solution of the present invention to solve the above technical problems is to provide an image colorization method based on dual-branch interactive fusion, comprising the following steps:

[0006] Step 1, data preprocessing: preprocess the collected raw data; use data enhancement strategies to ensure the diversity of the preprocessed raw data;

[0007] Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data;

[0008] The local feature extraction branch includes convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, instance normalization layer, L-type function, addition operation and splicing operation, which is used to realize multi-scale modeling of features through multi-level residual connections, enabling the network to extract feature information simultaneously in multiple receptive fields;

[0009] The global feature extraction branch includes a visual state space module and an adaptive linear attention module. The visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of boundary information. The adaptive linear attention module is used to encode the global information of the image, achieving accurate acquisition of global context information and strengthening global context modeling capabilities.

[0010] Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders and generate a feature representation containing more comprehensive semantic information;

[0011] Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch, and the interactive fusion module to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details;

[0012] The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features while maintaining the integrity of spatial information and enhancing the semantic expression ability of features;

[0013] Step 5, loss function correction: Design a loss function to minimize the loss function of the network output image and label; the loss function is a composite loss function, and the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss, and structural similarity loss.

[0014] Furthermore, in step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategy includes geometric transformation, illumination change and hybrid enhancement.

[0015] Furthermore, in step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor, and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent labels that are not spatially neighboring to enhance the distinction of image boundaries and prevent confusion in the sequence model; the adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear bias operation, and an addition operation, and the convolution kernel size is 3×3.

[0016] Furthermore, in step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation, and a splicing operation, and the convolution kernel size is 3×3.

[0017] Furthermore, in step 4, the context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation, and an addition operation, and the convolution kernel size is 3×3;

[0018] Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

[0019] Furthermore, the image colorization method based on dual-branch interactive fusion further includes the following steps:

[0020] Furthermore, in step 6, network performance is evaluated: peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance are selected to evaluate model performance.

[0021] To solve the above technical problems, the present invention further proposes an efficient infrared image colorization system based on dual-branch interactive fusion, the system comprising:

[0022] The data acquisition and preprocessing module uses infrared cameras to capture multi-scene and multi-type infrared images and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and uses data enhancement strategies to increase data diversity and prevent model overfitting.

[0023] Feature extraction and data construction module, used to extract the spatial, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images;

[0024] The colorization model construction and optimization module is used to design a network model to realize the colorization process of infrared images; it guides the model convergence through the loss function and uses hyperparameter tuning to improve model performance;

[0025] System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions;

[0026] The model iteration design and optimization module is used to optimize the network model or adjust the training strategy to continuously improve system performance and generation effects. The network model is continuously iterated and optimized based on evaluation feedback.

[0027] The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system to different user needs and actual tasks.

[0028] In order to solve the above technical problems, the present invention also proposes an efficient infrared image colorization electronic device based on dual-branch interactive fusion, including a memory, a processor, a communication interface and a bus; the memory, the processor and the communication interface are communicatively connected through the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described above.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The dual-branch encoder proposed in this paper has a local feature extraction branch that effectively captures fine-grained features in infrared images, while a global feature extraction branch focuses on long-range dependencies and overall image semantic information. The synergistic effect of the two improves the visual details and authenticity of the colorization results.

[0031] 2. The interactive fusion module proposed in this invention realizes the deep interactive fusion of local and global features. Through an efficient information transmission mechanism, it can accurately integrate multi-scale features, optimize the integrity of feature expression, and enhance the model's adaptability to image colorization in complex scenes.

[0032] 3. The context aggregation module proposed in this invention enhances the ability to aggregate multi-scale contextual semantic features, making the boundaries of the colorized results clearer and the details more expressive, while avoiding the problems of blurred boundaries and inconsistent semantic segmentation.

[0033] 4. This paper proposes a composite loss function consisting of adversarial loss, pixel loss, perceptual loss and structural similarity loss, which can learn the implicit relationship between infrared images and color images, and helps to promote the network to generate high-quality colorized images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0035] Figure 1 This is a flowchart of the steps of the image colorization method based on dual-branch interactive fusion of the present invention;

[0036] Figure 2 1 is a general structural diagram of the image colorization method based on dual-branch interactive fusion of the present invention;

[0037] Figure 3 Schematic diagram of the specific composition of the local feature extraction branch of the present invention;

[0038] Figure 4 Schematic diagram of the specific composition of the global feature extraction branch of the present invention;

[0039] Figure 5 Schematic diagram of the specific composition of the visual state space module of the present invention;

[0040] Figure 6 Schematic diagram of the specific composition of the high-efficiency scanning module of the present invention;

[0041] Figure 7 Schematic diagram of the specific composition of the adaptive linear attention module of the present invention;

[0042] Figure 8 Schematic diagram of the specific structure of the interactive fusion module of the present invention;

[0043] Figure 9 Schematic diagram of the specific structure of the context aggregation module of the present invention;

[0044] Figure 10 Schematic diagram of the specific structure of the feature correction module of the present invention;

[0045] Figure 11 Schematic diagram of the specific structure of the channel attention module of the present invention;

[0046] Figure 12 Schematic diagram of the specific structure of the spatial attention module of the present invention;

[0047] Figure 13 This is a schematic diagram comparing relevant indicators of the method proposed in the present invention;

[0048] Figure 14Schematic diagram of the image colorization system based on dual-branch interactive fusion of the present invention;

[0049] Figure 15 The figure is a schematic diagram of the internal structure of an electronic device according to the present invention, which is based on the image colorization method of the dual-branch interactive fusion. DETAILED DESCRIPTION

[0050] The present invention proposes an image colorization method, system and device based on dual-branch interactive fusion, aiming to design an efficient infrared image colorization method based on dual-branch interactive fusion.

[0051] The image colorization method based on dual-branch interactive fusion proposed by the present invention will be described in detail below in a specific embodiment:

[0052] In the technical solution of this embodiment, Figure 1 As shown, an image colorization method based on dual-branch interactive fusion includes the following steps:

[0053] Step 1, data preprocessing: preprocess the collected raw data; use data enhancement strategies to ensure the diversity of the preprocessed raw data;

[0054] Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data;

[0055] The local feature extraction branch includes convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, instance normalization layer, L-type function, addition operation and splicing operation, which is used to realize multi-scale modeling of features through multi-level residual connections, enabling the network to extract feature information simultaneously in multiple receptive fields;

[0056] The global feature extraction branch includes a visual state space module and an adaptive linear attention module. The visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of boundary information. The adaptive linear attention module is used to encode the global information of the image, achieving accurate acquisition of global context information and strengthening global context modeling capabilities.

[0057] Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders and generate a feature representation containing more comprehensive semantic information;

[0058] Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch, and the interactive fusion module to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details;

[0059] The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features while maintaining the integrity of spatial information and enhancing the semantic expression ability of features;

[0060] Step 5, loss function correction: Design a loss function to minimize the loss function of the network output image and label; the loss function is a composite loss function, and the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss, and structural similarity loss.

[0061] Furthermore, in step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategy includes geometric transformation, illumination change and hybrid enhancement.

[0062] Furthermore, in step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor, and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent labels that are not spatially neighboring to enhance the distinction of image boundaries and prevent confusion in the sequence model; the adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear bias operation, and an addition operation, and the convolution kernel size is 3×3.

[0063] Furthermore, in step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation, and a splicing operation, and the convolution kernel size is 3×3.

[0064] Furthermore, in step 4, the context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation, and an addition operation, and the convolution kernel size is 3×3;

[0065] Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

[0066] Furthermore, the image colorization method based on dual-branch interactive fusion further includes the following steps:

[0067] Step 6, evaluate network performance: select peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance to evaluate model performance.

[0068] Example 1:

[0069] An image colorization method based on dual-branch interactive fusion specifically includes the following steps:

[0070] Step 1: Data preprocessing: preprocess the collected raw data to eliminate the interference of noise data on model training; at the same time, adopt data enhancement strategies for application scenarios to ensure the diversity of input data;

[0071] Preprocessing operations include data cleaning, normalization, standardization, noise reduction, and data balancing. Data cleaning requires checking and screening the original acquired infrared images to remove samples with blur, interference, or invalid information to ensure the validity and representativeness of the input data. Normalization and standardization map pixel values ​​to a fixed range (such as [0, 1] or [-1, 1]) to improve the model's sensitivity and stability to the data. Noise reduction removes random noise from the image through methods such as Gaussian filtering, mean filtering, or median filtering to ensure the integrity of image detail information. Data balancing improves the balance of training data by adjusting data distribution.

[0072] Data augmentation operations include geometric transformation, illumination change, and hybrid enhancement. Geometric transformation can simulate different perspectives and layout changes of objects in real scenes through random rotation and random cropping, thereby improving the adaptability of the model. Illumination change simulates the response under different ambient lighting conditions by adjusting brightness and contrast, thereby enhancing the robustness of the model under complex lighting conditions. Hybrid enhancement generates new samples by mixing two images at the pixel or region level, which can effectively improve the generalization ability of the model.

[0073] Step 2: Feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the input image;

[0074] like Figure 2 As shown in the figure, the overall structure of the efficient infrared image colorization method based on dual-branch interactive fusion is shown. The initial convolution increases the expression ability of the model by increasing the dimension, enabling the network to better learn the complex features of the input data. It consists of a convolution layer with a convolution kernel size of 3×3; the tail convolution reconstructs the details and texture of the image and consists of a convolution layer with a convolution kernel size of 3×3; the T-type function prevents the overflow of image pixel values;

[0075] like Figure 3 As shown in the figure, the local feature extraction branch realizes multi-scale modeling of features through multi-level residual connections, enabling the network to extract feature information simultaneously in multiple receptive fields. It consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, instance normalization layer, L-type function, addition operation and splicing operation, and the convolution kernel size is 1×1 and 3×3;

[0076] like Figure 4 As shown in , the global feature extraction branch effectively captures the global context information of the image, reduces the computational complexity, and shows efficient colorization performance. It consists of a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multi-layer perceptron, a scaling factor, and an addition operation, and the convolution kernel size is 1×1; Figure 5 As shown in , the visual state space module can identify and process the boundary information of the image, reduce the ambiguity and potential confusion of the boundary information, and consists of a linear layer, a depth convolution layer, a sigmoid function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation. The convolution kernel size is 3×3; Figure 6 As shown in , the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent tags that are not spatially neighboring to enhance the distinction of image boundaries and prevent confusion in the sequence model; Figure 7 As shown in the figure, the adaptive linear attention module encodes the global information of the image, achieves accurate acquisition of global context information, and strengthens the global context modeling capability. It consists of a focus attention module, a deep convolutional layer, a pooling operation, a linear bias operation, and an addition operation. The convolution kernel size is 3×3; the scaling factor is learnable and controls the information of the skip connection.

[0077] Step 3: Feature information fusion: The interactive fusion module is used to merge the features extracted by the two branches, which realizes the effective complementarity and fusion of information between the two-branch encoders and generates a feature representation containing more comprehensive semantic information. Figure 8 As shown in the figure, the interactive fusion module ensures the consistency of input and output dimensions through information interaction of feature vectors and obtains more comprehensive information. It consists of a convolutional layer, a linear layer, a SE attention module, a pooling operation, and a splicing operation. The convolution kernel size is 3×3.

[0078] Step 4: feature information decoding; using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module, to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details; Figure 9As shown in Figure 2, the context aggregation module effectively combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features, while maintaining the integrity of spatial information and enhancing the semantic expression ability of features. It consists of a convolutional layer, a feature correction module, a channel attention module, a spatial attention module, a S-type function, a multiplication operation, and an addition operation, and the convolution kernel size is 3×3; Figure 10 As shown in Figure 1, the feature correction module ensures the spatial consistency of features at different levels during the fusion process by rescaling and fine-tuning the feature map, and optimizes the expression and reconstruction of spatial details. It consists of a convolution layer, upsampling operation, distortion operation and splicing operation, and the convolution kernel size is 1×1 and 3×3; Figure 11 As shown in , the channel attention module calculates the weight of each feature channel, so that the model can highlight the key information area, strengthen the channel containing important context information, and reduce the interference of redundant information. It consists of a normalization layer, a sigmoid function, and a multiplication operation. Figure 12 As shown in the figure, the spatial attention module helps the model focus on key areas in the feature space by capturing the spatial dependencies in the feature map. It consists of a convolutional layer, a sigmoid function, and a multiplication operation, with convolution kernel sizes of 1×1 and 3×3.

[0079] In order to ensure the robustness of the network, retain more structural information, and fully extract image features, the present invention uses four activation functions, namely L-type function, S-type function 1, S-type function 2, and T-type function. The definitions of L-type function, S-type function 1, S-type function 2, and T-type function are as follows:

[0080]

[0081] Step 5: Loss function correction: By designing a reasonable loss function, the loss function of the network output image and label is minimized to optimize the convergence effect of the model; the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss, and structural similarity loss;

[0082] In order to improve the similarity between the generated colorized image and the Ground Truth, the present invention adopts four loss functions to optimize the performance of the generator. Adversarial loss makes the generated image more realistic and delicate, and enhances its detail performance. Secondly, pixel loss ensures that the generated colorized image has minimal difference from the Ground Truth in chroma and brightness, so as to better match the real color. In addition, in order to reduce the difference in subjective perception, perceptual loss optimizes the visual consistency between the generated image and the Ground Truth through high-level feature constraints. Finally, structural similarity loss reduces the structural deviation between the generated image and the Ground Truth, thereby quantitatively measuring and improving the overall similarity between the two. The combination of these four loss functions helps to generate colorized images with higher visual fidelity and accuracy. The mathematical formulas of the four loss functions are as follows:

[0083]

[0084]

[0085] Step 6: Evaluate network performance. Select appropriate evaluation metrics to measure the accuracy of the algorithm, comprehensively evaluate model performance, and optimize and improve the model's performance in real-world scenarios. During the training of the network model, use appropriate evaluation metrics to evaluate the quality of the algorithm's colorization results, image similarity, and image distortion. Suitable evaluation metrics include peak signal-to-noise ratio (PSNR), structural similarity, perceptual image similarity, and Fréchet distance. PSNR measures the difference between the colorized result and the ground-truth image. A higher PSNR indicates better colorization quality. Structural similarity quantifies contour preservation during colorization by calculating the structural similarity between the colorized result and the ground-truth image. A higher structural similarity indicates a smaller difference. Perceptual image similarity focuses on the perceptual similarity between the colorized result and the ground-truth image. A lower perceptual image similarity indicates a more similar colorized result to the ground-truth image. Fréchet distance evaluates the perceptual quality of the generated image by comparing the distance between the distribution of the colorized result and the ground-truth image in deep feature embedding. A lower Fréchet distance indicates a higher similarity and better colorization quality. PSNR, structural similarity, perceptual image similarity, and Fréchet distance are defined as follows:

[0086]

[0087] where μ x , μ y denote the mean and variance of images x and y respectively, and Represents the standard deviation of image x and y, σ xyrepresents the covariance of images x and y, C1 and C2 are constants; d is the distance between x0 and x, w l is a trainable weight parameter; where g and r represent the generated image and the real image respectively, μ g and μ r represents the mean of each eigenvector, ∑ g and ∑ r denotes the covariance matrix of the respective eigenvectors, and Tr denotes the trace of the matrix.

[0088] All experiments were conducted on an Ubuntu 22.04.1LTS system, and an NVIDIA Tesla V100 GPU 16GB was used for algorithm acceleration. The training cycle was set to 200 rounds, and the learning rates of the generator and discriminator were set to 2e-4. The upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the number of images input to the network each time is within the range of 4-8, which can make the network training more stable and the training results better, and can ensure fast network fitting. The Adam optimizer is selected as the network parameter optimizer. Its advantages are simple implementation, high computational efficiency, low memory requirements, and parameter updates are not affected by gradient scaling, making the parameters relatively stable. When the discriminator's ability to judge fake images is balanced with the generator's ability to generate images that deceive the discriminator, the network is considered to have been basically trained. After the network training is completed, all parameters in the network need to be saved, and then the infrared image to be colored is input into the network to obtain a colored image. The network has no requirements for the size of the input image, and any size is acceptable.

[0089] Among them, the implementation of convolution, activation function, splicing operation and batch normalization are algorithms well known to those skilled in the art. The specific processes and methods can be found in corresponding textbooks or technical literature;

[0090] The present invention constructs an efficient infrared image colorization method based on dual-branch interactive fusion, which can directly generate a colorized image from an infrared image without going through other intermediate steps, thus avoiding the manual design of relevant colorization rules. Under the same conditions, the feasibility and superiority of the method are further verified by calculating the relevant indicators of the image obtained by the existing method. The relevant indicators of the existing technology and the method proposed in the present invention are compared. Figure 13 As shown;

[0091] from Figure 13 It can be seen that the method proposed in the present invention has higher peak signal-to-noise ratio, higher structural similarity, lower perceived image similarity, lower Fréchet distance, fewer parameters and lower GPU memory usage than the existing methods. These indicators also further demonstrate that the method proposed in the present invention has better colorization quality and lower computational complexity.

[0092] Example 2:

[0093] An efficient infrared image colorization system based on dual-branch interactive fusion, including:

[0094] The data acquisition and preprocessing module uses infrared cameras to capture multi-scene and multi-type infrared images and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and uses data enhancement strategies to increase data diversity and prevent model overfitting.

[0095] Feature extraction and data construction module, used to extract the spatial, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images;

[0096] The colorization model construction and optimization module is used to design a network model to realize the colorization process of infrared images; it guides the model convergence through the loss function and uses hyperparameter tuning to improve model performance;

[0097] System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions;

[0098] The model iteration design and optimization module is used to optimize the network model or adjust the training strategy to continuously improve system performance and generation effects. The network model is continuously iterated and optimized based on evaluation feedback.

[0099] The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system to different user needs and actual tasks.

[0100] In order to solve the above technical problems, the present invention also proposes an efficient infrared image colorization electronic device based on dual-branch interactive fusion, including a memory, a processor, a communication interface and a bus; the memory, the processor and the communication interface are communicatively connected through the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described above.

[0101] Furthermore, the data acquisition and preprocessing module acquires the image to be colored as an infrared image and a visible light image, and the infrared image can also be replaced with a normal grayscale image;

[0102] Furthermore, the data acquisition and preprocessing module,preprocessing mainly includes image cropping, image flipping and image translation,,which is divided into training set and test set with a ratio of 5:1;

[0103] Furthermore, the feature extraction and data construction module resizes each image from any size in the dataset to a fixed size of 256×256; a total of 200 rounds of training are performed with a batch size of 4; the number of filters in the first convolutional layer in the generator and discriminator is set to 64; the Adam optimizer is used; the discriminator and generator are trained alternately until the composite loss function converges;

[0104] Furthermore, the evaluation indicators of the system performance evaluation and feedback module are peak signal-to-noise ratio, structural similarity, perceived image similarity and Fréchet distance.

[0105] Example 3:

[0106] An efficient infrared image colorization electronic device based on dual-branch interactive fusion, such as Figure 15 As shown, it mainly includes a memory, a processor, a communication interface and a bus; wherein the memory, the processor and the communication interface realize communication connection with each other through the bus;

[0107] The memory may be a ROM, a static storage device, a dynamic storage device, or a RAM; the memory may store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to perform the various steps of the training method for the infrared image colorization network according to the embodiment of the present invention;

[0108] The processor may be a CPU, a microprocessor, an ASIC, a GPU, or one or more integrated circuits, and is used to execute relevant programs to implement the functions required to be performed by the units in the infrared image colorization training system of the present invention, or to perform the infrared image colorization training method of the present invention;

[0109] The processor may also be an integrated circuit chip having signal processing capabilities. During implementation, each step of the infrared image colorization training method of embodiment 1 of the present invention may be completed by hardware integrated logic circuits or software instructions in the processor. The aforementioned processor may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. The infrared image colorization method, steps, and logic block diagram of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the infrared image colorization method of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in a memory, and the processor reads information in the memory and, in combination with its hardware, completes the functions required to be executed by the units included in the infrared image colorization training system of the present invention, or executes the infrared image colorization training method of the present invention.

[0110] The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or communication networks; for example, the image to be processed or the initial feature map of the image to be processed can be obtained through the communication interface;

[0111] The bus may include a pathway for transferring information between various components of the system (e.g., memory, processor, communication interface);

[0112] The present invention also provides a computer-readable storage medium for efficient infrared image colorization based on dual-branch interactive fusion. The computer-readable storage medium may be the computer-readable storage medium included in the system described in the above embodiment; or it may be a separate computer-readable storage medium that is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present invention.

[0113] It should be noted that although Figure 15 The electronic device shown only shows the memory, processor, and communication interface. However, in the specific implementation process, those skilled in the art should understand that the system also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the system may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the system may also include only the devices necessary to implement the embodiments of the present invention, and does not necessarily include Figure 15 All devices shown in;

[0114] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An image colorization method based on dual-branch interactive fusion, characterized in that: The following steps are involved: Step 1, data preprocessing: preprocess the collected raw data; Adopt data enhancement strategies to ensure the diversity of preprocessed raw data; Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data; The local feature extraction branch includes convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, instance normalization layer, L-type function, addition operation and splicing operation, which is used to realize multi-scale modeling of features through multi-level residual connections, enabling the network to extract feature information simultaneously in multiple receptive fields; The global feature extraction branch includes a visual state space module and an adaptive linear attention module. The visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of boundary information. The adaptive linear attention module is used to encode the global information of the image, achieving accurate acquisition of global context information and strengthening global context modeling capabilities. Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders and generate a feature representation containing more comprehensive semantic information; Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch, and the interactive fusion module to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details; The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features while maintaining the integrity of spatial information and enhancing the semantic expression ability of features; Step 5, loss function correction: Design a loss function to minimize the loss function of the network output image and label; the loss function is a composite loss function, and the loss function used by the generator includes adversarial loss, pixel loss, perceptual loss, and structural similarity loss; The context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation and an addition operation, and the convolution kernel size is 3×3; Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

2. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: In step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategies include geometric transformation, illumination change and hybrid enhancement.

3. The image colorization method based on dual-branch interactive fusion according to claim 1, characterized in that: In step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor, and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, a sigmoid function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent markers that are not spatially adjacent to each other to enhance the distinction of image boundaries and prevent confusion in the sequence model; The adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear bias operation and an addition operation, and the convolution kernel size is 3×3.

4. The image colorization method based on dual-branch interactive fusion according to claim 1, characterized in that: In step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation, and a splicing operation, and the convolution kernel size is 3×3.

5. The image colorization method based on dual-branch interactive fusion according to claim 1, characterized in that: The following steps are also included: Step 6, evaluate network performance: select peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance to evaluate model performance.

6. An efficient infrared image colorization system based on dual-branch interactive fusion, for implementing the image colorization method based on dual-branch interactive fusion as described in any one of claims 1 to 5, characterized in that: The system comprises: The data acquisition and preprocessing module uses infrared cameras to capture multi-scene and multi-type infrared images and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and uses data enhancement strategies to increase data diversity and prevent model overfitting. Feature extraction and data construction module, used to extract the spatial, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images; The colorization model construction and optimization module is used to design a network model to realize the colorization process of infrared images; it guides the model convergence through the loss function and uses hyperparameter tuning to improve model performance; System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions; The model iteration design and optimization module is used to optimize the network model or adjust the training strategy to continuously improve system performance and generation effects. The network model is continuously iterated and optimized based on evaluation feedback. The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system to different user needs and actual tasks.

7. An efficient infrared image colorization electronic device based on dual-branch interactive fusion, characterized in that: It includes a memory, a processor, a communication interface and a bus; the memory, the processor and the communication interface are communicatively connected via the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described in any one of claims 1 to 5.

Images