Image colorization method, system and equipment based on double-branch interactive fusion

Abstract

The invention discloses an image colorization method, system and device based on double-branch interactive fusion. The method relates to the technical field of image processing, and comprises the following steps: preprocessing collected original data; extracting local spatial information and global semantic information from the processed original data by adopting a local feature extraction branch and a global feature extraction branch; combining the features extracted by the local feature extraction branch and the global feature extraction branch by adopting an interactive fusion module; a plurality of stacked context aggregation modules are adopted to perform complementary fusion among the features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module; designing a loss function, and minimizing the loss function of the network output image and the label; according to the method, a local feature extraction branch and a global feature extraction branch are provided, and an interactive fusion module and a context aggregation module are combined, so that the aggregation capability of multi-scale context semantic features is enhanced, and the boundary definition and detail expressive force of a colorization result are optimized.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image colorization method, system and device based on dual-branch interactive fusion. Background Art

[0002] Infrared images can capture thermal radiation information that is not perceptible to the human eye, and are particularly suitable for target detection and recognition in complex environments such as weak light, smoke, and bad weather. However, infrared images lack rich color, detail, and texture information, which makes the visual expression of the image poor and difficult to convey intuitive physical properties and scene information. In order to solve this problem, infrared image colorization enhances the visual expression and detail resolution ability of infrared images by introducing color information, making the image more realistic and easy to understand, and has shown broad prospects in many application fields such as military reconnaissance, autonomous driving, and security monitoring.

[0003] In recent years, infrared image colorization methods based on convolutional neural networks have been widely studied, especially using UNet as the backbone network to learn the mapping relationship between pixels. Although this type of method performs well in local feature extraction, it has obvious limitations in capturing global features and long-range dependencies. The introduction of Vision Transformer provides a new idea for the infrared image colorization task, which can effectively learn long-range dependencies in images and make up for the shortcomings of convolution operations in capturing global information. However, although this type of colorization method shows excellent performance in global modeling, the computational complexity of the self-attention mechanism grows quadratically with the increase in image size, resulting in a large consumption of computing resources.

[0004] Currently, state-space models have attracted the attention of many researchers, establishing long-range dependencies through efficient selective scanning mechanisms and showing linear complexity related to the input size. However, directly applying state-space models leads to limited colorization effects due to the inability to capture local fine-grained features. Summary of the invention

[0005] The technical solution of the present invention to solve the above technical problem is to provide an image colorization method based on dual-branch interactive fusion, comprising the following steps:

[0006] Step 1, data preprocessing: preprocess the collected raw data; adopt data enhancement strategy to ensure the diversity of the preprocessed raw data;

[0007] Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data;

[0008] The local feature extraction branch includes convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, instance normalization layer, L-type function, addition operation and concatenation operation, which are used to realize multi-scale modeling of features through multi-level residual connections, so that the network can extract feature information simultaneously on multiple receptive fields;

[0009] The global feature extraction branch includes a visual state space module and an adaptive linear attention module; the visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of the boundary information; the adaptive linear attention module is used to encode the global information of the image, realizing the accurate acquisition of global context information and strengthening the global context modeling capability;

[0010] Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders, and generate a feature representation containing more comprehensive semantic information;

[0011] Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module, so as to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details;

[0012] The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features and enhance the semantic expression ability of features while maintaining the integrity of spatial information;

[0013] Step 5, loss function correction: design a loss function to minimize the loss function of the network output image and the label; the loss function is a composite loss function, and the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss and structural similarity loss.

[0014] Furthermore, in step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategies include geometric transformation, illumination change and hybrid enhancement.

[0015] Furthermore, in step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent labels that are not neighborly in space to enhance the distinction of image boundaries and prevent confusion in the sequence model; the adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear deviation operation and an addition operation, and the convolution kernel size is 3×3.

[0016] Furthermore, in step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation and a splicing operation, and the convolution kernel size is 3×3.

[0017] Further, in step 4, the context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation and an addition operation, and the convolution kernel size is 3×3;

[0018] Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

[0019] Furthermore, the image colorization method based on dual-branch interactive fusion also includes the following steps:

[0020] Furthermore, in step 6, network performance is evaluated: peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance are selected to evaluate model performance.

[0021] In order to solve the above technical problems, the present invention also proposes an efficient infrared image colorization system based on dual-branch interactive fusion, the system comprising:

[0022] The data acquisition and preprocessing module uses infrared cameras to collect infrared images of multiple scenes and types, and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and increases data diversity through data enhancement strategies to prevent model overfitting;

[0023] Feature extraction and data construction module, used to extract the space, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images;

[0024] The colorization model construction and optimization module is used to design the network model and realize the colorization process of infrared images; guide the model convergence through the loss function, and use hyperparameter tuning to improve the model performance;

[0025] System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions;

[0026] Model iteration design and optimization module, used to optimize network models or adjust training strategies, continuously improve system performance and generation effects. Network models are continuously iterated and optimized based on evaluation feedback.

[0027] The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system in different user needs and actual tasks.

[0028] In order to solve the above technical problems, the present invention also proposes a high-efficiency infrared image colorization electronic device based on dual-branch interactive fusion, including a memory, a processor, a communication interface and a bus; the memory, the processor, and the communication interface are communicatively connected through the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described above.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. The dual-branch encoder proposed in the present invention, the local feature extraction branch effectively captures the fine-grained features in the infrared image, and the global feature extraction branch focuses on the long-range dependency and the overall image semantic information. The synergy of the two enhances the visual details and authenticity of the colorization results.

[0031] 2. The interactive fusion module proposed in the present invention realizes the deep interactive fusion of local and global features. Through the efficient information transmission mechanism, it can accurately integrate multi-scale features, optimize the integrity of feature expression, and enhance the model's adaptability to image colorization in complex scenes.

[0032] 3. The context aggregation module proposed in the present invention enhances the aggregation capability of multi-scale contextual semantic features, making the boundaries of the colorization results clearer and the details more expressive, while avoiding the problems of blurred boundaries and inconsistent semantic segmentation.

[0033] 4. The present invention proposes a composite loss function composed of adversarial loss, pixel loss, perceptual loss and structural similarity loss, which can learn the implicit relationship between infrared images and color images, and help drive the network to generate high-quality colorized images. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0035] Figure 1 It is a flowchart of the steps of the image colorization method based on double-branch interactive fusion of the present invention;

[0036] Figure 2 The figure is a general structural diagram of the image colorization method based on double-branch interactive fusion of the present invention;

[0037] Figure 3 It is a schematic diagram of the specific composition of the local feature extraction branch of the present invention;

[0038] Figure 4 A schematic diagram of the specific composition of the global feature extraction branch of the present invention;

[0039] Figure 5 Schematic diagram of the specific composition of the visual state space module of the present invention;

[0040] Figure 6 It is a schematic diagram of the specific composition of the high-efficiency scanning module of the present invention;

[0041] Figure 7 Schematic diagram of the specific composition of the adaptive linear attention module of the present invention;

[0042] Figure 8 It is a schematic diagram of the specific structure of the interactive fusion module of the present invention;

[0043] Fig. 9 It is a schematic diagram of the specific structure of the context aggregation module of the present invention;

[0044] Fig.10 It is a schematic diagram of the specific structure of the feature correction module of the present invention;

[0045] Fig.11 Schematic diagram of the specific structure of the channel attention module of the present invention;

[0046] Fig.12 Schematic diagram of the specific structure of the spatial attention module of the present invention;

[0047] Fig.13 This is a schematic diagram for comparing relevant indicators of the method proposed in the present invention;

[0048] Fig.14Schematic diagram of an image colorization system based on dual-branch interactive fusion according to the present invention;

[0049] Fig.15 The figure is a schematic diagram of the internal structure of an electronic device of the image colorization method based on dual-branch interactive fusion of the present invention. DETAILED DESCRIPTION

[0050] The present invention proposes an image colorization method, system and device based on double-branch interactive fusion, aiming to design an efficient infrared image colorization method based on double-branch interactive fusion.

[0051] The image colorization method based on dual-branch interactive fusion proposed by the present invention will be described in a specific embodiment below:

[0052] In the technical solution of this embodiment, Figure 1 As shown, an image colorization method based on dual-branch interactive fusion includes the following steps:

[0053] Step 1, data preprocessing: preprocess the collected raw data; adopt data enhancement strategy to ensure the diversity of the preprocessed raw data;

[0054] Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data;

[0055] The local feature extraction branch includes convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, instance normalization layer, L-type function, addition operation and concatenation operation, which are used to realize multi-scale modeling of features through multi-level residual connections, so that the network can extract feature information simultaneously on multiple receptive fields;

[0056] The global feature extraction branch includes a visual state space module and an adaptive linear attention module; the visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of the boundary information; the adaptive linear attention module is used to encode the global information of the image, realizing the accurate acquisition of global context information and strengthening the global context modeling capability;

[0057] Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders, and generate a feature representation containing more comprehensive semantic information;

[0058] Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module, so as to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details;

[0059] The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features and enhance the semantic expression ability of features while maintaining the integrity of spatial information;

[0060] Step 5, loss function correction: design a loss function to minimize the loss function of the network output image and the label; the loss function is a composite loss function, and the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss and structural similarity loss.

[0061] Furthermore, in step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategies include geometric transformation, illumination change and hybrid enhancement.

[0062] Furthermore, in step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent labels that are not neighborly in space to enhance the distinction of image boundaries and prevent confusion in the sequence model; the adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear deviation operation and an addition operation, and the convolution kernel size is 3×3.

[0063] Furthermore, in step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation and a splicing operation, and the convolution kernel size is 3×3.

[0064] Further, in step 4, the context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation and an addition operation, and the convolution kernel size is 3×3;

[0065] Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

[0066] Furthermore, the image colorization method based on dual-branch interactive fusion also includes the following steps:

[0067] Step 6, evaluate network performance: select peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance to evaluate model performance.

[0068] Embodiment 1:

[0069] An image colorization method based on dual-branch interactive fusion specifically comprises the following steps:

[0070] Step 1: Data preprocessing: preprocess the collected raw data to eliminate the interference of noise data on model training; at the same time, adopt data enhancement strategies for application scenarios to ensure the diversity of input data;

[0071] Preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing. Data cleaning requires checking and screening the original acquired infrared images to remove samples with blur, interference or invalid information to ensure the validity and representativeness of the input data. Normalization and standardization map pixel values ​​to a fixed range (such as [0,1] or [-1,1]) to improve the model's sensitivity and stability to data. Noise reduction removes random noise in the image through Gaussian filtering, mean filtering or median filtering to ensure the integrity of image detail information. Data balancing improves the balance of training data by adjusting data distribution.

[0072] Data augmentation operations include geometric transformation, illumination change, and hybrid enhancement. Geometric transformation can simulate different perspectives and layout changes of targets in real scenes through random rotation and random cropping of regions, thereby improving the adaptability of the model. Illumination change can simulate the response under different ambient lighting conditions by adjusting brightness and contrast, thereby enhancing the robustness of the model under complex lighting conditions. Hybrid enhancement can effectively improve the generalization ability of the model by generating new samples by mixing two images at the pixel level or region level.

[0073] Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the input image;

[0074] like Figure 2 As shown in the figure, the overall structure of the efficient infrared image colorization method based on dual-branch interactive fusion, in which the initial convolution increases the expression ability of the model by increasing the dimension, so that the network can better learn the complex features of the input data, consisting of a convolution layer with a convolution kernel size of 3×3; the tail convolution reconstructs the details and texture of the image, consisting of a convolution layer with a convolution kernel size of 3×3; the T-type function prevents the overflow of image pixel values;

[0075] like Figure 3 As shown in the figure, the local feature extraction branch realizes multi-scale modeling of features through multi-level residual connections, enabling the network to extract feature information simultaneously on multiple receptive fields. It consists of convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, instance normalization layer, L-type function, addition operation and splicing operation, and the convolution kernel size is 1×1 and 3×3;

[0076] like Figure 4 As shown in , the global feature extraction branch effectively captures the global context information of the image, reduces the computational complexity, and shows efficient colorization performance. It consists of a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multi-layer perceptron, a scaling factor, and an addition operation, and the convolution kernel size is 1×1; Figure 5 As shown in , the visual state space module can identify and process the boundary information of the image, reduce the ambiguity and potential confusion of the boundary information, and is composed of a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation. The convolution kernel size is 3×3; Figure 6 As shown in , the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent tags that are not spatially neighboring to enhance the distinction of image boundaries and prevent confusion in the sequence model; Figure 7 As shown in the figure, the adaptive linear attention module encodes the global information of the image, realizes the accurate acquisition of global context information, and strengthens the global context modeling capability. It consists of a focus attention module, a deep convolution layer, a pooling operation, a linear deviation operation, and an addition operation. The convolution kernel size is 3×3. The scaling factor can be learned to control the information of the jump connection.

[0077] Step 3: feature information fusion: The interactive fusion module is used to merge the features extracted by the two branches, which realizes the effective complementarity and fusion of information between the two-branch encoders and generates a feature representation containing more comprehensive semantic information; Figure 8 As shown in the figure, the interactive fusion module ensures the consistency of input and output dimensions through information interaction of feature vectors and obtains more comprehensive information. It consists of convolutional layer, linear layer, SE attention module, pooling operation and splicing operation, and the convolution kernel size is 3×3;

[0078] Step 4: feature information decoding; using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module, to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details; Fig. 9As shown in Figure 1, the context aggregation module effectively combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features, while maintaining the integrity of spatial information and enhancing the semantic expression ability of features. It consists of a convolutional layer, a feature correction module, a channel attention module, a spatial attention module, an S-type function, a multiplication operation, and an addition operation, and the convolution kernel size is 3×3; Fig.10 As shown in the figure, the feature correction module ensures the spatial consistency of features at different levels in the fusion process by rescaling and fine-tuning the feature map, and optimizes the expression and reconstruction of spatial details. It consists of convolutional layers, upsampling operations, distortion operations, and splicing operations, and the convolution kernel sizes are 1×1 and 3×3; Fig.11 As shown in , the channel attention module calculates the weight of each feature channel, so that the model can highlight the key information area, strengthen the channel containing important context information, and reduce the interference of redundant information. It consists of a normalization layer, a S-type function, and a multiplication operation; Fig.12 As shown in the figure, the spatial attention module helps the model focus on key areas in the feature space by capturing the spatial dependencies in the feature map. It consists of a convolutional layer, a sigmoid function, and a multiplication operation. The convolution kernel size is 1×1 and 3×3.

[0079] In order to ensure the robustness of the network, retain more structural information, and fully extract image features, the present invention uses four activation functions, namely L-type function, S-type function one, S-type function two, and T-type function. The definitions of L-type function, S-type function one, S-type function two, and T-type function are as follows:

[0080]

[0081] Step 5, loss function correction: By designing a reasonable loss function, the loss function of the network output image and the label is minimized to optimize the convergence effect of the model; the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss and structural similarity loss;

[0082] In order to improve the similarity between the generated colorized image and the Ground Truth, the present invention adopts four loss functions to optimize the performance of the generator. Adversarial loss makes the generated image more realistic and delicate, and enhances its detail performance. Secondly, pixel loss ensures that the generated colorized image has the smallest difference with the Ground Truth in chroma and brightness, so as to better match the real color. In addition, in order to reduce the difference in subjective perception, perceptual loss optimizes the visual consistency between the generated image and the Ground Truth through high-level feature constraints. Finally, the structural similarity loss reduces the structural deviation between the generated image and the Ground Truth, thereby quantitatively measuring and improving the overall similarity between the two. The combination of these four loss functions helps to generate colorized images with higher visual fidelity and accuracy. The mathematical formulas of the four loss functions are as follows:

[0083]

[0084]

[0085] Step 6, evaluate network performance; select appropriate evaluation indicators to measure the accuracy of the algorithm, comprehensively evaluate the model performance, and optimize and improve the performance of the model in actual scenes; in the process of training the network model, the quality of the algorithm colorization results, image similarity and image distortion are evaluated by appropriate evaluation indicators; appropriate evaluation indicators include peak signal-to-noise ratio, structural similarity, perceptual image similarity and Fréchet distance; peak signal-to-noise ratio is used to measure the difference between the colorization result and the real image. The higher the peak signal-to-noise ratio, the better the colorization quality; structural similarity quantifies the contour preservation in the colorization process by calculating the structural similarity between the colorization result and the real image. The higher the structural similarity, the smaller the difference; perceptual image similarity focuses on the perceptual similarity between the colorization result and the real image. The lower the perceptual image similarity, the more similar the colorization result and the real image are; Fréchet distance evaluates the perceptual quality of the generated image by comparing the distance between the colorization result and the real image distribution in the deep feature embedding. The lower the Fréchet distance, the higher the similarity and the better the colorization quality; peak signal-to-noise ratio, structural similarity, perceptual image similarity and Fréchet distance are defined as follows:

[0086]

[0087] where μ x , μ y Represent the mean and variance of images x and y respectively, and Represents the standard deviation of images x and y, σ xyrepresents the covariance of images x and y, C1 and C2 are constants; d is the distance between x0 and x, w l is a trainable weight parameter; where g and r represent the generated image and the real image respectively, μ g and μ r represents the mean of each eigenvector, ∑ g and∑ r represents the covariance matrix of the respective eigenvectors, and Tr represents the trace of the matrix.

[0088] The present invention conducted all experiments on the Ubuntu 22.04.1LTS system, and used NVIDIA Tesla V100GPU 16GB for algorithm acceleration. The training cycle was set to 200 rounds, and the learning rates of the generator and discriminator were set to 2e-4; the upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the number of images input to the network each time is in the range of 4-8, which can make the network training more stable and the training results better, and can ensure the rapid fitting of the network; the network parameter optimizer selects the Adam optimizer; its advantages mainly lie in simple implementation, efficient calculation, low memory requirements, and parameter updates are not affected by the scaling transformation of the gradient, making the parameters relatively stable; when the discriminator's ability to judge fake images is balanced with the generator's ability to generate images that deceive the discriminator, the network is considered to have been basically trained; after the network training is completed, all parameters in the network need to be saved, and then the infrared image to be colored is input into the network to obtain a colored image; the network has no requirements for the size of the input image, and any size is acceptable;

[0089] Among them, the implementation of convolution, activation function, splicing operation and batch normalization are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature;

[0090] The present invention constructs an efficient infrared image colorization method based on dual-branch interactive fusion, which can directly generate a colorized image from an infrared image without going through other intermediate steps, thus avoiding the manual design of relevant colorization rules; under the same conditions, the feasibility and superiority of the method are further verified by calculating the relevant indicators of the image obtained by the existing method; the relevant indicators of the existing technology and the method proposed by the present invention are compared. Fig.13 As shown;

[0091] from Fig.13 It can be seen that the method proposed in the present invention has higher peak signal-to-noise ratio, higher structural similarity, lower perceived image similarity, lower Fréchet distance, fewer parameters and lower GPU memory usage than the existing methods. These indicators also further illustrate that the method proposed in the present invention has better colorization quality and lower computational complexity.

[0092] Embodiment 2:

[0093] An efficient infrared image colorization system based on dual-branch interactive fusion, comprising:

[0094] The data acquisition and preprocessing module uses infrared cameras to collect infrared images of multiple scenes and types, and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and increases data diversity through data enhancement strategies to prevent model overfitting;

[0095] Feature extraction and data construction module, used to extract the space, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images;

[0096] The colorization model construction and optimization module is used to design the network model and realize the colorization process of infrared images; guide the model convergence through the loss function, and use hyperparameter tuning to improve the model performance;

[0097] System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions;

[0098] Model iteration design and optimization module, used to optimize network models or adjust training strategies, continuously improve system performance and generation effects. Network models are continuously iterated and optimized based on evaluation feedback.

[0099] The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system in different user needs and actual tasks.

[0100] In order to solve the above technical problems, the present invention also proposes a high-efficiency infrared image colorization electronic device based on dual-branch interactive fusion, including a memory, a processor, a communication interface and a bus; the memory, the processor, and the communication interface are communicatively connected through the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described above.

[0101] Furthermore, in the data acquisition and preprocessing module, the images to be colored are infrared images and visible light images, and the infrared images can also be replaced with normal grayscale images;

[0102] Furthermore, the data acquisition and preprocessing module, where preprocessing mainly includes image cropping, image flipping, and image translation, is divided into a training set and a test set with a ratio of 5:1;

[0103] Furthermore, the feature extraction and data construction module resizes each image from any size in the dataset to a fixed size of 256×256; a total of 200 rounds are trained with a batch size of 4; the number of filters in the first convolutional layer in the generator and discriminator is set to 64; the Adam optimizer is used; the discriminator and generator are trained alternately until the composite loss function converges;

[0104] Furthermore, the evaluation indicators of the system performance evaluation and feedback module are peak signal-to-noise ratio, structural similarity, perceived image similarity and Fréchet distance.

[0105] Embodiment 3:

[0106] An efficient infrared image colorization electronic device based on dual-branch interactive fusion, such as Fig.15 As shown, it mainly includes a memory, a processor, a communication interface and a bus; wherein the memory, the processor and the communication interface realize communication connection with each other through the bus;

[0107] The memory may be a ROM, a static storage device, a dynamic storage device or a RAM; the memory may store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to execute the various steps of the training method for the infrared image colorization network of the embodiment of the present invention;

[0108] The processor may be a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute relevant programs to implement the functions that the units in the infrared image colorization training system of the present invention need to perform, or to perform the infrared image colorization training method of the present invention;

[0109] The processor may also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the infrared image colorization training method of embodiment 1 of the present invention may be completed by hardware integrated logic circuits or software instructions in the processor. The above-mentioned processor may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The infrared image colorization method, steps and logic block diagram of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the infrared image colorization method of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and in combination with its hardware completes the functions required to be executed by the units included in the infrared image colorization training system of the present invention, or executes the infrared image colorization training method of the present invention.

[0110] The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or a communication network; for example, an image to be processed or an initial feature map of the image to be processed can be obtained through the communication interface;

[0111] The bus may include a pathway for transferring information between various components of the system (e.g., memory, processor, communication interface);

[0112] The present invention also provides a computer-readable storage medium for efficient infrared image colorization based on dual-branch interactive fusion. The computer-readable storage medium may be a computer-readable storage medium included in the system in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device; the computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present invention;

[0113] It should be noted that although Fig.15 The electronic device shown only shows a memory, a processor, and a communication interface. However, in the specific implementation process, those skilled in the art should understand that the system also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the system may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the system may also include only the devices necessary for implementing the embodiments of the present invention, and does not necessarily include Fig.15 All devices shown in;

[0114] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An image colorization method based on dual-branch interactive fusion, characterized in that: The following steps are involved: Step 1, data preprocessing: preprocess the collected raw data; Adopt data enhancement strategies to ensure the diversity of raw data after preprocessing; Step 2, feature information extraction: local feature extraction branch and global feature extraction branch are used to extract local spatial information and global semantic information from the processed raw data; The local feature extraction branch includes convolution layer 1, convolution layer 2, convolution layer 3, convolution layer 4, convolution layer 5, instance normalization layer, L-type function, addition operation and concatenation operation, which are used to realize multi-scale modeling of features through multi-level residual connections, so that the network can extract feature information simultaneously on multiple receptive fields; The global feature extraction branch includes a visual state space module and an adaptive linear attention module; the visual state space module is used to identify and process the boundary information of the image, reducing the ambiguity and potential confusion of the boundary information; the adaptive linear attention module is used to encode the global information of the image, realizing the accurate acquisition of global context information and strengthening the global context modeling capability; Step 3, feature information fusion: using an interactive fusion module to merge the features extracted by the local feature extraction branch and the global feature extraction branch, so as to achieve the complementarity and fusion of information between the two-branch encoders, and generate a feature representation containing more comprehensive semantic information; Step 4, feature information decoding: using stacked multiple context aggregation modules to perform complementary fusion of features output by the local feature extraction branch, the global feature extraction branch and the interactive fusion module, so as to maintain the integrity of spatial information and optimize the expression and reconstruction of spatial details; The context aggregation module combines the spatial details of low-level features with the semantic information of high-level features to achieve complementary fusion between features and enhance the semantic expression ability of features while maintaining the integrity of spatial information; Step 5, loss function correction: design a loss function to minimize the loss function of the network output image and the label; the loss function is a composite loss function, and the loss functions used by the generator include adversarial loss, pixel loss, perceptual loss and structural similarity loss.

2. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: In step 1, the preprocessing operations include data cleaning, normalization, standardization, noise reduction and data balancing; the data enhancement strategies include geometric transformation, illumination change and hybrid enhancement.

3. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: In step 2, the global feature extraction branch includes a layer normalization layer, a visual state space module, a convolution layer, an adaptive linear attention module, a multilayer perceptron, a scaling factor, and an addition operation; the visual state space module includes a linear layer, a deep convolution layer, an S-type function, an efficient scanning module, a layer normalization layer, a padding operation, a cropping operation, and a multiplication operation, and the convolution kernel size is 3×3; the efficient scanning module introduces a learnable filler in the scanning sequence, which is used between two adjacent markers that are not neighboring in space, to enhance the distinction of image boundaries and prevent confusion in the sequence model; The adaptive linear attention module includes a focus attention module, a deep convolution layer, a pooling operation, a linear bias operation and an addition operation, and the convolution kernel size is 3×3.

4. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: In step 3, the interactive fusion module includes a convolutional layer, a linear layer, an SE attention module, a pooling operation, and a splicing operation, and the convolution kernel size is 3×3.

5. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: In step 4, the context aggregation module includes a convolution layer, a feature correction module, a channel attention module, a spatial attention module, a sigmoid function, a multiplication operation and an addition operation, and the convolution kernel size is 3×3; Among them, the feature correction module includes a convolution layer, an upsampling operation, a distortion operation and a splicing operation, and the convolution kernel size is 1×1 and 3×3; the channel attention module includes a normalization layer, a S-type function and a multiplication operation; the spatial attention module includes a convolution layer, a S-type function and a multiplication operation, and the convolution kernel size is 1×1 and 3×3.

6. The image colorization method based on dual-branch interactive fusion according to claim 1 is characterized in that: The following steps are also included: Step 6, evaluate network performance: select peak signal-to-noise ratio, structural similarity, perceptual image similarity, and Fréchet distance to evaluate model performance.

7. An efficient infrared image colorization system based on dual-branch interactive fusion, characterized in that: The system comprises: The data acquisition and preprocessing module uses infrared cameras to collect infrared images of multiple scenes and types, and collects paired visible light images as labels; preprocesses the acquired raw data to improve data consistency and quality; and increases data diversity through data enhancement strategies to prevent model overfitting; Feature extraction and data construction module, used to extract the space, texture and edge features of infrared images and construct the mapping relationship between infrared images and color images; The colorization model construction and optimization module is used to design the network model and realize the colorization process of infrared images; guide the model convergence through the loss function, and use hyperparameter tuning to improve the model performance; System performance evaluation and feedback module, which uses a combination of subjective and objective evaluation to verify the colorization effect and provide targeted improvement suggestions; Model iteration design and optimization module, used to optimize network models or adjust training strategies, continuously improve system performance and generation effects. Network models are continuously iterated and optimized based on evaluation feedback. The result display and user verification module displays colorized images through a graphical interface, providing users with visual analysis tools to verify the applicability of the system in different user needs and actual tasks.

8. An efficient infrared image colorization electronic device based on dual-branch interactive fusion, characterized in that: It includes a memory, a processor, a communication interface and a bus; the memory, the processor and the communication interface are communicatively connected via the bus; the processor is used to execute the image colorization method based on dual-branch interactive fusion as described in any one of claims 1 to 6.

Images