Wide-spectrum image color reconstruction method

Abstract

The invention relates to a wide-spectrum image color reconstruction method. The method comprises the following specific steps: constructing a wide-spectrum image color reconstruction network which comprises a backbone network and a semantic sub-network; obtaining a wide-spectrum semantic segmentation data set, and pre-training a semantic sub-network; the wide-spectrum image is divided into two paths to be input into the backbone network and the semantic sub-network respectively, the middle output of the semantic sub-network is connected to the backbone network through the semantic guide cross attention module, and cross attention operation is carried out on the middle output and the middle features of the backbone network; fixing the weight of the semantic sub-network, only updating the parameters of the backbone network, and training the backbone network on the wide-spectrum image reconstruction data set until the network converges; the weight of the semantic sub-network is released, so that the parameters of the semantic sub-network and the parameters of the backbone network are updated together in training, and the semantic sub-network and the backbone network are jointly trained until convergence; and reconstructing and recovering the wide-spectrum image by using the trained semantic sub-network and the backbone network. According to the invention, the problem of color shift in wide-spectrum imaging can be effectively solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and relates to a wide-spectrum image color reconstruction method. Background Art

[0002] Low-light imaging technology plays a vital role in key areas such as environmental monitoring, biological research, national defense, and security monitoring. However, a major challenge it faces in extremely low-light environments is the limited number of photons, which reduces the signal-to-noise ratio and thus affects image clarity. Currently, there are two main strategies to address this challenge: The first strategy is to conduct in-depth research on the characteristics of imaging noise under low-light conditions and develop algorithms to reduce noise. The second strategy is to broaden the spectral acquisition range, that is, to use wide-spectrum imaging technology to increase the light flux during imaging, thereby improving the signal-to-noise ratio.

[0003] Wide-spectrum imaging technology enables the sensor to image in a wider spectral range (400-1100nm), which usually covers visible light (400-780nm) and near-infrared light (780-1100nm). In traditional imaging systems, in order to capture only visible light, a near-infrared cutoff filter is installed in front of the sensor to exclude unnecessary near-infrared light. But if this filter is removed, the sensor can work in a wider spectral range, significantly increasing the amount of light received.

[0004] Although wide-spectrum imaging technology can improve the image quality in low-light environments, it may also cause image color distortion. The intervention of near-infrared light makes the imaging color quite different from visible light, and color is crucial to human visual perception. Therefore, the key to whether wide-spectrum imaging technology can be widely adopted lies in whether it can effectively solve the problem of color distortion. Increasing the light flux of low-light imaging by collecting wide-spectrum images and using algorithms to correct color distortion in wide-spectrum imaging are effective methods to improve the signal-to-noise ratio of low-light imaging.

[0005] Existing wide-spectrum imaging algorithms all require additional near-infrared components as an aid. However, the introduction of additional near-infrared requires special design of the sensor or camera system, such as RGBN sensor where the N channel is a near-infrared channel, or other camera systems with independent near-infrared acquisition. Such system designs are complex and are not conducive to practical wide-spectrum imaging applications. Summary of the invention

[0006] In order to solve the above problems existing in the prior art, the present invention proposes a wide spectrum image color reconstruction method.

[0007] The specific technical solution adopted by the present invention is as follows:

[0008] A method for color reconstruction of a wide spectrum image, the method comprising the following steps:

[0009] Step 1: construct a wide-spectrum image color reconstruction network, including a backbone network and a semantic sub-network;

[0010] Step 2, obtaining a wide-spectrum semantic segmentation dataset and pre-training the semantic sub-network;

[0011] Step 3: The wide spectrum image is input into the backbone network and the semantic sub-network in two ways, and the intermediate output of the semantic sub-network is connected to the backbone network through the semantic guided cross attention module, and a cross attention operation is performed with the intermediate features of the backbone network;

[0012] Step 4, obtaining a wide spectrum image reconstruction data set, where the images in the data set are wide spectrum image-visible light image pairs;

[0013] Step 5, fixing the weight of the semantic sub-network so that the parameters of the semantic sub-network are not updated during the training process, and only updating the parameters of the backbone network; training the backbone network on the wide spectrum image reconstruction data set until the network converges;

[0014] Step 6, releasing the weight of the semantic sub-network so that the parameters of the semantic sub-network and the parameters of the backbone network are updated together during training; jointly training the semantic sub-network and the backbone network on the wide spectrum image reconstruction dataset until convergence;

[0015] Step 7: Use the trained semantic sub-network and backbone network to reconstruct and restore the wide spectrum image.

[0016] Furthermore, in step 1, the backbone network includes a convolutional layer, an encoding module, a semantically guided cross-attention module and a decoding module, and a skip layer connection is provided between the encoding module and the decoding module.

[0017] Furthermore, the encoding module includes a double-layer residual channel attention module, a convolutional downsampling layer and an activation layer. The outer layer of the double-layer residual channel attention module also adopts residual connection. The encoding module receives input data through the double-layer residual channel attention module and then outputs it to the convolutional downsampling layer for downsampling. Finally, the activation layer is nonlinearly activated and outputs data.

[0018] Furthermore, the decoding module includes a double-layer residual channel attention module, a transposed convolution upsampling layer and an activation layer. The outer layer of the double-layer residual channel attention module also adopts residual connection. The decoding module receives input data through the double-layer residual channel attention module and then outputs it to the transposed convolution upsampling layer for upsampling. Finally, the activation layer is nonlinearly activated and outputs data.

[0019] Furthermore, in step 3, the implementation steps of the semantically guided cross-attention module are:

[0020] Input semantic features and image features f i c , where the semantic features Outputted by the semantic sub-network, the image feature f i c Output from the backbone network;

[0021] Semantic features Dimensions are projected to the image features f i c Equal, obtain the converted semantic features and image features;

[0022] The converted semantic features are aggregated into channel dimension features through convolution and depthwise separable convolution, normalized, and then flattened into Q components through flattening operation;

[0023] The converted image features are divided into K component and V component through channel averaging and flattening operations;

[0024] The supervised image feature dimension is returned to the initial dimension through asymmetric attention calculation;

[0025] The output of the asymmetric attention calculation is reshaped and convolved before being combined with the image feature f i c Add, normalize the addition result and pass it through the forward layer to obtain the initial enhanced features;

[0026] The preliminary enhanced features are input into the self-attention module to obtain the enhanced image features.

[0027] Compared with the prior art, the advantages of the present invention are as follows:

[0028] (1) The method of the present invention effectively solves the color shift problem in wide-spectrum imaging, ensures that the color effect of the reconstructed image matches the human visual experience, and at the same time improves the imaging signal-to-noise ratio under low-light conditions, giving full play to the advantages of wide-spectrum imaging.

[0029] (2) The method of the present invention does not need to rely on additional independent near-infrared component constraints to decouple wide-spectrum images, and has strong scalability. Semantic prior information is introduced through a pre-trained semantic segmentation network, and a joint optimization algorithm and a semantic-guided cross-attention module are designed. These semantic information are used to reconstruct the color of wide-spectrum imaging, thereby improving the overall color reconstruction effect of wide-spectrum images.

[0030] (3) The method of the present invention can maximize the imaging performance of commercial RGB sensors under weak light conditions. Simply by removing the near-infrared cutoff filter at the front end of the commercial sensor to make it work in a wide spectrum band, and then processing the output data using the method of the present invention, an excellent reconstruction effect can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the overall structure diagram of the wide spectrum image color reconstruction network of the present invention;

[0032] Figure 2 is a flow chart of a method for color reconstruction of a wide spectrum image according to the present invention;

[0033] Figure 3 It is a structural diagram of the encoding and decoding modules of the wide spectrum image color reconstruction network of the present invention;

[0034] Figure 4 It is a structural diagram of the RCAB module of the wide spectrum image color reconstruction network of the present invention;

[0035] Figure 5 It is a structure diagram of the semantically guided cross-attention module of the wide-spectrum image color reconstruction network of the present invention;

[0036] Figure 6 This is a diagram showing the effect of the method of the present invention on color reconstruction of wide-spectrum images. DETAILED DESCRIPTION

[0037] The method of the present invention is described in detail below with reference to the accompanying drawings.

[0038] The structure of the wide spectrum image color reconstruction network constructed in this embodiment is as follows: Figure 1 As shown in the figure, the reconstruction network consists of two parts, namely the backbone network and the semantic sub-network. The backbone network includes convolutional layers, encoding modules, decoding modules, RCAB (Residual Channel Attention Block) and SGB (Semantic Guided cross attention Block). The wide spectrum image is input into the backbone network and the semantic sub-network in two ways. One way is input into the backbone network, passes through the convolutional layer and three encoding modules, and encodes the image features while downsampling. The other way of wide spectrum image is input into the semantic sub-network to extract semantic features. The image features and semantic features are subjected to the attention operation of SGB to obtain the enhanced image features. The enhanced image features are then reconstructed through multi-layer decoding modules and convolution modules to obtain a visible light picture with normal color. There is a skip layer connection constructed by RCAB between the encoding and decoding modules.

[0039] Specifically, the backbone network of the wide-spectrum image color reconstruction network is a codec structure with skip-layer connections. The encoding module Encoder consists of a double-layer RCAB and a convolutional downsampling layer. RCAB is the Residual Channel Attention Block, which combines residual connections and channel attention to extract network features. The specific design of the RCAB module can be seen in Figure 4 The outer layer of the double-layer RCAB module also uses residual connections.

[0040] The design of the encoding module and the decoding module is as follows Figure 3 As shown. The encoding module has multiple layers of nested residual connections. Specifically, the encoding module consists of a double-layer RCAB, a convolutional downsampling convolutional layer and a ReLu layer. The double-layer RCAB is connected with a skip layer before and after. This design is conducive to the gradual removal of near-infrared components layer by layer. The outer layer of the double-layer RCAB also uses residual connections. The encoding module receives input data through the double-layer RCAB and then outputs it to the convolutional downsampling layer for downsampling. Finally, the ReLu layer is nonlinearly activated and outputs the data. The design of the decoding module is consistent with the encoding module. The difference is that the downsampling convolutional layer is replaced by a transposed convolutional layer. The encoding module receives input data through the double-layer RCAB and then outputs it to the transposed convolutional upsampling layer for upsampling. Finally, the ReLu layer is nonlinearly activated and outputs the data. The encoding and decoding modules of the same layer of the backbone network are connected through RCAB. The shallow layer uses a single-layer RCAB, and the deep layer uses a double-layer cascaded RCAB module.

[0041] The specific process of the wide spectrum image color reconstruction method of this embodiment is as follows: Figure 2 shown.

[0042] First, prepare a wide-spectrum semantic segmentation dataset. The images in the dataset are wide-spectrum images and corresponding semantic segmentation image data pairs. Construct a suitable semantic subnetwork and use the wide-spectrum semantic segmentation dataset to pre-train the semantic subnetwork until convergence. The multi-classification cross entropy loss function of the pre-trained semantic subnetwork is:

[0043]

[0044] In this formula, L seg represents the cross entropy loss function, C represents the number of segmentation categories, y i represents the true label of the i-th category, p i is the predicted probability of category i. The number of segmentation categories in the wide spectrum semantic segmentation dataset is C=4.

[0045] Then the intermediate output of the semantic sub-network is connected to the backbone network through the semantic-guided cross-attention module, and the cross-attention operation is performed with the intermediate features of the backbone network: the backbone network and the semantic sub-network guided by semantics and trained on the wide-spectrum semantic segmentation dataset remove the last classification layer, and the extracted deep semantic features are combined with the enhanced backbone network through the semantic-guided cross-attention module. The semantic-guided cross-attention module is as follows: Figure 5 As shown. The input of this module is semantic features and image features f i c The semantic features are extracted from the semantic sub-network, and the feature dimensions are projected to be equal to the image features through the connection layer:

[0046]

[0047] The connection layer connect(·) contains two layers of convolution and one layer of ReLu and RCAB modules. The channel attention contained in the RCAB module is conducive to the initial aggregation of sparse semantic features. Then the semantic features converted by the connection layer are aggregated in the channel dimension through convolution and depth-separable convolution in the projection layer, and then normalized in the channel dimension layer LayerNorm(·):

[0048]

[0049] Where μ is the channel mean, σ 2 is the channel variance. It is then flattened into the Q component through a flattening operation (h×w×c→hw×c), Q∈(hw×c), c is the number of channels, h and w are the width and height of the feature map, respectively. The K and V components come from the backbone network. The image features are upscaled to 2k times through the projection layer, and then become K and V components through channel averaging and flattening operations (h×w×2k→2×hw×kc). Among them, K, V∈(hw×kc). Among them, k is an asymmetric factor, which is set to 1.5 in experimental experience. The dimensionality of image features in attention calculation makes it easier for semantic features to supervise image features more freely in the manifold space. Finally, the supervised image feature dimension is returned to the initial dimension through asymmetric attention calculation. The asymmetric attention operation is shown below.

[0050]

[0051] Where K T is the transpose of the K component. After two matrix multiplications to calculate the asymmetric transposed attention, the output component is reshaped and convolved with the initial image feature f i c The images are then added together and then go through layer normalization, a forward layer, and a symmetrical transposed self-attention module to obtain enhanced image features.

[0052] Prepare a wide-spectrum image reconstruction dataset, which contains several wide-spectrum image-visible light image pairs; fix the weight of the semantic sub-network so that the parameters of the semantic sub-network are not updated during the training process, and only the parameters of the backbone network are updated; train the backbone network on the wide-spectrum image reconstruction dataset until the network converges. The network loss function is L1 loss:

[0053]

[0054] Where i represents the image pixel, n is the total number of pixels, and y i is the prediction result, in this method it is the reconstruction result, is the true value, which is a visible light image in the method of this embodiment.

[0055] The weight of the semantic sub-network is released so that the parameters of the semantic sub-network and the backbone network are updated together during training; when training the backbone network on the wide-spectrum image reconstruction dataset, the L1 loss is also used to jointly train the semantic sub-network and the backbone network (collectively referred to as the total network) until the total network converges.

[0056] By adopting the semantically guided cross-attention module and the joint training strategy, the semantic prior information can be well used to reconstruct the wide-spectrum imaging color. The wide-spectrum image color reconstruction network trained in this embodiment is used to reconstruct the real wide-spectrum image. Figure 6 This is a display of the wide spectrum reconstruction results. The upper part is the wide spectrum input data, the lower part is the wide spectrum reconstruction data, and the right part is a local enlarged image of the box on the left. It can be seen that the reconstruction effect of this method is very good and the color accuracy of the reconstructed image is high.

Claims

1. A method for color reconstruction of a wide spectrum image, characterized in that: The method comprises the following steps: Step 1: construct a wide-spectrum image color reconstruction network, including a backbone network and a semantic sub-network; Step 2, obtaining a wide spectrum semantic segmentation dataset and pre-training the semantic sub-network; Step 3: The wide spectrum image is input into the backbone network and the semantic sub-network in two ways, and the intermediate output of the semantic sub-network is connected to the backbone network through the semantic guided cross attention module, and a cross attention operation is performed with the intermediate features of the backbone network; Step 4, obtaining a wide spectrum image reconstruction data set, where the images in the data set are wide spectrum image-visible light image pairs; Step 5, fixing the weight of the semantic sub-network so that the parameters of the semantic sub-network are not updated during the training process, and only updating the parameters of the backbone network; training the backbone network on the wide spectrum image reconstruction data set until the network converges; Step 6, releasing the weight of the semantic sub-network so that the parameters of the semantic sub-network and the parameters of the backbone network are updated together during training; jointly training the semantic sub-network and the backbone network on the wide spectrum image reconstruction dataset until convergence; Step 7: Use the trained semantic sub-network and backbone network to reconstruct and restore the wide spectrum image.

2. The method for color reconstruction of a wide spectrum image according to claim 1, characterized in that: In step 1, the backbone network includes a convolutional layer, an encoding module, a semantically guided cross-attention module and a decoding module, and a skip layer connection is provided between the encoding module and the decoding module.

3. A method for color reconstruction of a wide spectrum image according to claim 2, characterized in that: The encoding module includes a double-layer residual channel attention module, a convolutional downsampling layer and an activation layer. The outer layer of the double-layer residual channel attention module also adopts residual connection. The encoding module receives input data through the double-layer residual channel attention module and then outputs it to the convolutional downsampling layer for downsampling. Finally, the activation layer is nonlinearly activated and outputs data.

4. The method for color reconstruction of a wide spectrum image according to claim 2, characterized in that: The decoding module includes a double-layer residual channel attention module, a transposed convolution upsampling layer and an activation layer. The outer layer of the double-layer residual channel attention module also adopts residual connection. The decoding module receives input data through the double-layer residual channel attention module and then outputs it to the transposed convolution upsampling layer for upsampling. Finally, the activation layer is nonlinearly activated and outputs data.

5. The method for color reconstruction of a wide spectrum image according to claim 1, characterized in that: In step 3, the implementation steps of the semantically guided cross-attention module are: Input semantic feature f s n and image features f i c , where the semantic feature f s n Outputted by the semantic sub-network, the image feature f i c Output from the backbone network; The semantic feature f s n Dimensions are projected to the image features f i c Equal, obtain the converted semantic features and image features; The converted semantic features are aggregated into channel dimension features through convolution and depth-wise separable convolution, normalized, and then flattened into Q components through flattening operation; The converted image features are divided and flattened into K and V components through channel averaging; The supervised image feature dimension is returned to the initial dimension through asymmetric attention calculation; The output of the asymmetric attention calculation is reshaped and convolved before being combined with the image feature f i c Add, normalize the addition result and pass it through the forward layer to obtain the initial enhanced features; The preliminary enhanced features are input into the self-attention module to obtain the enhanced image features.

6. A method for color reconstruction of a wide spectrum image according to claim 5, characterized in that: The method for calculating the asymmetric cross attention is: Where Q∈(hw×c), K, V∈(hw×kc), k is an asymmetric factor, K T is the transpose of the K component, c is the number of channels, h and w are the width and height of the feature map respectively.

Images