SNN-based grayscale image coloring method

Abstract

The invention discloses an SNN-based grayscale image coloring method, which belongs to the technical field of image processing, and comprises the following steps of: firstly, sending an input image into a corresponding AI edge computing platform by using related image sensing equipment to serve as feature input of a coloring network; secondly, training the network model through a corresponding ImageNet data set to obtain a network model with certain coloring ability; and finally, performing a coloring reasoning process of the verification data set by using the obtained network model. According to the grayscale image coloring method based on the SNN, the lightweight spiking neural network can perform adaptive parameter adjustment on the network activation function layer through the training parameters, and the generalization ability of a network model can be effectively improved; fine-grained extraction of the feature gradient is carried out on the traditional jump connection layer, so that the redundancy of the jump connection to transmit feature information can be effectively reduced, and the feature extraction efficiency of the network is improved.

Description

Technical Field

[0001] The invention relates to a grayscale image coloring method based on SNN, and belongs to the technical field of image processing. Background Art

[0002] With the booming development of multimedia technology and digital economy, images and videos have become an indispensable carrier of information dissemination in people's daily lives. As an important research field of computer vision, image colorization has a wide range of applications in various aspects such as industrial production, medical treatment, education, and transportation. For example, in the early film and television industry, due to the limitations of equipment and technology, only black-and-white or grayscale videos or images can be shot and recorded. These colorless images or videos are often not lively enough and cannot meet people's viewing needs today, so it is necessary to colorize and remake the video or image; for example, in the animation production industry, the production of animation works is often that the animation author first draws a line drawing image without color, and then the artist performs the coloring process, and the coloring process is usually heavily dependent on manpower and inefficient. Today, artificial intelligence technology represented by deep learning is in the ascendant, and has brought great breakthroughs in computer vision, natural language processing and other fields. Therefore, grayscale image colorization can also be solved by this technology.

[0003] However, the current image colorization technology still has certain defects: the colorization method based on reference images requires manual search for suitable reference images, and the colorization effect is seriously dependent on the quality of the selected reference images; the colorization method based on coloring clues requires manual assignment of the colors of some pixels as marking information in order to expand to the entire image to complete the colorization, and the amount of marking information needs to be determined based on the complexity of the image structure. This process is relatively time-consuming and labor-intensive; therefore, in order to solve the above problems, a new image colorization method is urgently needed. Summary of the invention

[0004] To solve the above problems, the present invention proposes a grayscale image colorization method based on SNN, which can effectively improve the generalization ability of the network model, and can effectively reduce the redundancy of feature information transmitted by jump connections, thereby improving the feature extraction efficiency of the network.

[0005] The SNN-based grayscale image coloring method of the present invention comprises the following steps:

[0006] First, the input image is sent to the corresponding AI edge computing platform using relevant image sensing equipment as the feature input of the coloring network. Secondly, the network model is trained using the corresponding ImageNet dataset to obtain a network model with certain coloring capabilities. Finally, the obtained network model is used to verify the coloring inference process of the dataset.

[0007] Further, the coloring network includes a LW_SNN module, a gradient enhancement module and a pixel rearrangement module;

[0008] The LW_SNN module includes deep separable convolution, spiking neurons and residual connections; the spiking neurons are implemented through a leakage integral model, and the formula is as follows: u[t] = βu[t-1] + I[t], where u[t] is the neuron membrane potential at time t, β is the leakage coefficient, and I[t] is the current input current; when the membrane potential exceeds the threshold, the neuron will emit a pulse and reset the membrane potential: s[t] = Θ(u[t]-V th ), where Θ is a step function, V th is the threshold value;

[0009] The gradient enhancement module improves the local structure perception ability of the colorization process by enhancing the edge and gradient features of the image; the gradient enhancement is implemented by the following convolution operation: G(x)=x*K, where K is a predefined Laplacian operator kernel for extracting edge features;

[0010] The pixel rearrangement module implements downsampling and upsampling of the feature map through Pixel Unshuffle and Pixel Shuffle, respectively compressing and restoring the spatial resolution of the image.

[0011] Furthermore, the coloring network includes 6 layers of LW_SNN modules and 2 gradient enhancement modules, and the feature reuse capability of the network is further improved through residual connections.

[0012] Furthermore, the method specifically comprises the following steps:

[0013] First, perform network inference.

[0014] Assume that the input grayscale image is Where B is the batch size, H and W are the height and width of the image; the model inference process is as follows:

[0015] The first step is to perform convolution kernel spike neuron processing on the first layer.

[0016] The grayscale image is input to the first layer of deep convolutional neural network: conv1_1 and conv1_2, and then processed by spiking neurons; the process can be expressed as: cur1 =Conv2d(I gray , W conv1 ),

[0017] S1,M1=LIF(I cur1 , M1), where Conv2d represents the convolution operation, S1 is the output of the spike neuron, and M1 is the memory state;

[0018] The second step is gradient enhancement.

[0019] The output S1 of the first layer is subjected to gradient enhancement operation, using the following enhancement kernel K grad Perform convolution:

[0020] S grad1 =Conv2d(S1, K grad1 ), The enhanced gradient is concatenated with the output S1 of the spiking neuron to obtain the combined feature: I combined1 =concat(S1,S grad1 ), dim(I combined1 ) = B × 64 × H × W;

[0021] The third step is Pixel Unshuffle operation.

[0022] Perform Pixel Unshuffle operation to reduce the resolution of feature map, and perform feature fusion by reducing the resolution: I unshuffle =PixelUnshuffle(I combined1 );

[0023] The fourth step is to perform convolution kernel spike neuron processing on the second and third layers.

[0024] The feature map processed by Pixel Unshuffle is input to the second and third layers of convolution operations and spiking neurons: cur2 =Conv2d(I unshuffle , W conv2 ), S2,M2=LIF(I cur2 , M2);

[0025] I cur3 =Conv2d(S2,W conv3 ), S3,M3=LIF(I cur3 , M3);

[0026] The fifth step is gradient enhancement and splicing.

[0027] Perform gradient enhancement on the output S3 of the third layer and concatenate it with it: S grad2 =Conv2d(S3, K grad2 ), S combined2 =concat(S3,S grad2 );

[0028] Secondly, decode,

[0029] During decoding, the decoder performs reverse processing, which gradually restores the feature map to the output a and b components by reverse processing, using similar convolution, spiking neurons, and upsampling operations; specifically, as follows:

[0030] The first step is to perform convolution and pulse neuron processing on the fourth and fifth layers.

[0031] The feature map S after the fifth step of network inference is concatenated combined2 , input to the fourth and fifth convolutional and spiking neuron modules: I cur4 =Conv2d(S combined2 , W conv4 ), S4,M4=LIF(I cur4 , M4); I cur5 =Conv2d(S4,W conv5 ), S5,M5=LIF(I cur5 , M5);

[0032] The second step is Pixel Shuffle operation.

[0033] Use Pixel Shuffle to restore the resolution and get the decoded feature map:

[0034] I shuffle =PixelShuffle(S5);

[0035] The third step is to perform convolution and spike neuron processing on the sixth layer.

[0036] The feature map after the Pixel Shuffle operation is input to the sixth layer of convolution and pulse neurons to generate the final a and b components: ab =Conv2d(I shuffle , W conv6 );

[0037] The final output I lab It is the concatenated a and b components combined with the input grayscale image:

[0038] I lab =concat(I gray , I ab );

[0039] Finally, output the image,

[0040] The output is a Lab image, which contains grayscale L and color components a and b.

[0041] Compared with the prior art, the SNN-based grayscale image colorization method of the present invention, the lightweight pulse neural network can adaptively adjust the parameters of the network activation function layer through training parameters, which can effectively improve the generalization ability of the network model; the fine-grained extraction of feature gradients in the traditional jump connection layer can effectively reduce the redundancy of feature information transmitted by the jump connection, and improve the feature extraction efficiency of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the grayscale image coloring network structure of the present invention.

[0043] Figure 2 Schematic diagram of a gray image input of the present invention.

[0044] Figure 3 It is a schematic diagram of the coloring effect of the present invention. DETAILED DESCRIPTION

[0045] Embodiment 1:

[0046] The SNN-based grayscale image coloring method of the present invention comprises the following steps:

[0047] First, the input image is sent to the corresponding AI edge computing platform using relevant image sensing equipment as the feature input of the coloring network. Secondly, the network model is trained using the corresponding ImageNet dataset to obtain a network model with certain coloring capabilities. Finally, the obtained network model is used to verify the coloring inference process of the dataset.

[0048] Among them, the input of COLOR_SNN in the overall network structure is a single-channel grayscale image, and the output is a Lab color space image with three channels (where L represents grayscale information and ab represents color information); the network is mainly composed of the following modules: LW_SNN module, gradient enhancement module and pixel rearrangement module;

[0049] The LW_SNN module is the core computing unit of the network, which includes deep separable convolution, spiking neurons and residual connections; the spiking neurons are implemented by the Leaky Integrate-and-Fire (LIF) model, and the formula is as follows: u[t] = βu[t-1] + I[t], where u[t] is the neuron membrane potential at time t, β is the leakage coefficient, and I[t] is the current input current; when the membrane potential exceeds the threshold, the neuron will emit a spike and reset the membrane potential: s[t] = Θ(u[t]-V th ), where Θ is a step function, V th is the threshold value;

[0050] The gradient enhancement module improves the local structure perception ability of the colorization process by enhancing the edge and gradient features of the image; the gradient enhancement is implemented by the following convolution operation: G(x)=x*K, where K is a predefined Laplacian operator kernel for extracting edge features;

[0051] The pixel rearrangement module implements downsampling and upsampling of the feature map through Pixel Unshuffle and Pixel Shuffle, respectively compressing and restoring the spatial resolution of the image;

[0052] Specifically, the coloring network includes 6 layers of LW_SNN modules and 2 gradient enhancement modules, and the feature reuse capability of the network is further improved through residual connections.

[0053] In another embodiment, Figure 1 As shown, the method specifically comprises the following steps:

[0054] First, perform network inference.

[0055] Assume that the input grayscale image is Where B is the batch size, H and W are the height and width of the image; the model inference process is as follows:

[0056] The first step is to perform convolution kernel spike neuron processing on the first layer.

[0057] Will be like Figure 2 The grayscale image shown is input to the first layer of deep convolutional neural network: conv1_1 and conv1_2, and then processed by spiking neurons; this process can be expressed as:

[0058] I cur1 =Conv2d(I gray , W conv1 ), S1,M1=LIF(I cur1 , M1), where Conv2d represents the convolution operation, S1 is the output of the spike neuron, and M1 is the memory state;

[0059] The second step is gradient enhancement.

[0060] The output S1 of the first layer is subjected to gradient enhancement operation, using the following enhancement kernel K grad Perform convolution:

[0061] S grad1 =Conv2d(S1, K grad1 ), The enhanced gradient is concatenated with the output S1 of the spiking neuron to obtain the combined feature: I combined1 =concat(S1,S grad1 ), dim(Icombined1 ) = B × 64 × H × W;

[0062] The third step is Pixel Unshuffle operation.

[0063] Perform Pixel Unshuffle operation to reduce the resolution of feature map, and perform feature fusion by reducing the resolution: I unshuffle =PixelUnshuffle(I combined1 );

[0064] The fourth step is to perform convolution kernel spike neuron processing on the second and third layers.

[0065] The feature map processed by Pixel Unshuffle is input to the second and third layers of convolution operations and spiking neurons: cur2 =Conv2d(I unshuffle , W conv2 ), S2,M2=LIF(I cur2 , M2);

[0066] I cur3 =Conv2d(S2,W conv3 ), S3,M3=LIF(I cur3 , M3);

[0067] The fifth step is gradient enhancement and splicing.

[0068] Perform gradient enhancement on the output S3 of the third layer and concatenate it with it: S grad2 =Conv2d(S3, K grad2 ), S combined2 =concat(S3,S grad2 );

[0069] Secondly, decode,

[0070] During decoding, the decoder performs reverse processing. The decoder processes the feature map in reverse order and gradually restores it to the output a and b components, using similar convolution, pulse neurons, and upsampling (Pixel Shuffle) operations; the details are as follows:

[0071] The first step is to perform convolution and pulse neuron processing on the fourth and fifth layers.

[0072] The feature map S after the fifth step of network inference is concatenated combined2 , input to the fourth and fifth convolutional and spiking neuron modules: I cur4 =Conv2d(S combined2 , W conv4 ), S4,M4=LIF(Icur4 , M4); I cur5 =Conv2d(S4,W conv5 ), S5,M5=LIF(I cur5 , M5);

[0073] The second step is Pixel Shuffle operation.

[0074] Use Pixel Shuffle to restore the resolution and get the decoded feature map:

[0075] I shuffle =PixelShuffle(S5);

[0076] The third step is to perform convolution and spike neuron processing on the sixth layer.

[0077] The feature map after the Pixel Shuffle operation is input to the sixth layer of convolution and pulse neurons to generate the final a and b components: ab =Conv2d(I shuffle , W conv6 );

[0078] The final output I lab It is the concatenated a and b components combined with the input grayscale image:

[0079] I lab =concat(I gray , I ab );

[0080] Finally, output the image,

[0081] The output is a Lab image, which contains grayscale L and a, b color components. The coloring effect is as follows Figure 3 shown.

[0082] The above embodiments are only preferred implementations of the present invention, so all equivalent changes or modifications made according to the structures, features and principles described in the scope of the patent application of the present invention are included in the scope of the patent application of the present invention.

Claims

1. A grayscale image colorization method based on SNN, characterized in that: The following steps are involved: First, the input image is sent to the corresponding AI edge computing platform using relevant image sensing devices as the feature input of the coloring network; Secondly, the network model is trained through the corresponding ImageNet dataset to obtain a network model with certain colorization capabilities; finally, the obtained network model is used to verify the colorization inference process of the dataset.

2. The grayscale image colorization method based on SNN according to claim 1, characterized in that: The coloring network includes an LW_SNN module, a gradient enhancement module and a pixel rearrangement module; The LW_SNN module includes deep separable convolution, spiking neurons and residual connections; the spiking neurons are implemented through a leakage integral model, and the formula is as follows: u[t] = βu[t-1] + I[t], where u[t] is the neuron membrane potential at time t, β is the leakage coefficient, and I[t] is the current input current; when the membrane potential exceeds the threshold, the neuron will emit a pulse and reset the membrane potential: s[t] = Θ(u[t]-V th ), where Θ is a step function, V th is the threshold value; The gradient enhancement module improves the local structure perception ability of the colorization process by enhancing the edge and gradient features of the image; the gradient enhancement is implemented by the following convolution operation: G(x)=x*K, where K is a predefined Laplacian operator kernel for extracting edge features; The pixel rearrangement module implements downsampling and upsampling of the feature map through Pixel Unshuffle and Pixel Shuffle, respectively compressing and restoring the spatial resolution of the image.

3. The SNN-based grayscale image colorization method according to claim 2, characterized in that: The coloring network includes 6 layers of LW_SNN modules and 2 gradient enhancement modules.

4. The grayscale image colorization method based on SNN according to claim 1, characterized in that: The method specifically comprises the following steps: First, perform network inference. Assume that the input grayscale image is Where B is the batch size, H and W are the height and width of the image; the model inference process is as follows: The first step is to perform convolution kernel spike neuron processing on the first layer. The grayscale image is input to the first layer of deep convolutional neural network: conv1_1 and conv1_2, and then processed by spiking neurons; the process can be expressed as: cur1 =Conv2d(I gray , W conv1 ), S1,M1=LIF(I cur1 , M1), where Conv2d represents the convolution operation, S1 is the output of the spike neuron, and M1 is the memory state; The second step is gradient enhancement. The output S1 of the first layer is subjected to gradient enhancement operation, using the following enhancement kernel K grad Perform convolution: S grad1 =Conv2d(S1, K grad1 ), The enhanced gradient is concatenated with the output S1 of the spiking neuron to obtain the combined feature: I combined1 =concat(S1,S grad1 ), dim(I combined1 )=B×64×H×W; The third step is Pixel Unshuffle operation. Perform Pixel Unshuffle operation to reduce the resolution of feature map, and perform feature fusion by reducing the resolution: I unshuffle =PixelUnshuffle(I combined1 ); The fourth step is to perform convolution kernel spike neuron processing on the second and third layers. The feature map processed by Pixel Unshuffle is input to the second and third layers of convolution operations and spiking neurons: cur2 =Conv2d(I unshuffle , W conv2 ), S2,M2=LIF(I cur2 , M2); <h2 style=";text-align:left;direction:ltr">I<h2 style=";text-align:left;direction:ltr"> cur3 <h2 style=";text-align:left;direction:ltr"> =Conv2d(S2,W<h2 style=";text-align:left;direction:ltr"> conv3 <h2 style=";text-align:left;direction:ltr"> ),S3,M3=LIF(I<h2 style=";text-align:left;direction:ltr"> cur3 <h2 style=";text-align:left;direction:ltr"> (M3) The fifth step is gradient enhancement and splicing. Perform gradient enhancement on the output S3 of the third layer and concatenate it with it: S grad2 =Conv2d(S3, K grad2 ), S combined2 =concat(S3,S grad2 ); Secondly, decode, During decoding, the decoder performs reverse processing, which gradually restores the feature map to the output a and b components by reverse processing, using similar convolution, spiking neurons, and upsampling operations; specifically, as follows: The first step is to perform convolution and pulse neuron processing on the fourth and fifth layers. The feature map S after the fifth step of network inference is concatenated combined2 , input to the fourth and fifth convolutional and spiking neuron modules: I cur4 =Conv2d(S combined2 , W conv4 ), S4,M4=LIF(I cur4 , M4); I cur5 =Conv2d(S4,W conv5 ), S5,M5=LIF(I cur5 , M5); The second step is Pixel Shuffle operation. Use Pixel Shuffle to restore the resolution and get the decoded feature map: I shuffle =PixelShuffle(S5); The third step is to perform convolution and spike neuron processing on the sixth layer. The feature map after the Pixel Shuffle operation is input to the sixth layer of convolution and pulse neurons to generate the final a and b components: ab =Conv2d(I shuffle , W conv6 ); The final output I lab It is the concatenated a and b components combined with the input grayscale image: I lab =concat(I gray ,I ab ); Finally, output the image, The output is a Lab image, which contains grayscale L and color components a and b.

Images