A learnable semantic correspondence method and device based on example video colorization

Abstract

The present invention discloses a learnable semantic correspondence method and device based on example video colorization, relating to the field of image processing technology. The present invention utilizes a deep learning approach to learn the feature images of a reference image and an original image through a convolutional neural network, and obtains the semantic correspondence between the two images based on an image splicing operation. A deformed EfficientNet‑B0 network model is used as a correspondence generation network. Based on this correspondence generation network, a correspondence matrix between the reference image and the original image is obtained. While exploring deep spatial feature information, the method can obtain the vectors corresponding to the pixels and the relationship between the two in different dimensions, thereby improving the accuracy of the semantic correspondence between the reference image and the image to be colored (i.e., the original image), thereby improving the colorization performance.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a learnable semantic correspondence method and device based on example video coloring. Background Art

[0002] In recent years, with the continuous development of computer science, video colorization technology has played an important role in the fields of film and television material protection, cultural relics restoration, medical image processing and space exploration, and has become a hot topic in current image research.

[0003] The purpose of video colorization research is to make grayscale videos have realistic and reasonable colors. Among them, the deep learning-based methods usually use multiple neural networks to transform the L channel X of each frame of the color video into l As input, and obtain the timing information between frames, using the ab channel X of the frame ab Supervised learning is performed so that the network can output the corresponding ab channels for the grayscale video frames in the inference stage, and merge them with the L channel to generate a final series of color video frames.

[0004] In the field of deep learning video colorization, example-based colorization methods are currently the mainstream approach. They utilize reference frames with good color throughout the video colorization cycle to control the color style of the output frames and guide the colorization network to colorize the image. Throughout this process, colors are transferred from the reference example image to the grayscale image. This approach typically consists of two stages: the first stage obtains semantic correspondence between the reference image and the frame to be colored and aligns them. The second stage performs color matching propagation based on the semantic correspondence. Semantic correspondence links the features of the reference image and the image to be colored based on similarity, completing the alignment of the reference image and the image to be colored. Subsequent colorization stages require color matching based on this semantic correspondence. Failure to accurately match the semantics of the reference and image to be colored will lead to color propagation problems, compromising the colorization performance of each frame. Therefore, obtaining semantic correspondence is a critical step in example-based video colorization methods, and accurately acquiring this semantic correspondence has become a current research focus.

[0005] Establishing correspondences requires leveraging features between images, evolving from initial brightness and texture features to low-level handcrafted features, and now relying on deep image features derived from deep learning methods. Currently, commonly used semantic correspondences are established using the feature vectors output by a convolutional neural network of an input grayscale image and a reference image. The relationship between the two feature vectors is then determined through a simple inner product correlation operation. While this method is convenient and computationally inefficient, the inner product calculations in existing semantic correspondence determination methods only focus on the relationship between the corresponding vectors of each pair of pixels, failing to explore deeper spatial feature information. Furthermore, they are unable to obtain the corresponding vectors of pixels or the relationship between the two vectors across different dimensions. Summary of the Invention

[0006] In order to solve the above problems existing in the prior art, the present invention provides a learnable semantic correspondence method and device based on example video colorization.

[0007] To achieve the above object, the present invention provides the following solutions:

[0008] A learnable semantic correspondence method for example-based video colorization, comprising:

[0009] Input the reference image into the feature extraction network to obtain a first feature map, and input the original image into the feature extraction network to obtain a second feature map;

[0010] Performing a splicing operation on the first feature map and the second feature map to obtain a spliced ​​feature map;

[0011] Inputting the spliced ​​feature map into a correspondence generation network to obtain an output matrix; the correspondence generation network is a deformed EfficientNet-B0 network model;

[0012] A deformation operation is performed on the output matrix to obtain a correspondence matrix; the correspondence matrix is ​​used to represent the semantic correspondence between the reference image and the original image.

[0013] Optionally, the feature extraction network is VGG19.

[0014] Optionally, in the deformed EfficientNet-B0 network model, the stride of the convolution layer in the first stage is 1 and the convolution kernel size is 3×3. The second to eighth stages are stacks of moving bottleneck convolutions, and a multiplication factor is set after each moving bottleneck convolution. The convolution kernel size of the convolution layer in the ninth stage is 1×1.

[0015] Optionally, in each moving bottleneck convolution, a convolution layer with a convolution kernel size of 1×1 is first used, and then a separable convolution is split into a channel-by-channel convolution and a point-by-point convolution, and a channel attention module is set between the channel-by-channel convolution and the point-by-point convolution.

[0016] Optionally, a squeeze-excitation network is used as the channel attention module.

[0017] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0018] The present invention provides a learnable semantic correspondence method based on example video colorization. It uses a deep learning method to learn the feature images of a reference image and an original image through a convolutional neural network, and obtains the semantic correspondence between the two images based on an image splicing operation. The deformed EfficientNet-B0 network model is used as a correspondence generation network. Based on this correspondence generation network, a correspondence matrix between the reference image and the original image is obtained. While exploring deep spatial feature information, the vectors corresponding to the pixel points and the relationship between the two in different dimensions can be obtained, so as to improve the accuracy of the semantic correspondence between the reference image and the image to be colored (i.e., the original image), thereby improving the colorization performance.

[0019] A learnable semantic correspondence apparatus for example-based video colorization, comprising:

[0020] An image input device, used to obtain a reference image and an original image;

[0021] memory for storing computer programs;

[0022] A processor is connected to the memory and the image input device respectively, and is used to call and execute the computer program to implement the learnable semantic correspondence method based on example video coloring provided above.

[0023] Optionally, the memory is a computer-readable storage medium.

[0024] Optionally, the processor includes:

[0025] A feature extraction module is used to input the reference image into the feature extraction network to obtain a first feature map, and input the original image into the feature extraction network to obtain a second feature map;

[0026] a feature splicing module, configured to perform a splicing operation on the first feature map and the second feature map to obtain a spliced ​​feature map;

[0027] A matrix output module is used to input the spliced ​​feature map into a correspondence generation network to obtain an output matrix; the correspondence generation network is a deformed EfficientNet-B0 network model;

[0028] The matrix deformation module is used to perform a deformation operation on the output matrix to obtain a correspondence matrix; the correspondence matrix is ​​used to represent the semantic correspondence between the reference image and the original image.

[0029] Since the technical effect achieved by the device provided by the present invention is the same as the technical effect achieved by the above-mentioned learnable semantic correspondence method based on example video colorization, it will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 A schematic diagram of a process for obtaining semantic correspondences using a DeepExemplar-based video colorization method according to an embodiment of the present invention;

[0032] Figure 2 An algorithm flow chart of a learnable semantic correspondence method based on example video colorization provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the basic structure of MBConv provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0035] The purpose of the present invention is to provide a learnable semantic correspondence method and device based on example video colorization, which can obtain the vectors corresponding to pixel points and the relationship between the two in different dimensions while exploring deep spatial feature information, so as to improve the accuracy of the semantic correspondence between the reference image and the image to be colored, thereby improving the colorization performance.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0037] The existing feature semantic correspondence method is represented by the DeepExemplar-based video colorization method. For example, Figure 1 As shown, first the grayscale image x l (i.e. original image) and reference image y lab Input to the feature extraction network VGG19 to obtain the feature map of the corresponding scale H×W×C, and then deform the two feature maps into two feature vectors: F x (i) and F y (j). F x (i) and F y (j) All belong to That is, each pixel has a C-dimensional feature vector. The semantic correspondence matrix can be obtained by performing inner product operations on two feature vectors. Each correspondence in the semantic correspondence matrix The formula for determining is:

[0038]

[0039] The above formula uses the normalization method, where and Represent the average eigenvectors, the fraction above represents the inner product of the difference between the two eigenvectors and the average eigenvector, and the fraction below represents the product of the Euclidean distance between the two vectors and the average eigenvector. The result of softmax operation is the same as the ab channel y of the reference image ab The inner product can align the grayscale image with the reference image.

[0040] The learnable semantic correspondence method for example-based video colorization proposed in this paper is an innovative modification of the existing DeepExemplar-based video colorization method. Essentially, it replaces formula (1) with a learnable convolutional neural network. The method for extracting reference image and grayscale image features remains unchanged.

[0041] like Figure 2 As shown, the present invention provides a learnable semantic correspondence method based on example video coloring, including:

[0042] Step 1: Directly concatenate the two feature maps (i.e., the first feature map and the second feature map) output by the feature extraction network VGG19. The size of the concatenated feature map is H×W×2C.

[0043] Step 2: Input the spliced ​​feature map into the correspondence generation network to obtain the output matrix. The correspondence generation network used in the present invention is a reasonable convolutional neural network, and the output size of the network is H×W×HW. For example, a deformed network of the EfficientNet-B0 network model can be used as the correspondence generation network. Among them, on the basis of the original EfficientNet-B0 network model, the stride value of the convolution at each stage is set to 1 to ensure that the size of the output of each stage is consistent with the size of the input feature map. In the last stage, HW convolution kernels of size 1×1 are used for convolution operation to obtain a tensor with an output size of H×W×HW. The specific structure of the correspondence generation network is shown in Table 1 below.

[0044] Table 1 Correspondence relationship generated network structure table

[0045]

[0046] Table 1 shows that the modified EfficientNet-B0 network model still has nine stages. The convolutional layer in the first stage is a standard convolutional layer with a stride of 1 and a kernel size of 3×3. The rest of the structure of the first stage is the same as the existing EfficientNet-B0 network model, including the batch normalization layer and the swish activation function.

[0047] The core of the network is the second to eighth stages. These six stages are all stacked Mobile Inverted Bottleneck convolution (MBConv). Among them, there is a multiplication factor after each MBConv, and the multiplication factor is set to 1 or 6. The multiplication factor indicates that in the first convolution layer of MBConv, the number of convolution kernels is a multiple of the number of channels of the input feature matrix. k×k represents the convolution kernel size used in the DepwiseConv convolution layer in the MBConv structure, k=3 or k=5. The channels column indicates the number of channels of the output feature matrix after each operation (or each stage). Layers indicates the number of stacking times.

[0048] like Figure 3As shown in Figure 1, in MBConv, each MBConv first goes through a normal convolution layer with a kernel size of 1×1 and a stride of 1 to increase the dimension, and then splits a separable convolution (i.e., a DepwiseConv convolution layer with a kernel size of k×k and a stride of 1) into a channel-by-channel convolution and a point-by-point convolution (a normal convolution with a kernel size of 1×1 and a stride of 1). Channel-by-channel convolution does not change the number of channels, while the number of convolution kernels in point-by-point convolution is the number of channels corresponding to the output of each stage in Table 1. A channel attention module, namely Squeeze-and-Excitation Networks (SE), is added between the two convolution structures: channel-by-channel convolution and point-by-point convolution. As a channel attention module, SE can learn the dependencies between channels and enhance the useful information of channels. Set a Dropout layer. The Dropout layer exists when there is a shortcut, that is, when the number of MBConv stacks is greater than 1. Set the corresponding dropout_rate (loss probability). According to this loss probability, some neurons are lost during training and the original parameters are preserved to prevent overfitting. Figure 2 In the above figure, BN stands for batch normalization, Swish is an activation function, and s1 is the step size.

[0049] The ninth stage consists of a convolutional layer with a kernel size of 1×1×HW.

[0050] Step 3: Transform the H×W×HW matrix output by the correspondence network into the final HW×HW correspondence matrix. After the above correspondence generation network, a H×W×HW tensor is obtained, and the final correspondence matrix can be obtained by transformation.

[0051] In this paper, the correspondence generation network is not trained independently, but rather trained along with the final output of the entire network. This method uses an end-to-end approach, directly utilizing the loss function of the entire coloring network for backpropagation training to adjust the parameters of the correspondence network.

[0052] In addition, when the computer program in the above-mentioned memory is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk.

[0053] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0054] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A learnable semantic correspondence method based on example video colorization, characterized in that include: Input the reference image into the feature extraction network to obtain a first feature map, and input the original image into the feature extraction network to obtain a second feature map; Performing a splicing operation on the first feature map and the second feature map to obtain a spliced ​​feature map; Inputting the spliced ​​feature map into a correspondence generation network to obtain an output matrix; the correspondence generation network is a deformed EfficientNet-B0 network model; A deformation operation is performed on the output matrix to obtain a correspondence matrix; the correspondence matrix is ​​used to represent the semantic correspondence between the reference image and the original image.

2. The learnable semantic correspondence method based on example video colorization according to claim 1, characterized in that The feature extraction network is VGG19.

3. The learnable semantic correspondence method based on example video colorization according to claim 1, characterized in that In the deformed EfficientNet-B0 network model, the stride of the convolution layer in the first stage is 1 and the convolution kernel size is 3×3. The second to eighth stages are stacked mobile bottleneck convolutions, and a multiplication factor is set after each mobile bottleneck convolution. The convolution kernel size of the convolution layer in the ninth stage is 1×1.

4. The learnable semantic correspondence method based on example video colorization according to claim 3, characterized in that In each moving bottleneck convolution, a convolution layer with a convolution kernel size of 1×1 is first used, and then a separable convolution is split into a channel-by-channel convolution and a point-by-point convolution, and a channel attention module is set between the channel-by-channel convolution and the point-by-point convolution.

5. The learnable semantic correspondence method based on example video colorization according to claim 4, characterized in that The squeeze-excitation network is used as the channel attention module.

6. A learnable semantic correspondence device based on example video colorization, characterized in that include: An image input device, used to obtain a reference image and an original image; memory for storing computer programs; A processor is connected to the memory and the image input device respectively, and is used to call and execute the computer program to implement the learnable semantic correspondence method based on example video coloring as described in any one of claims 1 to 5.

7. The learnable semantic correspondence device based on example video colorization according to claim 6, characterized in that The memory is a computer-readable storage medium.

Images