Super-resolution reconstruction method and device of image, equipment, storage medium and product

Abstract

The invention provides an image super-resolution reconstruction method and device, equipment, a storage medium and a product, and relates to the technical field of image processing. The method comprises the following steps: acquiring a depth image and a color image of the same object; extracting the edge of the color image to obtain a first edge image; taking the edge of the up-sampled depth image to obtain a second edge image; fusing the first edge graph and the second edge graph to obtain an optimized edge graph; carrying out feature extraction on the up-sampled depth image and the optimized edge image through a convolutional network in a double-flow deep convolutional neural network model to obtain a low-resolution depth image feature and a high-resolution color image edge feature; and performing feature extraction and feature fusion on the low-resolution depth image features and the high-resolution color image edge features at different scales through a convolutional layer and a multi-scale residual module to obtain a reconstructed high-resolution depth image. According to the method, the texture copying phenomenon can be reduced, and the reconstruction effect of the super-resolution depth image is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment, storage medium and product for super-resolution reconstruction of an image. Background Art

[0002] At present, super-resolution reconstruction of a single image is mostly based on sample learning, such as the typical sparse coding method, which extracts features from the image and then encodes it into a low-resolution dictionary. The sparse coefficients are passed to the high-resolution dictionary to reconstruct the high-resolution part as output.

[0003] In the super-resolution reconstruction scenario of images, commonly used methods include interpolation methods, edge-based methods, and statistics-based methods.

[0004] However, interpolation methods increase the resolution by interpolating pixels, resulting in blurred images; edge-based methods use edge information to enhance image details, but are not effective for images with complex textures and weak structures; statistical methods reconstruct by learning the mapping relationship between low-resolution images and high-resolution images, but require a large number of training samples and feature engineering. Summary of the invention

[0005] The present application aims to provide a method, apparatus, device, storage medium and product for super-resolution reconstruction of an image, which can solve the problems of blurred edges of depth images, easy generation of texture replication artifacts, and the need for a large number of training samples and feature engineering.

[0006] In a first aspect, an embodiment of the present application discloses a method for super-resolution reconstruction of an image, the method comprising:

[0007] Get depth image and color image of the same object;

[0008] Extracting edges from the color image to obtain a first edge map, and extracting edges from the upsampled depth image to obtain a second edge map;

[0009] fusing the first edge map and the second edge map to obtain an optimized edge map;

[0010] Performing feature extraction on the upsampled depth image and the optimized edge map through a convolutional network in a two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features;

[0011] Through the convolutional layers and multi-scale residual modules in the dual-stream deep convolutional neural network model, feature extraction and feature fusion are performed on the low-resolution depth image features and the high-resolution color image edge features at different scales to obtain a reconstructed high-resolution depth image.

[0012] Optionally, the two-stream deep convolutional neural network model includes a feature fusion convolutional network, and the feature fusion convolutional network includes a first convolutional layer, a multi-scale residual module, a second convolutional layer and a deconvolutional layer;

[0013] The method extracts and fuses the low-resolution depth image features and the high-resolution color image edge features at different scales through the convolutional layer and the multi-scale residual module in the dual-stream deep convolutional neural network model to obtain a reconstructed high-resolution depth image, including:

[0014] Fusing the low-resolution depth image features and the high-resolution color image edge features through the first convolutional layer;

[0015] By means of the multi-scale residual module, the low-resolution depth image features and the high-resolution color image edge features are subjected to feature extraction at different scales, so as to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features;

[0016] fusing the feature maps mapped to the high-resolution features through the second convolutional layer;

[0017] The super-resolution image is reconstructed using the fusion results of the deconvolution layer and the second convolution layer to obtain a reconstructed high-resolution depth image.

[0018] Optionally, the multi-scale residual module includes a first residual module and a second residual module;

[0019] The multi-scale residual module is used to extract the low-resolution depth image features and the high-resolution color image edge features at different scales to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features, including:

[0020] Inputting the fused feature map output by the first convolutional layer into the first residual module, the first residual module is jump-connected to the second residual module, and the scales of feature extraction of the first residual module and the second residual module are different;

[0021] The output result of the first residual module is input into the second residual module to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features.

[0022] Optionally, before extracting edges from the color image to obtain a first edge map and extracting edges from the upsampled depth image to obtain a second edge map, the method further includes:

[0023] Mapping the depth image and the color image to the same coordinate system;

[0024] Matching pixels having the same row and column positions in the depth image and the color image one by one;

[0025] The depth image is subjected to cubic bi-interpolation so that the depth image is upsampled to a pixel size of the color image.

[0026] Optionally, the fusing the first edge map and the second edge map to obtain an optimized edge map includes:

[0027] A logical “AND” operation is performed on the first edge map and the second edge map to obtain the optimized edge map.

[0028] Optionally, the method further comprises:

[0029] Obtaining a mean square error function of the two-stream deep convolutional neural network model;

[0030] Constructing a total variation loss function of the depth image according to the size and gradient of the depth image;

[0031] Constructing a total variation gradient loss function of the color image according to the size of the depth image, the color image and the first edge map;

[0032] Obtaining a target loss function of the dual-stream deep convolutional neural network model according to the mean square error function, the total variation loss function, the total variation gradient loss function and a preset weight ratio;

[0033] The two-stream deep convolutional neural network model is trained by the target loss function.

[0034] Optionally, the training of the two-stream deep convolutional neural network model by the target loss function includes:

[0035] Obtaining a loss value of the objective loss function;

[0036] When the loss value is greater than a preset loss threshold, updating the preset model parameters according to the loss value to obtain a target neural network model;

[0037] Until the loss value is less than or equal to the preset loss threshold, and the target neural network model corresponding to the loss value is used as the dual-stream deep convolutional neural network model.

[0038] In a second aspect, an embodiment of the present application discloses a super-resolution reconstruction device for an image, the device comprising:

[0039] An acquisition module, used for acquiring a depth image and a color image of the same object;

[0040] An edge extraction module, configured to extract edges from the color image to obtain a first edge map, and to extract edges from the upsampled depth image to obtain a second edge map;

[0041] An edge fusion module, used for fusing the first edge map and the second edge map to obtain an optimized edge map;

[0042] A first model application module is used to extract features of the upsampled depth image and the optimized edge map through a convolutional network in a two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features;

[0043] The second model application module is used to extract and fuse the low-resolution depth image features and the high-resolution color image edge features at different scales through the convolutional layers and multi-scale residual modules in the dual-stream deep convolutional neural network model to obtain a reconstructed high-resolution depth image.

[0044] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0045] In a fourth aspect, an embodiment of the present application further discloses a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0046] In a fifth aspect, a computer program product comprises instructions or transactions, and when the instructions or transactions are executed by a processor in an electronic device, the electronic device executes the steps of the method described in the first aspect.

[0047] In summary, in this embodiment, on the one hand, through the dual-stream deep convolutional neural network model, the depth image and the color image are subjected to feature extraction, feature fusion and reconstruction to obtain a high-resolution depth image. Due to the use of a feedforward network model, a large number of training samples and feature engineering are not required, and the complementary information of the color image and the depth map can be fully combined to effectively reduce the texture duplication phenomenon and the edge discontinuity effect. On the other hand, on the basis of the deep neural network, a multi-scale feature extraction method combining multi-residuals is used to extract multi-scale features of the image from different levels. The multi-scale residual module can better fuse and denoise the extracted features, thereby improving the reconstruction effect of the super-resolution depth image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the attached picture:

[0049] Figure 1 is a flowchart of the steps of a super-resolution reconstruction method of an image provided by an embodiment of the present application;

[0050] Figure 2 It is a flowchart of the specific steps of a super-resolution reconstruction method of an image provided by an embodiment of the present application;

[0051] Figure 3 This is a network framework diagram of a multi-scale residual module provided in an embodiment of the present application;

[0052] Figure 4 It is a network framework diagram of a two-stream deep convolutional neural network model provided in an embodiment of the present application;

[0053] Figure 5 is a block diagram of a super-resolution reconstruction device for an image provided by an embodiment of the present application;

[0054] Figure 6 is a block diagram of an electronic device according to an embodiment of the present application;

[0055] Figure 7 It is a block diagram of an electronic device of another embodiment provided by the embodiments of the present application. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0057] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0058] In the description of the present disclosure, unless otherwise specified, "multiple" means two or more than two, and other quantifiers are similar thereto; "at least one item", "one item or multiple items" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one item a can represent any number of a; for another example, one item or multiple items among a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a kind of description of the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " indicates that the associated objects before and after are in an "or" relationship.

[0059] Although operations or steps are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood that it is required to perform these operations or steps in the specific order shown or in a serial order, or to perform all the operations or steps shown to obtain the desired results. In the embodiments of the present disclosure, these operations or steps can be performed in series; these operations or steps can also be performed in parallel; or some of these operations or steps can be performed.

[0060] First, the application scenarios of the present disclosure are described.

[0061] Image super-resolution is a problem that studies how to obtain a high-resolution image (HR) when a low-resolution image (LR) is input. Super-resolution technology (SR) refers to reconstructing a corresponding high-resolution image from an observed low-resolution image, and has important application value in monitoring equipment, satellite images, medical imaging and other fields.

[0062] With the rapid development of three-dimensional (3D) imaging, image super-resolution technology has been widely used in medical imaging, digital image enhancement, video surveillance and other fields, and is very effective for many computer vision tasks. At present, 3D scanning devices based on active three-dimensional imaging technology are still limited in terms of resolution and random noise. For example, the depth images obtained by laser radar, time of flight (TOF) camera, and three-dimensional structured light have low resolution, while human posture estimation and three-dimensional scene tasks have high requirements for depth maps. Most super-resolution convolutional neural network frameworks stack convolution kernels of a single size. Therefore, building a network with powerful feature extraction capabilities and reconstructing high-quality and high-resolution depth images has become a research hotspot in the field of computer vision.

[0063] Commonly used image super-resolution reconstruction methods include interpolation methods, edge-based methods, and statistical methods. Interpolation methods increase resolution by interpolating pixels, resulting in blurred images; edge-based methods use edge information to enhance image details, but are not effective for images with complex textures and weak structures; statistical methods reconstruct by learning the mapping relationship between low-resolution images and high-resolution images, but require a large number of training samples and feature engineering.

[0064] The deep learning-based method uses deep neural networks to learn the LR→HR mapping relationship, which can automatically learn features and has a better reconstruction effect.

[0065] At present, the super-resolution reconstruction algorithm based on the super-resolution deep neural network model (Super-Resolution Convolutional Neural Networks, SRCNN) is an image super-resolution reconstruction method based on deep learning. Its core principle is to map low-resolution images to high-resolution images by training a deep neural network to achieve super-resolution reconstruction of images.

[0066] The traditional SRCNN mainly consists of three parts: convolution layer, nonlinear mapping layer and deconvolution layer. Among them, the convolution layer and deconvolution layer realize the extraction and reconstruction of image features, and the nonlinear mapping layer realizes the nonlinear transformation of feature mapping. The three-layer convolution structure can also be explained as three steps: extraction and feature representation of image blocks, feature nonlinear mapping and reconstruction.

[0067] (1) Image block extraction and feature representation

[0068] The purpose of this step is to find a set of "basis" that can express the image block obtained after the previous preprocessing. These "basis" are along the edges in different directions, and the sparse coefficient is the weight assigned to each basis. Optionally, this part can be converted into a certain number of filters (convolution kernels) instead.

[0069] (2) Nonlinear mapping

[0070] This step maps the high-dimensional vector expressing the image block obtained in the first step to another high-dimensional vector, and uses this high-dimensional vector to express the high-resolution image block for the final reconstruction. Optionally, this step can use 1*1 convolution to achieve the transformation of vector dimension.

[0071] (3) Reconstruction

[0072] This step aggregates the high-resolution image blocks obtained at last (averages the overlapping positions) to form the final high-resolution image. Optionally, this part can be regarded as a linear operation, and a linear function (without activation function) can be constructed to achieve it.

[0073] However, most traditional super-resolution convolutional neural network frameworks simply stack convolution kernels of a single size and are unable to extract favorable guiding information from high-resolution color images of the same scene.

[0074] In order to solve the above-mentioned problems, the present disclosure provides a method, device, equipment, storage medium and product for super-resolution reconstruction of an image. The specific implementation methods of the present disclosure are described in detail below with reference to the accompanying drawings.

[0075] Figure 1 This embodiment provides a super-resolution reconstruction method for an image. Figure 1 , the method may include the following steps:

[0076] Step 101: Acquire a depth image and a color image of the same object.

[0077] It should be noted that a depth map is a grayscale image that contains the distance information of each pixel from the camera. It is a commonly used image representation in computer vision to describe the three-dimensional structure of a scene.

[0078] In some embodiments, obtaining a depth image of the same object in step 101 may include any of the following: obtaining depth information through sensors such as lidar or structured light, and then converting it into a depth image; calculating the depth using parallax information of a binocular or multi-camera, and then converting it into a depth image; analyzing the image using prior knowledge or a model to infer the depth information of each pixel.

[0079] It should be noted that a color image (RGB map) means that each pixel in the image is divided into three primary color components: R, G, and B. Each primary color component directly determines the intensity of its primary color. The color produced in this way is called true color, so a color image has three channels.

[0080] For example, if the image depth is 24, and R:G:B=8:8:8 is used to represent the color, then R, G, and B each occupy 8 bits to represent the intensity of their respective primary color components, and the intensity level of each primary color component is 2. 8 = 256 types. Images can accommodate 2 24 = 16M colors (24-bit color). 24-bit color is called true color, which can reach the limit of human eye resolution. The number of colors is more than 16.77 million, which is 2 to the 24th power.

[0081] In some embodiments, step 101 may include: acquiring a depth image and a color image of the same object by using a depth camera and a color camera.

[0082] In some embodiments, acquiring a color image of the same object in step 101 may include: acquiring a color image of a target object by a color camera, where the target object is an object represented by a depth image.

[0083] Step 102: extract edges from the color image to obtain a first edge map, and extract edges from the upsampled depth image to obtain a second edge map.

[0084] It should be noted that edge detection is a basic tool in image processing, computer vision and machine vision. It is usually used for feature extraction and feature detection. It aims to detect edges or discontinuous areas with obvious changes in a digital image. In one-dimensional space, similar operations are called step detection. An edge is a boundary line between different areas in an image. Usually an edge image is a binary image. The purpose of edge detection is to capture areas with sharp changes in brightness.

[0085] In some embodiments, extracting edges from the color image to obtain a first edge map in step 102 may include: extracting edges from the color image using a first edge filter to obtain the first edge map; extracting edges from the upsampled depth image to obtain a second edge map in step 102 may include: extracting edges from the upsampled depth image using a second edge filter to obtain the second edge map. The first edge filter and the second edge filter may be the same filter, or they may be different filters.

[0086] For example, edge filters may include: Sobel operator, Laplace operator or Canny operator. Among them, the Sobel operator combines Gaussian smoothing and differential derivation. It is a first-order derivative edge detection operator, which uses a convolution kernel to perform convolution and operation on each pixel in the image, and then uses a suitable threshold to extract the edge. The Soble operator has two convolution kernels, corresponding to the x and y directions respectively, and its calculation process is as follows: derivatives are taken in the x and y directions respectively; at each pixel point of the image, the approximate gradient is obtained by combining the above two results. The Laplace operator is a second-order derivative operator. In the edge area of ​​the image, the pixel value will change greatly, and extreme values ​​will appear when the derivative is taken for these pixels. At these extreme value positions, the second-order derivative is 0, so the second-order derivative can also be used to detect the edge of the image. The basic idea of ​​the Canny operator is to find the local maximum value of the gradient. First, a Gaussian smoothing filter is used for convolution noise reduction, and then a pair of convolution arrays are used to calculate the edge gradient and direction, and then non-maximum suppression is used to remove non-edge lines, and finally a hysteresis threshold (high threshold and low threshold) is used to detect and connect the edges.

[0087] It should be noted that upsampling is also known as image enlargement and image interpolation; the main purpose of upsampling is to enlarge the original image so that it can be displayed on a display device with higher resolution; upsampling principle: image enlargement almost always adopts the interpolation method, that is, new elements are inserted between pixels based on the original image pixels using a suitable interpolation algorithm; interpolation algorithms also include traditional interpolation, edge image-based interpolation, and area-based image interpolation.

[0088] In some embodiments, the method of obtaining the upsampled depth image in step 102 may include any one of the following: bilinear interpolation, deconvolution, and unpooling.

[0089] Step 103: fuse the first edge map and the second edge map to obtain an optimized edge map.

[0090] It is understandable that the edge extraction algorithm has low precision, high computational complexity, and the detected edges are usually discontinuous and irregular. In order to overcome the above defects, in this embodiment, it is necessary to optimize the edge area with higher entropy value to improve the display effect of the edge area of ​​the image.

[0091] In some embodiments, step 103 may include: performing a logical “AND” operation on the first edge map and the second edge map to obtain an optimized edge map.

[0092] In other embodiments, step 103 may include: selecting pixels to be optimized from pixels based on the entropy value of each pixel in the edge area of ​​the image; determining a target pixel associated with the pixel to be optimized, the target pixel being a pixel whose change state is determined; optimizing the probability value of the pixel to be optimized through a preset water ripple propagation model based on the probability value of the target pixel, the preset water ripple propagation model being used to represent the probability value relationship between each pixel; determining the change state of the pixel to be optimized based on the optimized probability value, and optimizing the edge area of ​​the image based on the change state.

[0093] Step 104, performing feature extraction on the upsampled depth image and the optimized edge map through the convolutional network in the two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features.

[0094] It should be noted that Convolutional Neural Networks is a deep learning model or a multi-layer perceptron similar to artificial neural networks, which is often used to analyze visual images. Convolutional neural networks are essentially a mapping from input to output. They can learn a large number of mapping relationships between inputs and outputs without requiring any precise mathematical expressions between inputs and outputs. As long as the convolutional neural network is trained with a known pattern, it will have the mapping ability between input and output pairs.

[0095] In some embodiments, step 104 may include: performing convolution operations and linear rectification unit activation on the upsampled depth image and optimized edge map through multiple convolution layers in a dual-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features.

[0096] In other embodiments, the dual-stream deep convolutional neural network model includes a first convolutional network and a second convolutional network, and the first convolutional network and the second convolutional network have the same structure; step 104 may include: inputting the upsampled depth image into the first convolutional network to obtain output low-resolution depth image features; inputting the optimized edge map into the second convolutional network to output high-resolution color image edge features.

[0097] In a possible implementation, each convolutional network includes three convolutional layers. The first convolutional network includes convolutional layer Conv1-1, convolutional layer Conv1-2, and convolutional layer Conv1-3. Convolutional layer Conv1-1 and convolutional layer Conv1-2 perform convolution operations and linear rectification unit activation, and convolutional layer Conv1-3 performs convolution operations. Similarly, the second convolutional network includes convolutional layer Conv2-1, convolutional layer Conv2-2, and convolutional layer Conv2-3. Convolutional layer Conv2-1 and convolutional layer Conv2-2 perform convolution operations and linear rectification unit activation, and convolutional layer Conv2-3 only performs convolution operations.

[0098] Optionally, first, the first convolutional layers Conv1-1 and Conv2-1 both pass through n1 filters of size f1×f1 and output n1 feature maps. Secondly, similar to the first convolutional layers Conv1-1 and Conv2-1, the second convolutional layers Conv1-2 and Conv2-2 both have n2 filters of size f2×f2. Then, the third convolutional layers Conv1-3 and Conv2-3 both have n3 filters of size f3×f3. The networks for depth images and color images share the same structure and parameters. Except for specific convolutional layers (such as Conv1-3 and Conv2-3), other convolutional layers are usually activated by linear rectifier units (ReLU). Finally, the outputs of the third convolutional layers Conv1-3 and Conv2-3 are used as inputs to the feature fusion part and combined in the subsequent network. Therefore, Conv1-3 and Conv2-3 output low-resolution depth image features and high-resolution color image edge features, respectively.

[0099] For example, through n1 filters of size f1×f1, n1 feature maps are output using the following formula:

[0100] F1(X)=max(0,W1*X+B1)

[0101] Among them, W represents the weight of the convolution kernel, corresponding to n1 c×f1×f1 convolution kernels, c is the number of channels of the input image, f1×f1 is the convolution kernel size, B1 represents the weight bias of the convolution kernel, which is an n1-dimensional vector, and * represents the convolution operation; that is, through the above formula, n1 convolutions are applied to the input single image, and n1 feature maps are output, where the size of each convolution kernel is c×f1×f1.

[0102] Similarly, for convolutional layers Conv1-2 and Conv2-2, Conv1-2 inputs the output of Conv1-1, and Conv2-2 inputs the output of Conv2-1. Both convolutional layers pass through n2 filters of size f2×f2 and output n2 feature maps. Similarly, convolutional layers Conv1-3 and Conv2-3 output n3 feature maps.

[0103] Exemplarily, the ReLU activation function is defined as follows:

[0104] ReLU(x)=max(0,x)

[0105] It should be noted that the ReLU activation function can increase the nonlinearity of the convolutional network and help the model learn more complex features. This is because the output of the ReLU activation function remains unchanged for positive inputs, but outputs 0 for negative inputs. This nonlinear characteristic alleviates the gradient vanishing problem and accelerates the training process.

[0106] Step 105, through the convolution layer and multi-scale residual module in the dual-stream deep convolutional neural network model, feature extraction and feature fusion are performed on the low-resolution depth image features and the high-resolution color image edge features at different scales to obtain a reconstructed high-resolution depth image.

[0107] It should be noted that the residual module is an important structure in deep learning. It solves the gradient vanishing and gradient exploding problems in deep neural network training by introducing skip connections or identity mapping. The core idea of ​​the residual module is to let the network learn the residual between the input and output, that is, the difference part, instead of directly learning the mapping relationship. The residual module usually consists of two or three convolutional layers, including a batch normalization layer (BNL) and an activation function. In the residual module, the input is not only processed by these convolutional layers, but also passed directly to the output through skip connections.

[0108] In this embodiment, the multi-scale residual module includes multiple convolutional layers and activation functions. Each convolutional layer is a convolution kernel of a different scale. The data input to the residual module first passes through a convolutional layer, and then enters the second convolutional layer after activation. The result is added to the original input and activated again to obtain the final output.

[0109] In some embodiments, the two-stream deep convolutional neural network model includes a feature fusion convolutional network, which includes a first convolutional layer, a multi-scale residual module, a second convolutional layer and a deconvolutional layer.

[0110] In some embodiments, the feature fusion convolutional network may also be referred to as a "third convolutional network", which includes three convolutional layers Conv3-1, Conv3-2 and Conv3-3.

[0111] In summary, in this embodiment, on the one hand, through the dual-stream deep convolutional neural network model, the depth image and the color image are subjected to feature extraction, feature fusion and reconstruction to obtain a high-resolution depth image. Due to the use of a feedforward network model, a large number of training samples and feature engineering are not required, and the complementary information of the color image and the depth map can be fully combined to effectively reduce the texture duplication phenomenon and the edge discontinuity effect. On the other hand, on the basis of the deep neural network, a multi-scale feature extraction method combining multi-residuals is used to extract multi-scale features of the image from different levels. The multi-scale residual module can better fuse and denoise the extracted features, thereby improving the reconstruction effect of the super-resolution depth image.

[0112] Figure 2 is a flowchart of the specific steps of a super-resolution reconstruction method of an image provided by an embodiment of the present application, referring to Figure 2 The super-resolution reconstruction method of the above image may include the following steps:

[0113] Step 201: Acquire a depth image and a color image of the same object.

[0114] The method of this step has been described in the aforementioned step 101 and will not be repeated here.

[0115] Step 202: extract edges from the color image to obtain a first edge map, and extract edges from the upsampled depth image to obtain a second edge map.

[0116] The method of this step has been described in the aforementioned step 102 and will not be repeated here.

[0117] Exemplarily, step 202 may include: using a Canny operator to extract edges from a color image to obtain a first edge map, and using a Canny operator to extract edges from an upsampled depth image to obtain a second edge map.

[0118] Step 203: Perform a logical “AND” operation on the first edge map and the second edge map to obtain an optimized edge map.

[0119] In this embodiment, the logic “AND” operation may include: the result is 1 only when both operands are 1, and is 0 in other cases.

[0120] Through the above technical means, the extracted edge map is optimized, which improves the display effect of the edge area of ​​the image, which is conducive to improving the accuracy of subsequent reconstruction of super-resolution depth images.

[0121] In some embodiments, before step 203, the method further includes: mapping the depth image and the color image to the same coordinate system; making one-to-one correspondence between pixels having the same row and column positions in the depth image and the color image; and performing cubic bilinear interpolation on the depth image so that the depth image is upsampled to the pixel size of the color image.

[0122] Exemplarily, performing cubic bilinear interpolation on the depth image so that the depth image is upsampled to the pixel size of the color image may include: using the original high-resolution depth image as a label, downsampling the depth image in the Middlebury dataset by 2 times, 4 times, and 8 times, and then re-upsampling it to the pixel size of the original depth image.

[0123] Through the above technical means, the depth image is upsampled to the pixel size of the color image, which provides the prerequisite for subsequent feature extraction, feature fusion and reconstruction of super-resolution depth image.

[0124] Step 204, performing feature extraction on the upsampled depth image and the optimized edge map through the convolutional network in the two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features.

[0125] The method of this step has been described in the aforementioned step 104 and will not be repeated here.

[0126] Step 205: Fusing the low-resolution depth image features and the high-resolution color image edge features through the first convolutional layer.

[0127] In this embodiment, the dual-stream deep convolutional neural network model includes a feature fusion convolutional network, which includes a first convolutional layer, a multi-scale residual module, a second convolutional layer and a deconvolutional layer.

[0128] In some embodiments, the feature fusion convolutional network may also be referred to as a "third convolutional network", which includes three convolutional layers Conv3-1, Conv3-2 and Conv3-3.

[0129] Exemplarily, the first convolutional layer refers to the convolutional layer Conv3-1, which is a convolutional layer with a convolution kernel size of 9×9.

[0130] Step 206, extracting the low-resolution depth image features and the high-resolution color image edge features at different scales through a multi-scale residual module, and obtaining feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features.

[0131] The method of this step has been described in the aforementioned step 105 and will not be repeated here.

[0132] In this embodiment, the multi-scale residual module includes a first residual module and a second residual module.

[0133] For example, step 206 may include the following sub-steps:

[0134] Sub-step 2061, inputting the fused feature map output by the first convolutional layer into the first residual module, and the first residual module is jump-connected to the second residual module.

[0135] The first residual module and the second residual module extract features at different scales.

[0136] It should be noted that the residual module is an important structure in deep learning. It solves the gradient vanishing and gradient exploding problems in deep neural network training by introducing skip connections or identity mapping. The core idea of ​​the residual module is to let the network learn the residual between the input and output, that is, the difference part, instead of directly learning the mapping relationship. The residual module usually consists of two or three convolutional layers, including a batch normalization layer (BNL) and an activation function. In the residual module, the input is not only processed by these convolutional layers, but also passed directly to the output through skip connections.

[0137] In this embodiment, the output of the first residual module can be directly passed to the subsequent layer to retain the low-level feature information. The second residual module is similar to the first residual module and has the same characteristics. The two residual modules (Res-block) can utilize the ideas of recursion and residual learning to reduce the burden of the two-stream deep convolutional neural network model and control the parameters of the convolutional network.

[0138] Sub-step 2062, inputting the output result of the first residual module into the second residual module, and obtaining feature maps of different scales after non-linear mapping from low-resolution features to high-resolution features.

[0139] In this embodiment, the first residual module and the second residual module are used to nonlinearly map low-resolution features to high-resolution features, and the output result of the second residual module is a feature map of different scales after nonlinear mapping from low-resolution features to high-resolution features.

[0140] In some embodiments, the output result of the second residual module is generated based on recursive and residual learning, classification or regression methods.

[0141] Through the above technical means, it is possible to ensure smoother information flow and avoid information loss and gradient problems when the number of convolutional network layers increases.

[0142] Step 207, the feature map mapped to the high-resolution feature is fused through the second convolutional layer.

[0143] Exemplarily, the second convolutional layer refers to convolutional layer Conv3-2.

[0144] Optionally, the second convolutional layer inputs the output result of the second residual module, passes through n5 filters of size f5×f5, and outputs n5 feature maps. The convolutional layer Conv3-2 is used for high-resolution feature fusion. The convolution kernel size of the convolutional layer Conv3-2 is 1×1, and its input comes from the feature map after nonlinear mapping from low-resolution features to high-resolution features.

[0145] For example, Figure 3 This is a network framework diagram of a multi-scale residual module provided in an embodiment of the present application, see Figure 3 First, the convolution layer Conv3-1 outputs the fused feature map and inputs it into the first residual module. Secondly, the first residual module is connected to the second residual module through the first jump connection. Thirdly, the second residual module is connected to the output layer through the first jump connection. The output layer generates a feature map after high-resolution features are generated based on recursive and residual learning, classification or regression methods. Finally, the feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features are output to the convolution layer Conv3-2 through the output layer.

[0146] Step 208, reconstructing the super-resolution image using the fusion results of the deconvolution layer and the second convolution layer to obtain a reconstructed high-resolution depth image.

[0147] In this embodiment, the second convolutional layer and the deconvolutional layer are used for two-stream convolutional network reconstruction.

[0148] Exemplarily, the deconvolution layer refers to the convolution layer Conv3-3, the convolution kernel size of the convolution layer Conv3-3 is 5×5, there is no activation function, and the final high-resolution depth image is output.

[0149] Through the above technical means, the use of a two-stream deep convolutional neural network model will not damage the original information of the depth image and has a better super-resolution reconstruction effect.

[0150] In some embodiments, the above-mentioned image super-resolution reconstruction method further includes the following sub-steps:

[0151] Sub-step A1, constructing the target loss function of the two-stream deep convolutional neural network model;

[0152] Sub-step A2, train the two-stream deep convolutional neural network model through the target loss function.

[0153] For example, sub-step A1 may include the following sub-steps:

[0154] Sub-step B1, obtaining the mean square error function of the two-stream deep convolutional neural network model.

[0155] It should be noted that mean-square error (MSE) is a measure of the difference between the estimated value and the estimated value. The application scenarios of mean square error are mostly in image reconstruction, image compression, and image super-resolution. It has the characteristics of simplicity, intuition, global error, and optimization objectives. Among them, simple and intuitive: MSE evaluates image quality by calculating the sum of the squares of the pixel differences between the predicted image and the real image. It is a very intuitive and simple evaluation method; global error: MSE reflects the error of the entire image, but is insensitive to changes in local features, which sometimes causes the image to look blurred; optimization objective: MSE is often used to optimize the objective function, especially in regression tasks.

[0156] Exemplarily, the mean square error formula is as follows:

[0157]

[0158] Among them, θ represents the weight parameter of the convolution kernel, N represents the number of samples during training, represents the initial upsampled depth image, represents the optimized depth image, E i represents the optimized edge graph, Represents the training set.

[0159] Sub-step B2, constructing a total variation loss function of the depth image according to the size and gradient of the depth image.

[0160] It should be noted that the total variation loss (TV loss) is a loss function used in image processing, which measures the degree of difference between adjacent pixels in the image. The TV loss function is mostly used in image denoising, image restoration, and image enhancement. The TV loss function has the characteristics of smoothness, structure preservation, and structure preservation; among them. Smoothness: The TV loss function is mainly used to reduce noise and smooth edges in the image, making the image smoother. It achieves this by integrating the absolute value of the image gradient; Structure preservation: The TV loss function helps to maintain the structural characteristics of the image, especially retaining edge information while removing noise; Structure preservation: The TV loss function has a certain inhibitory effect on noise and is suitable for denoising and image restoration tasks.

[0161] Exemplarily, the formula of TV loss function is as follows:

[0162]

[0163] in They represent the gradients in the directions of the depth image, M·N represents the size of the depth image, and i represents the subscript of each pixel.

[0164] Sub-step B3, constructing a total variation gradient loss function of the color image according to the size of the depth image, the color image and the first edge map.

[0165] It should be noted that the quality of acquiring color images and depth images is not only affected by the performance of the camera itself, but also by environmental factors such as light, temperature, complexity of reconstruction and color information during the acquisition process. At the same time, most super-resolution convolutional neural network frameworks in the convolutional layer processing process only stack convolution kernels of a single size, which cannot extract favorable guiding information from high-resolution color images in the same scene. The dual-stream deep convolutional neural network model provided in this application can extract color edge information, thereby adding weight factors to the evaluation of the TV loss gradient function of the color image.

[0166] Exemplarily, the formula of the total variation gradient loss function is as follows:

[0167]

[0168] Among them, M·N represents the size of the depth map; C g is a real and clear RGB image, and C is the RGB image after edge extraction. represents the square of the 2-norm.

[0169] Sub-step B4, obtaining the target loss function of the two-stream deep convolutional neural network model according to the mean square error function, the total variation loss function, the total variation gradient loss function and the preset weight ratio.

[0170] It should be noted that since the error of color images is greater in scenes that are greatly affected by light, and the error of depth images is greater in data stream scenes, different weight ratios are set for the mean square error function, total variation loss function, and total variation gradient loss function according to the impact on the final image resolution. The weight ratio can be obtained in advance through a large number of experiments, and the weight parameters can be modified according to different application scenarios.

[0171] Exemplarily, the formula of the objective loss function is as follows:

[0172] L=λ1L(θ)+λ2L tv +λ3L cgrad

[0173] Among them, λ3 decreases in the scene with greater influence of light, and λ1 increases in the data stream scene, and the sum of the weight values ​​of λ1, λ2 and λ3 is 1.

[0174] It should be noted that this embodiment evaluates the final quality of the image by combining a simple TV loss function and a mean square error. The simple TV loss function and the mean square error each have their own advantages. Their combined use can give full play to their complementarity and improve the performance of image processing tasks. By combining the structure preservation and noise resistance of the TV loss function and the mean square global error evaluation of the MSE, better results can be achieved in tasks such as image denoising, restoration, reconstruction and generation. This method has the following advantages: 1. Complementary performance: structure preservation and detail recovery: The TV loss function performs well in maintaining image structure and edge information, while the MSE is simple and intuitive in global error evaluation. Combining the two can simultaneously maintain the structural features and global consistency of the image. 2. Noise resistance and smoothness: The TV loss function helps to denoise and smooth the image, while the MSE can ensure that the overall error of the image is minimized. The combined use can better handle noise problems while maintaining the clarity of the image. 3. Optimization effect: During the optimization process, the MSE can help reduce the global error, while the TV loss function can prevent the model from over-smoothing the image, thereby achieving a better balance in the optimization process.

[0175] Through the above technical means, the combination of TV loss function and MSE at the image restoration level can more effectively remove noise from the image while retaining edge and texture information; at the image super-resolution processing level, in the super-resolution task, the combination of the two can improve the image resolution while maintaining the structure and details of the image; at the image generation level: when generating adversarial networks such as SRCNN models, combining TV loss function and MSE can improve the quality of generated images, making the obtained images closer to real images.

[0176] In a possible implementation, sub-step A2 may include the following sub-steps:

[0177] Sub-step B5, obtaining the loss value of the target loss function;

[0178] Sub-step B6, when the loss value is greater than a preset loss threshold, updating the preset model parameters according to the loss value to obtain a target neural network model;

[0179] Sub-step B7, until the loss value is less than or equal to the preset loss threshold, and the target neural network model corresponding to the loss value is used as the dual-stream deep convolutional neural network model.

[0180] In this embodiment, the weights in the network can be updated by calculating the gradient through back propagation to minimize the loss function.

[0181] It should be noted that the goal of backpropagation is to find the partial derivatives of the loss function with respect to each parameter and use these gradients to adjust the model parameters. The steps of backpropagation may include: 1. Calculate the gradient of the loss function: For each parameter, calculate the partial derivative (gradient) of the loss function with respect to it. 2. Weight update: Use the optimizer to update the value of each parameter based on the gradient information.

[0182] In this embodiment, the preset loss threshold is obtained based on experiments and can be set according to actual needs.

[0183] Through the above technical means, the TV loss gradient function is set for the color image and the depth image respectively, the network parameters are iteratively updated to optimize the loss function, and a two-stream deep convolutional neural network model is trained to effectively improve the edge quality of the depth image, solve the problem of blurred edges of the depth image, and facilitate the reconstruction of high-resolution images with realistic detailed textures.

[0184] Figure 4 It is a network framework diagram of a two-stream deep convolutional neural network model provided in an embodiment of the present application. The two-stream deep convolutional neural network model may include a first convolutional network, a second convolutional network and a third convolutional network.

[0185] In the two-stream deep convolutional neural network model, the first convolutional network includes convolutional layers Conv1-1, Convolutional layers Conv1-2, and Convolutional layers Conv1-3; the second convolutional network includes convolutional layers Conv2-1, Convolutional layers Conv2-2, and Convolutional layers Conv2-3; and the third convolutional network includes convolutional layers Conv3-1, a first residual module, a second residual module, convolutional layers Conv3-2, and convolutional layers Conv3-3.

[0186] In the above network framework, first, Conv1-1, convolution layer Conv1-2, and convolution layer Conv1-3 are used to extract features from the input upsampled depth image to obtain low-resolution depth image features. Secondly, Conv2-1, convolution layer Conv2-2, and convolution layer Conv2-3 are used to extract features from the optimized edge map after color feature extraction and optimization to obtain high-resolution color image edge features. Thirdly, Conv1-3 and Conv2-3 output low-resolution depth image features and high-resolution color image edge features to convolution layer Conv3-1 respectively. Convolution layer Conv3-1 is used to fuse low-resolution depth image features and high-resolution color image edge features to obtain a fused feature map, and output the fused feature map to the first residual module; the first residual module and the second residual module are used to nonlinearly map low-resolution features to high-resolution features, and output the mapping results to convolution layer Conv3-2. Finally, the convolution layer Conv3-2 is used for high-resolution feature fusion, and the convolution layer Conv3-3 outputs the final high-resolution depth image to complete the two-stream convolutional network reconstruction.

[0187] In this embodiment, through the above network architecture, first, it is possible to fully combine the complementary information of color images and depth, and extract multi-scale features of images from different levels based on the multi-scale feature extraction method combining SCNN and multi-residuals. The residual block will better fuse and denoise the extracted features, and the improved architecture effect is evaluated from the mean square error caused by the combined architecture. Second, the structure is simple, the number of filters and layers is moderate, and the running speed is relatively fast even on the central processing unit (CPU) because it is a feedforward network and there is no need to worry about optimization issues when using it. Third, the above two-stream deep convolutional neural network model can perform feature extraction and feature fusion at the same time. The two tasks are independent and can improve their respective performance ceilings. This method combines the information of depth images with color images to effectively reduce texture duplication and edge discontinuity effects.

[0188] refer to Figure 5 , which shows a super-resolution reconstruction device 30 for an image provided by an embodiment of the present application, the super-resolution reconstruction device 30 for an image comprises:

[0189] An acquisition module 301 is used to acquire a depth image and a color image of the same object;

[0190] The edge extraction module 302 is used to extract edges from the color image to obtain a first edge map, and to extract edges from the upsampled depth image to obtain a second edge map;

[0191] An edge fusion module 303 is used to fuse the first edge map and the second edge map to obtain an optimized edge map;

[0192] A first model application module 304 is used to extract features from the upsampled depth image and the optimized edge map through a convolutional network in a two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features;

[0193] The second model application module 305 is used to extract and fuse low-resolution depth image features and high-resolution color image edge features at different scales through the convolutional layers and multi-scale residual modules in the dual-stream deep convolutional neural network model to obtain a reconstructed high-resolution depth image.

[0194] Optionally, the two-stream deep convolutional neural network model includes a feature fusion convolutional network, which includes a first convolutional layer, a multi-scale residual module, a second convolutional layer and a deconvolutional layer; the second model application module 305 includes:

[0195] A first edge fusion submodule, used for fusing low-resolution depth image features and high-resolution color image edge features through a first convolutional layer;

[0196] A feature extraction submodule is used to extract features of low-resolution depth images and edge features of high-resolution color images at different scales through a multi-scale residual module, and obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features;

[0197] A first feature fusion submodule is used to fuse the feature map mapped to the high-resolution feature through the second convolutional layer;

[0198] The image reconstruction submodule reconstructs the super-resolution image using the fusion results of the deconvolution layer and the second convolution layer to obtain a reconstructed high-resolution depth image.

[0199] Optionally, the multi-scale residual module includes a first residual module and a second residual module; and the feature extraction submodule includes:

[0200] A first feature extraction unit is used to input the fused feature map output by the first convolutional layer into a first residual module, the first residual module is jump-connected to the second residual module, and the first residual module and the second residual module have different scales of feature extraction;

[0201] The second feature extraction unit is used to input the output result of the first residual module into the second residual module to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features.

[0202] Optionally, the image super-resolution reconstruction device 30 further includes:

[0203] A first mapping module, used for mapping the depth image and the color image to the same coordinate system;

[0204] A second mapping module, used for making one-to-one correspondence between pixels having the same row and column positions in the depth image and the color image;

[0205] The sampling module is used to perform cubic double interpolation on the depth image so that the depth image is upsampled to the pixel size of the color image.

[0206] Optionally, the edge fusion module 303:

[0207] The edge fusion submodule is used to perform a logical "AND" operation on the first edge map and the second edge map to obtain an optimized edge map.

[0208] Optionally, the image super-resolution reconstruction device 30 further includes:

[0209] The second acquisition module is used to obtain the mean square error function of the two-stream deep convolutional neural network model;

[0210] The first function construction module is used to construct a total variation loss function of the depth image according to the size and gradient of the depth image;

[0211] The second function construction module is used to construct a total variation gradient loss function of the color image according to the size of the depth image, the color image and the first edge map;

[0212] A third function construction module is used to obtain a target loss function of a two-stream deep convolutional neural network model according to a mean square error function, a total variation loss function, a total variation gradient loss function and a preset weight ratio;

[0213] Model training module, used to train a two-stream deep convolutional neural network model through an objective loss function.

[0214] Optionally, the model training module is used to include:

[0215] Get the submodule, which is used to obtain the loss value of the target loss function;

[0216] An updating submodule, used to update the preset model parameters according to the loss value to obtain the target neural network model when the loss value is greater than the preset loss threshold;

[0217] The training submodule is used to train the target neural network model corresponding to the loss value until the loss value is less than or equal to the preset loss threshold, and use the target neural network model corresponding to the loss value as the dual-stream deep convolutional neural network model.

[0218] In summary, in this embodiment, on the one hand, through the dual-stream deep convolutional neural network model, the depth image and the color image are subjected to feature extraction, feature fusion and reconstruction to obtain a high-resolution depth image. Due to the use of a feedforward network model, a large number of training samples and feature engineering are not required, and the complementary information of the color image and the depth map can be fully combined to effectively reduce the texture duplication phenomenon and the edge discontinuity effect. On the other hand, on the basis of the deep neural network, a multi-scale feature extraction method combining multi-residuals is used to extract multi-scale features of the image from different levels. The multi-scale residual module can better fuse and denoise the extracted features, thereby improving the reconstruction effect of the super-resolution depth image.

[0219] Reference Figure 6 , the electronic device 400 may include one or more of the following components: a processing component 402 , a memory 404 , a power component 406 , a multimedia component 408 , an audio component 410 , an input / output (I / O) interface 412 , a sensor component 414 , and a communication component 416 .

[0220] The processing component 402 generally controls the overall operation of the electronic device 400, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 402 may include one or more modules to facilitate the interaction between the processing component 402 and other components. For example, the processing component 402 may include a multimedia module to facilitate the interaction between the multimedia component 408 and the processing component 402.

[0221] The memory 404 is used to store various types of data to support the operation of the electronic device 400. Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phone book data, messages, pictures, multimedia, etc. The memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0222] The power supply component 406 provides power to the various components of the electronic device 400. The power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 400.

[0223] The multimedia component 408 includes an interface that provides an output interface between the electronic device 400 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 408 includes a front camera and / or a rear camera. When the electronic device 400 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0224] The audio component 410 is used to output and / or input audio signals. For example, the audio component 410 includes a microphone (MIC), and when the electronic device 400 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 404 or sent via the communication component 416. In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

[0225] The input / output I / O interface 412 provides an interface between the processing component 402 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0226] The sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for the electronic device 400. For example, the sensor assembly 414 can detect the open / closed state of the electronic device 400, the relative positioning of components, such as the display and keypad of the electronic device 400, and the sensor assembly 414 can also detect the position change of the electronic device 400 or a component of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration / deceleration of the electronic device 400, and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 414 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0227] The communication component 416 is used to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0228] In an exemplary embodiment, the electronic device 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement a super-resolution image reconstruction method provided in an embodiment of the present application.

[0229] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, and the instructions can be executed by a processor 420 of an electronic device 400 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0230] Figure 7 is a block diagram of an electronic device 500 according to another embodiment of the present invention. For example, the electronic device 500 may be provided as a server. Figure 7 The electronic device 500 includes a processing component 522, which further includes one or more processors, and a memory resource represented by a memory 532 for storing instructions executable by the processing component 522, such as an application. The application stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 522 is configured to execute instructions to perform a super-resolution reconstruction method of an image provided in an embodiment of the present application.

[0231] In an embodiment of the present application, the memory 532 may be used to store software programs and various data. The memory 532 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 532 may include a volatile memory or a non-volatile memory, or the memory 532 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 532 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0232] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0233] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image super-resolution reconstruction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0234] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0235] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the super-resolution reconstruction method embodiment of the above-mentioned image, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0236] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0237] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0238] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A super-resolution reconstruction model training method for an image, characterized in that: The method comprises: Get depth image and color image of the same object; Extracting edges from the color image to obtain a first edge map, and extracting edges from the upsampled depth image to obtain a second edge map; fusing the first edge map and the second edge map to obtain an optimized edge map; Performing feature extraction on the upsampled depth image and the optimized edge map through a convolutional network in a two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features; Through the convolutional layers and multi-scale residual modules in the dual-stream deep convolutional neural network model, feature extraction and feature fusion are performed on the low-resolution depth image features and the high-resolution color image edge features at different scales to obtain a reconstructed high-resolution depth image.

2. The method according to claim 1, characterized in that: The two-stream deep convolutional neural network model includes a feature fusion convolutional network, and the feature fusion convolutional network includes a first convolutional layer, a multi-scale residual module, a second convolutional layer and a deconvolutional layer; The method extracts and fuses the low-resolution depth image features and the high-resolution color image edge features at different scales through the convolutional layer and the multi-scale residual module in the dual-stream deep convolutional neural network model to obtain a reconstructed high-resolution depth image, including: Fusing the low-resolution depth image features and the high-resolution color image edge features through the first convolutional layer; By means of the multi-scale residual module, the low-resolution depth image features and the high-resolution color image edge features are subjected to feature extraction at different scales, so as to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features; fusing the feature maps mapped to the high-resolution features through the second convolutional layer; The super-resolution image is reconstructed using the fusion results of the deconvolution layer and the second convolution layer to obtain a reconstructed high-resolution depth image.

3. The method according to claim 1, characterized in that The multi-scale residual module includes a first residual module and a second residual module; The multi-scale residual module is used to extract the low-resolution depth image features and the high-resolution color image edge features at different scales to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features, including: Inputting the fused feature map output by the first convolutional layer into the first residual module, the first residual module is jump-connected to the second residual module, and the scales of feature extraction of the first residual module and the second residual module are different; The output result of the first residual module is input into the second residual module to obtain feature maps of different scales after nonlinear mapping from low-resolution features to high-resolution features.

4. The method according to claim 1, characterized in that Before extracting edges from the color image to obtain a first edge map, and extracting edges from the upsampled depth image to obtain a second edge map, the method further includes: Mapping the depth image and the color image to the same coordinate system; Matching pixels having the same row and column positions in the depth image and the color image one by one; The depth image is subjected to cubic bi-interpolation so that the depth image is upsampled to a pixel size of the color image.

5. The method according to claim 1, characterized in that The step of fusing the first edge map and the second edge map to obtain an optimized edge map includes: Performing a logical AND operation on the first edge map and the second edge map to obtain the optimized edge map.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a mean square error function of the two-stream deep convolutional neural network model; Constructing a total variation loss function of the depth image according to the size and gradient of the depth image; Constructing a total variation gradient loss function of the color image according to the size of the depth image, the color image and the first edge map; Obtaining a target loss function of the dual-stream deep convolutional neural network model according to the mean square error function, the total variation loss function, the total variation gradient loss function and a preset weight ratio; The two-stream deep convolutional neural network model is trained by the target loss function.

7. The method according to claim 6, characterized in that The training of the dual-stream deep convolutional neural network model by the target loss function includes: Obtaining a loss value of the objective loss function; When the loss value is greater than a preset loss threshold, updating the preset model parameters according to the loss value to obtain a target neural network model; Until the loss value is less than or equal to the preset loss threshold, and the target neural network model corresponding to the loss value is used as the dual-stream deep convolutional neural network model.

8. A super-resolution reconstruction device for an image, characterized in that: The device comprises: An acquisition module, used for acquiring a depth image and a color image of the same object; An edge extraction module, configured to extract edges from the color image to obtain a first edge map, and to extract edges from the upsampled depth image to obtain a second edge map; An edge fusion module, used for fusing the first edge map and the second edge map to obtain an optimized edge map; A first model application module is used to extract features of the upsampled depth image and the optimized edge map through a convolutional network in a two-stream deep convolutional neural network model to obtain low-resolution depth image features and high-resolution color image edge features; The second model application module is used to extract and fuse the low-resolution depth image features and the high-resolution color image edge features at different scales through the convolutional layers and multi-scale residual modules in the dual-stream deep convolutional neural network model to obtain a reconstructed high-resolution depth image.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: When the instructions or transactions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method for super-resolution reconstruction of an image according to any one of claims 1 to 7.

11. A computer program product, characterized in that The method comprises instructions or transactions, which, when executed by a processor in an electronic device, cause the electronic device to execute the method for super-resolution reconstruction of an image according to any one of claims 1 to 7.

Images