An image processing method, apparatus and electronic device

Abstract

The application discloses an image processing method and device and electronic equipment, and relates to the field of information technology, to solve the problem of poor denoising effect of the existing model. The method comprises the following steps: performing encoding processing on a first image to obtain a feature map of the first image; determining the feature points corresponding to each pixel point in the first image according to the position information of each pixel point in the first image and the position information of each feature point in the feature map; and performing decoding processing on the feature map according to the relative positions of each pixel point and the corresponding feature point in the first image to obtain a denoised first image. By introducing the position information of the input image, the spatial structure of the input feature map can be better understood, and thus a better image denoising effect can be obtained.

Description

Technical Field

[0001] This application relates to the field of information technology, and in particular to an image processing method, apparatus and electronic device. Background Technology

[0002] In existing 3D face recognition technology, depth face data is usually used for face recognition. However, the depth image data acquired by depth cameras often contains a lot of noise, which can affect the learning and generalization ability of the network and even interfere with the final recognition result.

[0003] To address this issue, existing technologies have proposed a 3D Feature Refinement Module (3D-FRM) based on an encoder-decoder architecture for denoising and refining deep face data to improve its quality. However, this model is entirely composed of traditional convolutional neural networks, which have poor spatial awareness, resulting in inadequate denoising performance. Summary of the Invention

[0004] This application provides an image processing method, apparatus, and electronic device to solve the problem of poor denoising effect in existing models.

[0005] In a first aspect, embodiments of this application provide an image processing method, including:

[0006] The first image is encoded to obtain a feature map of the first image;

[0007] Based on the position information of each pixel in the first image and the position information of each feature point in the feature map, the feature points corresponding to each pixel in the first image are determined.

[0008] Based on the relative positions of each pixel in the first image and its corresponding feature points, the feature map is decoded to obtain the denoised first image.

[0009] Optionally, determining the feature point corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map includes:

[0010] Based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, the distances between the position of the first pixel and the positions of each feature point in the feature map are determined, wherein the first pixel is any pixel in the first image;

[0011] The feature point in the feature map that is closest to the position of the first pixel is determined as the feature point corresponding to the first pixel.

[0012] Optionally, the step of decoding the feature map based on the relative positions of each pixel in the first image and its corresponding feature points to obtain the denoised first image includes:

[0013] Determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image;

[0014] The first relative position coordinates are concatenated with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the concatenated feature vector corresponding to the first pixel.

[0015] The concatenated feature vectors corresponding to each pixel in the first image are decoded to obtain the denoised first image.

[0016] Optionally, decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes:

[0017] The concatenated feature vectors corresponding to each pixel in the first image are input into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

[0018] Optionally, the first image is a face depth image;

[0019] Decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes:

[0020] The concatenated feature vectors corresponding to each pixel in the face depth image are decoded to obtain the depth value of each pixel in the face depth image, and the depth values ​​of each pixel constitute the denoised face depth image.

[0021] Optionally, the encoding process of the first image to obtain a feature map of the first image includes:

[0022] The first image is input into an encoding network, wherein the encoding network includes a first convolutional layer and M cascaded downsampling modules. Each downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer.

[0023] Feature extraction is performed on the first image through the first convolutional layer;

[0024] The image features of the first image extracted by the first convolutional layer are processed by the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

[0025] Optionally, the downsampling module further includes a feature connection layer for adding the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

[0026] Secondly, embodiments of this application also provide an image processing apparatus, comprising:

[0027] The encoding module is used to encode the first image to obtain a feature map of the first image;

[0028] The first determining module is used to determine the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map.

[0029] The decoding module is used to decode the feature map according to the relative position of each pixel in the first image and the corresponding feature point to obtain the denoised first image.

[0030] Optionally, the first determining module includes:

[0031] The first determining unit is configured to determine the distances between the position of the first pixel and the positions of each feature point in the feature map based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, wherein the first pixel is any pixel in the first image.

[0032] The second determining unit is used to determine the feature point in the feature map whose position is closest to the first pixel as the feature point corresponding to the first pixel.

[0033] Optionally, the decoding module includes:

[0034] The third determining unit is used to determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image;

[0035] The splicing unit is used to splice the first relative position coordinates with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the spliced ​​feature vector corresponding to the first pixel.

[0036] The decoding unit is used to decode the spliced ​​feature vectors corresponding to each pixel in the first image to obtain the denoised first image.

[0037] Optionally, the decoding unit is used to input the concatenated feature vectors corresponding to each pixel in the first image into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

[0038] Optionally, the first image is a face depth image;

[0039] The decoding unit is used to decode the spliced ​​feature vector corresponding to each pixel in the face depth image to obtain the depth value of each pixel in the face depth image, and the depth value of each pixel constitutes the denoised face depth image.

[0040] Optionally, the encoding module is an encoding network, which includes a first convolutional layer and M cascaded downsampling modules. The downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer.

[0041] The encoded network is used to extract features from the first image through the first convolutional layer, and to process the image features of the first image extracted by the first convolutional layer through the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

[0042] Optionally, the downsampling module further includes a feature connection layer for adding the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

[0043] Thirdly, embodiments of this application also provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in the image processing method described above.

[0044] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the image processing method described above.

[0045] In this embodiment, the first image is encoded to obtain a feature map of the first image; based on the position information of each pixel in the first image and the position information of each feature point in the feature map, the feature points corresponding to each pixel in the first image are determined; based on the relative positions of each pixel in the first image and the corresponding feature points, the feature map is decoded to obtain a denoised first image. Thus, by introducing the position information of the input image, it is possible to better understand the spatial structure of the input feature map, thereby obtaining a better image denoising effect. Attached Figure Description

[0046] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This is a flowchart of the image processing method provided in the embodiments of this application;

[0048] Figure 2 This is a general framework diagram of the face denoising and recognition scheme provided in the embodiments of this application;

[0049] Figure 3 This is a structural diagram of the denoising network provided in the embodiments of this application;

[0050] Figure 4 This is a structural diagram of the image encoder network provided in the embodiments of this application;

[0051] Figure 5 This is a structural diagram of the residual block provided in an embodiment of this application;

[0052] Figure 6 This is a structural diagram of the feature decoder provided in the embodiments of this application;

[0053] Figure 7 This is a structural diagram of the image processing apparatus provided in the embodiments of this application;

[0054] Figure 8 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] To make the embodiments of this application clearer, the relevant technical knowledge involved in the embodiments of this application will be introduced as follows:

[0057] 3D faces are defined in contrast to 2D faces. A 2D image is essentially a projection of a 3D object onto a 2D plane. Because the human face inherently possesses 3D attributes, using 3D face data for face modeling is more advantageous. Currently, the most common method for acquiring 3D face data is to directly utilize 3D imaging equipment, which primarily includes high-precision 3D scanners and low-precision RGB-D cameras.

[0058] Deep learning is a machine learning technique that uses multi-layered neural networks to learn and represent complex data, especially unstructured data such as speech, images, and videos. Deep learning plays a crucial role in face-related tasks, using convolutional neural networks to extract features and classify facial images, improving recognition accuracy and speed. Through training on large amounts of data, deep learning models can recognize various details in a face, such as facial contours, eyes, mouth, and nose.

[0059] Implicit neural representation is a novel method for parameterizing various signals. Traditional signal representations are typically discrete; for example, two-dimensional image signals are discrete grids of pixels, one-dimensional audio signals are discrete samples of amplitude, and three-dimensional objects are usually parameterized as voxels, point clouds, or triangular grids. The general idea of ​​implicit neural representation is to use a continuous function to represent the signal and to approximate this function using a neural network. Generally, this neural network is a multi-layered feedforward neural network, where each layer consists of a set of parameters. Through the backpropagation algorithm, the parameters of the neural network can be adjusted based on the prediction error, thereby optimizing the prediction accuracy and generalization ability. This method can transform the input two-dimensional image into the weights of an implicit function without requiring explicit representation. Embodiments of this application can use implicit neural representation technology to achieve denoising and thinning of low-quality deep face data.

[0060] To address the issue of coarse and noisy data acquired by depth sensors, some researchers have proposed a 3D Feature Refinement Module (3D-FRM) model for face denoising and thinning, which can effectively improve the quality of noisy depth faces. 3D-FRM has an encoder-decoder structure, where the encoder applies depth-separable convolutions to capture raw discriminative information by fusing features from different receptive fields. The decoder utilizes sub-pixel convolutions and multi-scale feature fusion to achieve face shape restoration.

[0061] In the encoder section, 3D-FRM first uses two convolutional layers with a kernel size of 3×3 and 8 channels to extract complementary features from the input depth map and normal map of the face. Then, four downsampling blocks are used to encode the input features to reduce noise and preserve facial shape details; these four downsampling blocks have output channels of 16, 32, 64, and 128, respectively. The decoder section uses four upsampling blocks to progressively restore resolution; these four upsampling blocks have output channels of 64, 32, 16, and 8, respectively. Finally, a 1×1 convolutional layer with 1 output channel is used, followed by a tangent activation function layer to obtain the final denoised depth map.

[0062] The encoder and decoder of 3D-FRM are entirely composed of convolutional neural networks. Although convolutional neural networks are very suitable for image processing tasks and have the characteristics of parameter sharing, local connectivity, and translation invariance, traditional convolutional neural networks ignore the coordinate information of the input image and lack spatial awareness, resulting in poor denoising performance.

[0063] Because depth-sensor-acquired face data often contains significant noise, it can affect the learning and generalization abilities of the network, and even interfere with the final recognition results. Therefore, this application proposes using the latest implicit neural representation technology, combined with the location information of the input image, to model the relationship between the feature map feature vectors and their coordinates and the output depth value using a decoder. This imbues the denoising network with spatial awareness, enabling better denoising and thinning of low-quality depth-sensor face data. The denoised and thinned face depth map is then fed into the recognition network, achieving more accurate and robust 3D face recognition.

[0064] The image processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0065] See Figure 1 , Figure 1This is a flowchart of the image processing method provided in the embodiments of this application, such as... Figure 1 As shown, it includes the following steps:

[0066] Step 101: Encode the first image to obtain the feature map of the first image.

[0067] The first image mentioned above can be any image that needs to be denoised, especially a face image whose image quality needs to be improved. For example, it can be the original face depth map captured by a depth camera. In this embodiment, a face depth map is mainly used as an example for illustration.

[0068] In this step, the first image to be denoised can be encoded, for example, by using a suitable encoding network such as a convolutional network, thereby obtaining the feature map of the first image after encoding.

[0069] The embodiments of this application can be used to denoise and refine raw, low-quality face depth maps, thereby improving the quality of the original depth data. The denoised and refined face depth maps can then be fed into a recognition network to achieve more accurate and robust face recognition. The overall framework of this scheme is as follows: Figure 2 As shown.

[0070] Figure 2 The framework shown mainly consists of two parts: a denoising network and a recognition network. The recognition network can be any backbone network used for feature extraction, such as the Visual Geometry Group (VGG) model, Residual Network (ResNet), etc. This application embodiment will not go into detail about the recognition network, but mainly introduces... Figure 2 The denoising network part within the dashed box has the following model structure: Figure 3 As shown.

[0071] like Figure 3 As shown, the denoising network consists of an image encoder. and a feature decoder f θ The input image, such as a low-quality depth face image, is encoded by an encoder and represented as a 2D feature map FM∈R. C×H×W The feature map FM can be input into the decoder f along with the coordinate grid of the first image. θ decoder f θ The feature map FM can be decoded by referring to the position coordinate information of the input image to recover the input image and output the first denoised image, such as the denoised face depth data. Decoder f θ The form can be:

[0072] s = f θ (z,x)

[0073] Here, z represents the feature vector at a certain location in the feature map output by the encoder, x∈X represents the discrete 2D coordinates in the continuous image domain, and s∈S represents the signal predicted based on the location coordinates and the feature vector. For example, for a face depth image, the predicted signal is the depth value. When z is determined, each feature vector z can represent a different continuous function f. θ (z,·:X→S) This function can query the corresponding depth value signal based on the input coordinates, thereby realizing the implicit modeling of continuous image signals using neural networks.

[0074] Optionally, step 101 includes:

[0075] The first image is input into an encoding network, wherein the encoding network includes a first convolutional layer and M cascaded downsampling modules. Each downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer.

[0076] Feature extraction is performed on the first image through the first convolutional layer;

[0077] The image features of the first image extracted by the first convolutional layer are processed by the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

[0078] In one implementation, an encoder network consisting of convolutional layers and multiple downsampling modules can be used to extract features from the input image. Each downsampling module consists of a downsampling layer, multiple residual connection blocks (ResBlocks), and a convolutional layer.

[0079] Specifically, the structure of the coding network can be as follows: Figure 4 As shown, the input raw depth image D in ∈R 1 ×128×128 First, the feature map is passed through a convolutional layer with a kernel size of 3×3 and 64 output channels to extract preliminary features. Then, the extracted feature map is continuously fed into several identical downsampling modules, such as two identical downsampling modules, to gradually reduce the spatial resolution of the feature map and extract higher-level semantic features while reducing noise.

[0080] Each downsampling module consists of a downsampling layer, several residual connection blocks, and a 3×3 convolutional layer. The downsampling layer can use a 3×3 convolutional layer with a stride of 2 to achieve downsampling of the spatial resolution of the input feature map by two times.

[0081] Alternatively, the structure of the residual connection block can be as follows: Figure 5 As shown, the network consists of two 3×3 convolutional layers and a Rectified Linear Unit (ReLU) activation function. The residual connection block utilizes residual connections, ensuring that the network extracts higher-level features while minimizing the loss of original information and accelerating convergence. The feature map output from the last residual connection block is then processed by another 3×3 convolutional layer for feature integration.

[0082] All convolutional layers in the encoding network have 64 output channels. Therefore, after two downsampling modules, the final output feature map FM∈R of the encoding network is... 64×32×32 That is, the number of channels is 64 and the spatial resolution is 32×32.

[0083] In this embodiment, the parameter settings for the entire image encoder, i.e., the encoding network, can be as shown in Table 1 below:

[0084] Table 1 Image Encoder Network Parameter Settings

[0085]

[0086] In this way, this implementation method can ensure better feature extraction from the input image, minimize the loss of original information, and accelerate the convergence speed of the network.

[0087] Optionally, the downsampling module further includes a feature connection layer for adding the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

[0088] In one implementation, to achieve better shape recovery, the downsampling module further includes a feature connection layer, such as... Figure 4 As shown, the feature map output by the downsampling layer is added pixel by pixel to the feature map output by the residual connection and the final convolutional layer, that is, the feature values / feature vectors corresponding to the same pixel are added to obtain the final feature map.

[0089] Step 102: Determine the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map.

[0090] In this embodiment of the application, in order to enable the denoising network to have spatial awareness and help the network better understand the spatial structure of the input feature map, the position information of each pixel in the input image can be introduced to establish the correspondence between the position of each pixel in the input image and its position in the feature map. Based on this correspondence, the feature vector corresponding to each pixel can be found, decoded, and the shape of the object in the input image can be restored, such as the shape of a human face.

[0091] Therefore, in this step, based on the position information of each pixel in the first image and the position information of each feature point in the feature map, for each pixel in the first image, its corresponding feature point in the feature map can be found, thereby determining the feature vector corresponding to each pixel based on the correspondence. Specifically, based on the position distribution of each pixel in the first image, the feature points with corresponding positions can be found in the feature map. For example, the pixel in the upper left corner of the image corresponds to the feature point in the upper left corner of the feature map, the pixel in the lower right corner of the image corresponds to the feature point in the lower right corner of the feature map, and the pixel in the middle of the image corresponds to the feature point in the middle of the feature map. Alternatively, based on the position coordinates of each pixel in the first image and the position coordinates of each feature point in the feature map, the distance between each pixel and each feature point can be calculated, thereby determining the feature point corresponding to each pixel based on the nearest distance.

[0092] Optionally, step 102 includes:

[0093] Based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, the distances between the position of the first pixel and the positions of each feature point in the feature map are determined, wherein the first pixel is any pixel in the first image;

[0094] The feature point in the feature map that is closest to the position of the first pixel is determined as the feature point corresponding to the first pixel.

[0095] In one implementation, the feature points corresponding to each pixel in the first image can be determined by calculating the distances, such as Euclidean distances, between the positions of each pixel in the first image and the positions of each feature point in the feature map. This ensures the accuracy of the determined feature points corresponding to each pixel in the image and helps the network accurately understand the spatial structure of the feature map.

[0096] Specifically, the position coordinates of each pixel in the first image and the position coordinates of each feature point in the feature map can be normalized to the same coordinate system, such as taking values ​​between [-1, 1]. Then, for any pixel in the first image, such as the first pixel, the Euclidean distance between the position coordinates of the first pixel in the first image and the position coordinates of each feature point in the feature map can be calculated. The feature point closest to the position coordinates of the first pixel is determined as the feature point corresponding to the first pixel, so that the feature vector of the feature point can be used as the latent code of the first pixel.

[0097] In this way, the correspondence between the position of each point in the image and the feature vector in the feature map can be accurately determined, which helps the decoding network to better understand the spatial structure of the feature map, and thus can better realize face shape restoration.

[0098] Step 103: Based on the relative positions of each pixel in the first image and its corresponding feature points, decode the feature map to obtain the denoised first image.

[0099] After determining the feature points corresponding to each pixel in the first image, the feature map can be decoded based on the relative positions of each pixel and its corresponding feature points in the first image. Specifically, the feature vector of each pixel corresponding to its corresponding feature point can be determined based on the relative positions of each pixel and its corresponding feature points in the first image. Then, the feature vector is combined with the relative positions of each pixel and its corresponding feature points to perform decoding, thereby obtaining the prediction signal corresponding to each pixel, thus recovering the object shape in the first image and obtaining the denoised first image.

[0100] For example, the relative positions of each pixel and its corresponding feature points and feature vectors can be concatenated together as the input vector of the decoder. After processing by the decoder, the output is a face depth map with the same resolution as the original face depth image and denoised.

[0101] Optionally, step 103 includes:

[0102] Determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image;

[0103] The first relative position coordinates are concatenated with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the concatenated feature vector corresponding to the first pixel.

[0104] The concatenated feature vectors corresponding to each pixel in the first image are decoded to obtain the denoised first image.

[0105] In one embodiment, after determining the correspondence between each pixel in the first image and each feature point in the feature map, the relative position coordinates of the corresponding pixel and feature point can be concatenated with the feature vector as input to the decoding network, so that the decoding network can perform more detailed and accurate decoding based on the spatial structure of the feature map, thereby improving the image denoising effect.

[0106] Specifically, the position coordinates of each pixel in the first image and the position coordinates of each feature point in the feature map can be pre-normalized to the same coordinate system. For any pixel in the first image, such as the first pixel, the difference between the position coordinates of the first pixel and the position coordinates of the feature point corresponding to the first pixel can be calculated to obtain their relative position coordinates. Then, their relative position coordinates can be concatenated with the feature vector of the feature point corresponding to the first pixel, specifically concatenating them along the channel dimension.

[0107] By performing the same operation on each pixel in the first image as on the first pixel itself, the relative position coordinates of each pixel in the first image can be concatenated with the feature vector, resulting in a new feature map composed of the concatenated feature vectors. This feature map contains the spatial position information of the feature vectors. This feature map can then be used as the input feature map for a decoding network. The decoding network decodes this feature map, ultimately outputting a first image with the same resolution as the original input first image, but with denoised resolution.

[0108] Thus, in this embodiment, by concatenating the relative position coordinates of corresponding pixels and feature points with the feature vector as input to the decoding network, the decoding network can perform more detailed and accurate decoding based on the spatial structure of the feature map, thereby improving the image denoising effect.

[0109] Optionally, decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes:

[0110] The concatenated feature vectors corresponding to each pixel in the first image are input into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

[0111] In one implementation, a multi-layer perceptron structure decoding network can be used to decode feature vectors with spliced ​​relative position coordinates, accurately recovering the object shape information in the original image.

[0112] Right now Figure 3 decoder f in θIt is a parameterized (θ as network weight parameters) multilayer perceptron shared across all images. The structure diagram of the decoding network, i.e., the feature decoder, can be shown as follows: Figure 6 The diagram shows a multilayer perceptron structure, consisting of several linear layers (also called fully connected layers), such as five, and activation function layers. The first four linear layers have an output dimension of 256, and each linear layer is followed by a ReLU activation function layer. The last linear layer has an output dimension of 1, representing the predicted image signal, such as depth value. The output of the last linear layer is followed by a Tanh activation function layer to normalize the output between [-1, 1].

[0113] The parameter settings for the entire feature decoder network are shown in Table 2 below:

[0114] Table 2 Feature Decoder Network Parameter Settings

[0115]

[0116]

[0117] Taking a face depth map as an example, in the actual implementation, the input low-quality face depth map has a resolution of 128×128 and a channel count of 1. After being encoded... The output feature map FM∈R 64×32×32 The number of channels is 64, and the resolution is 32×32. Query coordinates x. query ∈R 2 Based on the Euclidean distance in the feature map's feature vector space, select the nearest coordinate position v in the feature map FM. * ∈R 2 The eigenvector z at point * ∈R C This serves as the latent encoding corresponding to the query coordinates. After decoding by a multilayer perceptron, the predicted depth value corresponding to the query coordinate location can be expressed as:

[0118] d(x query )=f θ (z * x query -v * )

[0119] At this point, the input to the multilayer perceptron is the difference x between the two coordinates. query -v * and eigenvector z * The input vector z′∈R obtained after concatenation C+2The last layer of the multilayer perceptron has an output dimension of 1, followed by a Tanh activation function that restricts the output depth value to the range [-1, 1]. When the input query coordinates are a 128×128 resolution grid, a denoised output with the same resolution as the original input can be obtained.

[0120] In this embodiment, by parameterizing the multilayer perceptron, the neural network can implicitly model continuous image signals. By inputting coordinates on the continuous image domain and the corresponding latent feature encoding, the depth prediction value at the corresponding location can be queried. Furthermore, due to the introduction of additional image coordinate information, the denoising network acquires spatial awareness, which helps to better achieve face shape restoration.

[0121] Optionally, the first image is a face depth image;

[0122] Decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes:

[0123] The concatenated feature vectors corresponding to each pixel in the face depth image are decoded to obtain the depth value of each pixel in the face depth image, and the depth values ​​of each pixel constitute the denoised face depth image.

[0124] In one embodiment, the image processing method in this application can be applied to denoising facial depth images and can achieve a good facial denoising effect. Using the denoised facial depth image for facial recognition can obtain more accurate and stable facial recognition results.

[0125] When denoising a face depth image using the above implementation method, the decoded output is the depth value of each pixel in the face depth image. Specifically, given the coordinates of a query location in the face depth image, the decoding network can output the predicted depth value for that location.

[0126] This implementation method can improve the denoising effect on low-quality face depth images, and improve the face recognition accuracy when the denoised face depth image is applied to 3D face recognition.

[0127] Taking a face depth map as an example, in the specific implementation of this application embodiment, a coordinate grid with a resolution of 128×128 is first used as the query coordinate x. query ∈R 2×128×128 Based on the Euclidean distance between each query coordinate and the position coordinates of each feature vector in the feature map, the feature vector closest to the query coordinate is selected as the latent encoding for that position. Simultaneously, the difference between the query coordinate and the position coordinates of the feature vector is calculated to obtain the relative coordinate x. rel∈R 2×128×128 The relative coordinates and their corresponding feature vectors are concatenated to obtain the input features of the decoder, feature ∈ R. (C +2)×128×128 After the input feature (feat) is fed into the decoder, the output is a denoised depth map D with the same resolution as the original input. out ∈R 1×128×128 .

[0128] During the model training phase, the denoised depth map D output by the decoder can be used as a reference. out ∈R 1×128×128 High-quality, noise-free depth map D under supervision gt ∈R 1×128×128 The L1 loss for image restoration is calculated and used for gradient backpropagation to update network parameters. The formula for calculating the L1 loss is as follows:

[0129]

[0130] Where p represents a pixel in the image, and N represents the total number of pixels in the image.

[0131] Using the denoising network structure described in this application embodiment, denoising and thinning of original low-quality deep face data can be achieved, improving the accuracy of subsequent recognition networks. Furthermore, since the implicit neural representation technology used in this application embodiment introduces additional coordinate information compared to traditional convolutional neural networks, it helps the network better understand the spatial structure of the input feature map, giving the denoising network spatial awareness and thus better achieving face shape restoration. Simultaneously, because implicit neural representation technology models continuous image signals, the denoising network structure proposed in this application embodiment is not only applicable to denoising tasks but can also be easily extended to super-resolution tasks (simply adjusting the resolution of the network output is sufficient), further improving the quality of deep faces and enhancing the performance of face recognition networks.

[0132] This application utilizes the latest implicit representation technology to design and implement a novel denoising network for low-quality 3D faces; it designs a 3D face image encoder that extracts high-level noise-free semantic features while preserving the shape information of the original face image as much as possible; it designs and implements a 3D face feature decoder based on implicit representation technology, introducing additional coordinate information on the basis of traditional convolutional neural networks, enabling the denoising network to have spatial awareness; this denoising method can be easily extended to super-resolution tasks, further improving the quality of 3D face data.

[0133] Existing technologies address the issue of high noise levels in depth camera-acquired data by proposing encoder-decoder-based deep face denoising and thinning models. However, these models are entirely composed of traditional convolutional neural networks, neglecting the coordinate information of the input image. Image coordinate information, however, helps the network better understand the spatial structure of the input feature map. Unlike existing technologies, this application employs implicit neural representation technology, using a multilayer perceptron as the decoder for the denoising network. By parameterizing this multilayer perceptron, the neural network implicitly models continuous image signals. By inputting coordinates in the continuous image domain and the corresponding latent feature encoding, the depth prediction value at the corresponding location can be queried. Compared to traditional convolutional neural networks, this application introduces additional image coordinate information, giving the denoising network spatial awareness, which helps to better restore face shape. Furthermore, since implicit neural representation models continuous image signals, the network structure proposed in this application is not only applicable to denoising tasks but can also be easily extended to super-resolution tasks, further improving the quality of deep faces and enhancing the performance of subsequent face recognition.

[0134] The image processing method of this application embodiment encodes a first image to obtain a feature map of the first image; determines the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map; and decodes the feature map based on the relative positions of each pixel in the first image and its corresponding feature points to obtain a denoised first image. In this way, by introducing the position information of the input image, it helps to better understand the spatial structure of the input feature map, enabling the denoising network to have spatial awareness capabilities, thereby achieving better image denoising results.

[0135] This application also provides a terminal. See [link to relevant documentation]. Figure 7 , Figure 7 This is a structural diagram of the image processing apparatus provided in the embodiments of this application. Since the principle of the image processing apparatus in solving the problem is similar to that of the image processing method in the embodiments of this application, the implementation of this image processing apparatus can refer to the implementation of the method, and repeated details will not be described again.

[0136] like Figure 7 As shown, the image processing apparatus 700 includes:

[0137] The encoding module 701 is used to encode the first image to obtain a feature map of the first image;

[0138] The first determining module 702 is used to determine the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map.

[0139] The decoding module 703 is used to decode the feature map according to the relative position of each pixel in the first image and the corresponding feature point to obtain the denoised first image.

[0140] Optionally, the first determining module 702 includes:

[0141] The first determining unit is configured to determine the distances between the position of the first pixel and the positions of each feature point in the feature map based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, wherein the first pixel is any pixel in the first image.

[0142] The second determining unit is used to determine the feature point in the feature map whose position is closest to the first pixel as the feature point corresponding to the first pixel.

[0143] Optionally, the decoding module 703 includes:

[0144] The third determining unit is used to determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image;

[0145] The splicing unit is used to splice the first relative position coordinates with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the spliced ​​feature vector corresponding to the first pixel.

[0146] The decoding unit is used to decode the spliced ​​feature vectors corresponding to each pixel in the first image to obtain the denoised first image.

[0147] Optionally, the decoding unit is used to input the concatenated feature vectors corresponding to each pixel in the first image into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

[0148] Optionally, the first image is a face depth image;

[0149] The decoding unit is used to decode the spliced ​​feature vector corresponding to each pixel in the face depth image to obtain the depth value of each pixel in the face depth image, and the depth value of each pixel constitutes the denoised face depth image.

[0150] Optionally, the encoding module is an encoding network, which includes a first convolutional layer and M cascaded downsampling modules. The downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer.

[0151] The encoded network is used to extract features from the first image through the first convolutional layer, and to process the image features of the first image extracted by the first convolutional layer through the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

[0152] Optionally, the downsampling module further includes a feature connection layer for adding the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

[0153] The image processing apparatus 700 provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0154] The image processing apparatus 700 of this application embodiment encodes a first image to obtain a feature map of the first image; determines the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map; and decodes the feature map based on the relative positions of each pixel in the first image and its corresponding feature points to obtain a denoised first image. In this way, by introducing the position information of the input image, it helps to better understand the spatial structure of the input feature map, enabling the denoising network to have spatial awareness, thereby achieving better image denoising results.

[0155] This application also provides an electronic device. Since the principle by which the electronic device solves the problem is similar to the image processing method in this application, the implementation of this electronic device can be found in the implementation of the method, and repeated details will not be described again. Figure 8 As shown, the electronic device according to an embodiment of this application includes:

[0156] Processor 800 is used to read the program from memory 820 and execute the following procedures:

[0157] The first image is encoded to obtain a feature map of the first image;

[0158] Based on the position information of each pixel in the first image and the position information of each feature point in the feature map, the feature points corresponding to each pixel in the first image are determined.

[0159] Based on the relative positions of each pixel in the first image and its corresponding feature points, the feature map is decoded to obtain the denoised first image.

[0160] Among them, Figure 8 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 800) and memory (memory 820). The bus architecture can also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides the interface. Processor 800 is responsible for managing the bus architecture and general processing, and memory 820 can store data used by processor 800 during operation.

[0161] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0162] Based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, the distances between the position of the first pixel and the positions of each feature point in the feature map are determined, wherein the first pixel is any pixel in the first image;

[0163] The feature point in the feature map that is closest to the position of the first pixel is determined as the feature point corresponding to the first pixel.

[0164] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0165] Determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image;

[0166] The first relative position coordinates are concatenated with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the concatenated feature vector corresponding to the first pixel.

[0167] The concatenated feature vectors corresponding to each pixel in the first image are decoded to obtain the denoised first image.

[0168] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0169] The concatenated feature vectors corresponding to each pixel in the first image are input into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

[0170] Optionally, the first image is a face depth image;

[0171] The processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0172] The concatenated feature vectors corresponding to each pixel in the face depth image are decoded to obtain the depth value of each pixel in the face depth image, and the depth values ​​of each pixel constitute the denoised face depth image.

[0173] Optionally, the processor 800 is also used to read the program from the memory 820 and perform the following steps:

[0174] The first image is input into an encoding network, wherein the encoding network includes a first convolutional layer and M cascaded downsampling modules. Each downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer.

[0175] Feature extraction is performed on the first image through the first convolutional layer;

[0176] The image features of the first image extracted by the first convolutional layer are processed by the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

[0177] Optionally, the downsampling module further includes a feature connection layer for adding the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

[0178] The electronic device provided in this application embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0179] Furthermore, the computer-readable storage medium of this application embodiment is used to store a computer program, which can be executed by a processor. Figure 1 Each step in the method embodiment shown.

[0180] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically included separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0182] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized by, include: The first image is encoded to obtain a feature map of the first image; Based on the position information of each pixel in the first image and the position information of each feature point in the feature map, the feature points corresponding to each pixel in the first image are determined. Based on the relative positions of each pixel in the first image and its corresponding feature points, the feature map is decoded to obtain the denoised first image. The step of decoding the feature map based on the relative positions of each pixel in the first image and its corresponding feature points to obtain the denoised first image includes: Determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image; The first relative position coordinates are concatenated with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the concatenated feature vector corresponding to the first pixel. The concatenated feature vectors corresponding to each pixel in the first image are decoded to obtain the denoised first image.

2. The method of claim 1, wherein, The step of determining the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map includes: Based on the position information of the first pixel in the first image and the position information of each feature point in the feature map, the distances between the position of the first pixel and the positions of each feature point in the feature map are determined, wherein the first pixel is any pixel in the first image; The feature point in the feature map that is closest to the position of the first pixel is determined as the feature point corresponding to the first pixel.

3. The method of claim 1, wherein, Decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes: The concatenated feature vectors corresponding to each pixel in the first image are input into the decoding network to obtain the denoised first image output by the decoding network, wherein the decoding network includes a multilayer perceptron.

4. The method of claim 1, wherein, The first image is a face depth image; Decoding the concatenated feature vectors corresponding to each pixel in the first image to obtain the denoised first image includes: The concatenated feature vectors corresponding to each pixel in the face depth image are decoded to obtain the depth value of each pixel in the face depth image, and the depth values ​​of each pixel constitute the denoised face depth image.

5. The method according to any one of claims 1 to 4, characterized in that, The encoding process of the first image to obtain the feature map of the first image includes: The first image is input into an encoding network, wherein the encoding network includes a first convolutional layer and M cascaded downsampling modules. Each downsampling module includes a downsampling layer, N cascaded residual connection blocks, and a second convolutional layer, where N is an integer greater than 1 and M is a positive integer. Feature extraction is performed on the first image through the first convolutional layer; The image features of the first image extracted by the first convolutional layer are processed by the M cascaded downsampling modules to obtain the feature map. The downsampling layer is used to downsample the input image features, the N cascaded residual connection blocks are used to perform residual connection on the downsampled image features, and the second convolutional layer is used to integrate the image features after residual connection.

6. The method of claim 5, wherein, The downsampling module further includes a feature connection layer, which is used to add the image features output by the second convolutional layer and the image features output by the downsampling layer pixel by pixel.

7. An image processing apparatus characterized by comprising: include: The encoding module is used to encode the first image to obtain a feature map of the first image; The first determining module is used to determine the feature points corresponding to each pixel in the first image based on the position information of each pixel in the first image and the position information of each feature point in the feature map. The decoding module is used to decode the feature map according to the relative position of each pixel in the first image and the corresponding feature point to obtain the denoised first image; The decoding module includes: The third determining unit is used to determine the first relative position coordinates between the first pixel and the feature point corresponding to the first pixel, wherein the first pixel is any pixel in the first image; The splicing unit is used to splice the first relative position coordinates with the feature vector of the feature point corresponding to the first pixel in the channel dimension to obtain the spliced ​​feature vector corresponding to the first pixel. The decoding unit is used to decode the spliced ​​feature vectors corresponding to each pixel in the first image to obtain the denoised first image.

8. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that the processor is configured to read the program from the memory to implement the steps of the image processing method as described in any one of claims 1 to 6.

9. A computer readable storage medium for storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the image processing method as described in any one of claims 1 to 6.

Images