Cross-camera face fusion super-resolution method and device based on channel correlation
Through the super-resolution method of cross-camera face fusion based on channel correlation, high-quality face images are reconstructed using hidden features and spatial features, solving the problem of inconsistent and serious degradation process in security criminal investigation scenarios, and achieving high-quality people Reconstruction and recognition of face images.
Patent Information
- Application Number
- CN202410880428.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-07-02
AI Technical Summary
The prior art is difficult to deal with cross-camera face images with inconsistent degradation process and severe degradation, resulting in increased difficulty in facial recognition in security criminal investigation scenarios, and it is impossible to effectively improve the clarity and recognition of super-resolution images.
Using a cross-camera face fusion super-resolution method based on channel correlation, the cross-camera face super-resolution network model is constructed, and the model is trained using paired low-resolution-high-resolution face image dataset, combined with hidden features, Spatial features and fusion factors generate fusion enhancement features to reconstruct high-quality super-resolution face images.
The problem of inconsistent cross-camera face quality reduction process and serious degree of quality reduction was successfully solved, and high-quality and more reliable face images were reconstructed, which was suitable for face recognition in security criminal investigation scenarios.
Smart Images

Figure CN119067846A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of digital image technology, and in particular to a cross-camera face fusion super-resolution method and device based on channel correlation. Background Art
[0002] In security and criminal investigation scenarios, facial images captured by surveillance cameras have become one of the increasingly important clues in daily public security and criminal investigation scenarios. However, in actual surveillance scenarios, due to the limitations of camera costs, image storage costs and other inherent factors, coupled with the combined influence of environmental factors such as lighting conditions, shooting angles, and shooting distances, the facial images actually collected and stored are often low-quality images with real complex degradation, which undoubtedly increases the difficulty of using surveillance facial images to identify suspects in actual criminal investigation scenarios. Due to the high coverage rate of surveillance cameras, in actual criminal investigation scenarios, there are often multiple different cameras that capture multiple facial images of suspects at different times, which provides a possible way to solve the problem of missing image information under strong degradation conditions, and thus improve the image clarity and identity recognition of super-resolution faces. However, in practice, faces captured by different cameras have problems such as inconsistent degradation processes and severe degradation, making it difficult to directly perform face fusion super-resolution.
[0003] With the development of deep learning in the field of computer vision, research on multi-frame image fusion super-resolution algorithms has made significant progress and achieved remarkable results. In related technologies, the StyleGAN network with powerful face generation capabilities can be introduced into the face blind restoration task through a simple spatial feature change layer based on the face generation prior, thereby achieving the function of high-quality face blind restoration images; the performance of super-resolution can be significantly improved by combining additional information from multiple reference images and with the help of an attention mechanism; and super-resolution of VR videos can be achieved through a loop structure with an attention mechanism based on its specially constructed data set.
[0004] However, in the related technologies, the blind face restoration algorithm is limited to blind restoration of a single frame face image, and cannot be directly used to solve the cross-camera face super-resolution problem. The image super-resolution algorithm based on the reference frame uses high-quality images to improve the image quality of low-resolution input, but in the security and criminal investigation scenario, high-quality photos of relevant personnel cannot be used as reference images, which does not meet the actual needs; and the related technologies mainly focus on super-resolution of low-quality images with simple and known degradation processes, which is not suitable for cross-camera face super-resolution conditions with inconsistent degradation processes and severe degradation, and needs to be improved. Summary of the invention
[0005] The present application provides a cross-camera face fusion super-resolution method and device based on channel correlation to solve the technical problems in the related art that the method is limited to the blind restoration of a single-frame face image and focuses on super-resolution of low-quality images with a simple and known degradation process, is not suitable for cross-camera face super-resolution conditions with inconsistent degradation processes and severe degradation, and is difficult to apply to security and criminal investigation scenarios.
[0006] The first aspect of the present application provides a cross-camera face fusion super-resolution method based on channel correlation, which is applied to the model building stage, wherein the method includes the following steps: obtaining an original high-definition face dataset, and using the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; constructing a cross-camera face super-resolution network model, and using the cross-camera face super-resolution network model to fuse and enhance low-resolution face images with a resolution less than a first preset resolution to high-resolution face images greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution. A preset resolution; using the paired cross-camera low-resolution-high-resolution face image dataset to train the cross-camera face super-resolution network model to obtain a trained cross-camera face super-resolution network model; using the paired cross-camera low-resolution face image dataset to test the trained cross-camera face super-resolution network model to obtain a test face image, and comparing the test face image with the corresponding face image in the original high-definition face dataset to obtain a test result, and when the test result meets the preset convergence condition, using the trained cross-camera face super-resolution network model as the final cross-camera face super-resolution network model, so as to use the final cross-camera face super-resolution network model to perform cross-camera face fusion.
[0007] Optionally, in one embodiment of the present application, the cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model, including: extracting corresponding face hidden features and face spatial features from the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating fusion factors based on the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; combining the face spatial features and the fusion factors to generate face fusion enhanced spatial features of the corresponding layer; converting the face hidden features into hidden codes; reconstructing a fused super-resolution face image using the hidden codes and the fused enhanced face spatial features; calculating the total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset to obtain a calculation result, and using the calculation result to update the parameters of the cross-camera face super-resolution network model.
[0008] Optionally, in one embodiment of the present application, the calculation expression of the total loss function is:
[0009]
[0011] Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id , L adv denote reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively, λ1, λ2, λ3, λ4 and λ5 denote the corresponding weights of the reconstruction loss, the pixel loss, the perceptual loss, the identity loss and the adversarial loss respectively, L cid They represent the fusion loss and the cross-identity loss respectively, and λ6 and λ7 represent the corresponding weights of the fusion loss and the cross-identity loss respectively.
[0012] Optionally, in one embodiment of the present application, the combining of the face spatial features and the fusion factors to generate the face fusion enhanced spatial features of the corresponding layer includes: generating corresponding identity features and fusion factors based on the low-resolution face image; preprocessing the face spatial features and the fusion factors using a preset convolutional network block to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; according to the matching weights, matching and fusing the preprocessed features using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block to obtain the face fusion enhanced spatial features.
[0013] The second aspect of the present application provides a cross-camera face fusion super-resolution method based on channel correlation, which is applied to the model usage stage, wherein the method includes the following steps: obtaining multiple face images of the same target and modality from different cameras; inputting the multiple face images into a pre-built cross-camera face super-resolution network model to obtain a reconstructed fusion enhanced face image with a resolution higher than a preset resolution threshold, wherein the cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.
[0014] The third aspect of the present application provides a cross-camera face fusion super-resolution device based on channel correlation, which is applied to the model construction stage, wherein the device includes: an acquisition module, used to acquire an original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; a construction module, used to construct a cross-camera face super-resolution network model, so as to use the cross-camera face super-resolution network model to fuse and enhance a low-resolution face image with a resolution less than a first preset resolution to a high-resolution face image greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution. Set resolution; a training module, used to train the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model; a testing module, used to test the trained cross-camera face super-resolution network model using the paired cross-camera low-resolution face image dataset to obtain a test face image, and compare the test face image with the corresponding face image in the original high-definition face dataset to obtain a test result, and when the test result meets a preset convergence condition, use the trained cross-camera face super-resolution network model as the final cross-camera face super-resolution network model to perform cross-camera face fusion using the final cross-camera face super-resolution network model.
[0015] Optionally, in one embodiment of the present application, the training module includes: an extraction unit, used to extract corresponding facial latent features and facial spatial features from the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a first generation unit, used to generate a fusion factor based on the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a second generation unit, used to generate facial fusion enhanced spatial features of the corresponding layer in combination with the facial spatial features and the fusion factor; a conversion unit, used to convert the facial latent features into latent codes; a reconstruction unit, used to reconstruct a fused super-resolution facial image using the latent codes and the fused enhanced facial spatial features; an updating unit, used to calculate the total loss function based on the fused super-resolution facial image and the corresponding high-resolution facial image in the paired cross-camera low-resolution-high-resolution facial image dataset, obtain a calculation result, and use the calculation result to update the parameters of the cross-camera face super-resolution network model.
[0016] Optionally, in one embodiment of the present application, the calculation expression of the total loss function is:
[0017]
[0018] Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id , L adv denote reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively, λ1, λ2, λ3, λ4 and λ5 denote the corresponding weights of the reconstruction loss, the pixel loss, the perceptual loss, the identity loss and the adversarial loss respectively, L cid They represent the fusion loss and the cross-identity loss respectively, and λ6 and λ7 represent the corresponding weights of the fusion loss and the cross-identity loss respectively.
[0019] Optionally, in one embodiment of the present application, the second generation unit includes: a generation subunit, used to generate corresponding identity features and fusion factors based on the low-resolution face image; a first processing subunit, used to preprocess the face spatial features and the fusion factors using a preset convolutional network block to obtain preprocessed features; a second processing subunit, used to normalize the preprocessed features to obtain normalized features, and use the normalized features to obtain corresponding matching weights; a fusion subunit, used to match and fuse the preprocessed features according to the matching weights using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block to obtain the face fusion enhanced spatial features.
[0020] The fourth aspect of the present application provides a cross-camera face fusion super-resolution device based on channel correlation, which is applied to the model usage stage, wherein the device includes: an acquisition module, used to acquire multiple face images of the same target and modality from different cameras; a reconstruction module, used to input the multiple face images into a pre-built cross-camera face super-resolution network model to obtain a reconstructed fusion-enhanced face image with a resolution higher than a preset resolution threshold, wherein the cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.
[0021] The fifth aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the cross-camera face fusion super-resolution method based on channel correlation as described in the above embodiment.
[0022] The sixth aspect embodiment of the present application provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the cross-camera face fusion super-resolution method based on channel correlation as described in the above embodiment.
[0023] The seventh aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned cross-camera face fusion super-resolution method based on channel correlation.
[0024] The embodiment of the present application can obtain an original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing, and use the paired cross-camera low-resolution-high-resolution face image dataset to train the constructed cross-camera face super-resolution network model to obtain a trained cross-camera face super-resolution network model, and use the paired cross-camera low-resolution face image dataset to test the trained cross-camera face super-resolution network model to obtain a test face image, and compare the test face image with the corresponding face image in the original high-definition face dataset, so that when the test result meets the preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, and the final cross-camera face super-resolution network model is used to perform cross-camera face fusion, so as to utilize the complementarity between cross-camera low-resolution faces to reconstruct a high-quality and more reliable face image. Thereby, the technical problems in the related technology, which are limited to the blind restoration of a single-frame facial image and focus on super-resolution of low-quality images with a simple and known degradation process, are solved. The technology is not suitable for cross-camera facial super-resolution conditions with inconsistent degradation processes and severe degradation, and is difficult to apply to security and criminal investigation scenarios.
[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:
[0027] Figure 1 A flowchart of a cross-camera face fusion super-resolution method based on channel correlation provided according to an embodiment of the present application;
[0028] Figure 2 A network diagram of a fusion factor generation module according to an embodiment of the present application;
[0029] Figure 3 A network diagram of a feature fusion super-resolution module according to an embodiment of the present application;
[0030] Figure 4 A network diagram of a cross-camera face super-resolution network model according to an embodiment of the present application;
[0031] Figure 5 A schematic diagram of the structure of a cross-camera face fusion super-resolution device based on channel correlation provided according to an embodiment of the present application;
[0032] Figure 6 A flowchart of another cross-camera face fusion super-resolution method based on channel correlation provided according to an embodiment of the present application;
[0033] Figure 7 A schematic diagram of the structure of another cross-camera face fusion super-resolution device based on channel correlation provided according to an embodiment of the present application;
[0034] Figure 8 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0035] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.
[0036] The following describes the cross-camera face fusion super-resolution method and device based on channel correlation of an embodiment of the present application with reference to the accompanying drawings. In view of the technical problem that the related technologies mentioned in the above background technology are limited to the blind restoration of single-frame face images, and focus on super-resolution of low-quality images with simple and known degradation processes, and are not suitable for cross-camera face super-resolution conditions with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios, the present application provides a cross-camera face fusion super-resolution method based on channel correlation, in which the original high-definition face dataset can be obtained, and the original high-definition face dataset can be used to construct a paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing, and the paired cross-camera low-resolution-high-resolution face dataset can be used to construct a paired cross-camera low-resolution-high-resolution face dataset for training. The cross-camera face super-resolution network model constructed by training the image data set is obtained, and the trained cross-camera face super-resolution network model is tested by using the paired cross-camera low-resolution face image data set to obtain the test face image, and the test face image is compared with the corresponding face image in the original high-definition face data set, so that when the test result meets the preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, and the final cross-camera face super-resolution network model is used for cross-camera face fusion, so as to utilize the complementarity between cross-camera low-resolution faces to reconstruct a high-quality and more reliable face image. Thus, the technical problem that the related technology is limited to blind restoration of single-frame face images, and focuses on super-resolution of low-quality images with simple and known degradation processes, is not applicable to cross-camera face super-resolution working conditions with inconsistent degradation processes and severe degradation, and is difficult to be applied to security and criminal investigation scenes is solved.
[0037] Specifically, Figure 1 A flow chart of a cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of the present application.
[0038] like Figure 1 As shown, the cross-camera face fusion super-resolution method based on channel correlation is applied to the model building stage, wherein the method includes the following steps:
[0039] In step S101, an original high-definition face dataset is obtained, and the original high-definition face dataset is used to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing.
[0040] During the actual execution process, the embodiments of the present application can pre-process the facial images in the original high-definition facial dataset, construct a paired cross-camera low-resolution-high-resolution facial image dataset required for training the facial super-resolution network model, and construct a paired cross-camera low-resolution facial image dataset required for testing.
[0041] For example, in the embodiment of the present application, the RetinaFace model can be used to perform binocular calibration-based basic alignment on the original high-definition face photo pair dataset, and then the bicubic interpolation method can be used to convert its resolution to 512×512, and then the simulated degradation model can be used to generate low-quality-high-quality face image pairs; the dataset is divided into a training set and a test set; according to the identity information, a paired cross-camera low-resolution-high-resolution face image dataset required for the training network and a paired low-resolution face image dataset required for the test network are formed. In the embodiment of the present application, the record They are respectively the low-resolution face X, Y and the corresponding high-resolution face X, Y after the dataset is processed.
[0042] In step S102, a cross-camera face super-resolution network model is constructed to utilize the cross-camera face super-resolution network model to fuse and enhance a low-resolution face image having a resolution less than a first preset resolution into a high-resolution face image having a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution.
[0043] The cross-camera face super-resolution network model can be used to fuse and enhance cross-camera low-resolution faces into high-resolution faces.
[0044] For example, Figure 2 and Figure 3 As shown, the cross-camera face super-resolution network model can structurally include: a feature extraction module, a feature fusion super-resolution module and a feature reconstruction module.
[0045] The embodiment of the present application can use the feature extraction module to obtain the basic features of the two face images, input the obtained basic features into the feature fusion super-resolution module to obtain the corresponding fusion features, and input the fusion features into the feature reconstruction module to obtain two final output faces.
[0046] Among them, the feature extraction module of the cross-camera face super-resolution network model can adopt a 7-layer Unet structure, which can be subdivided into two parts: feature encoder and feature decoder, consisting of 7 downsampling modules and 7 upsampling modules; the cross-camera low-resolution face image is passed through the feature encoder to obtain the face hidden features; the face hidden features are passed through the feature decoder to obtain 7 layers of face spatial features and corresponding image-level outputs.
[0047] The feature fusion super-resolution module of the cross-camera face super-resolution network can be divided into a fusion factor generation stage, a pre-convolution stage, a matching weight generation stage, and a matching convolution fusion stage during use.
[0048] The fusion factor generation stage is divided into an identity feature generation sub-stage and a fusion factor conversion sub-stage. The identity feature generation sub-stage can be performed by the face recognition network MagFace, and the fusion factor conversion sub-stage can be performed by the fusion factor conversion sub-module Transfer. The fusion factor conversion sub-module consists of a fully connected layer, normalization and activation functions ReLu and Sigmoid. The fusion factor generation sub-stage generates corresponding identity features and fusion factors according to the input face image.
[0049] In the pre-convolution stage, a 3×3 convolutional network block can be used to preprocess the face spatial features extracted by the feature extraction module and the fusion factors obtained in the fusion factor generation stage.
[0050] In the matching weight generation stage, GroupNormalization can be used to normalize the preprocessed features, and then the normalized features can be transformed and matrix multiplied to obtain the corresponding matching weights.
[0051] In the matching convolution fusion stage, according to the obtained matching weights, a 3×3 matching convolution network block based on channel correlation adjustment and a 3×3 convolution network block are used to match and fuse the preprocessed features to obtain the final fused enhanced features.
[0052] Combination Figure 2 and Figure 3 As shown, the construction process may include the following steps:
[0053] Step S1: The cross-camera face super-resolution network model uses a low-resolution face image is the input;
[0054] Step S2: Input the two face images obtained in step S1 into a feature extraction module to obtain corresponding face hidden features and face spatial features;
[0055] In the embodiments of the present application:
[0056] 1) Feature encoder based on face image and As input, the corresponding hidden features of the face are obtained. The process is as follows:
[0057]
[0058] Among them, Encoder() represents the feature encoder, X feature , Yfeature is the corresponding hidden feature of the face.
[0059] 2) The feature decoder takes the hidden features of the face as input and obtains the spatial features of the face at each level and the corresponding image-level output. The process is shown as follows:
[0060]
[0061]
[0062] Among them, Decoder i () represents the i-th layer of the feature decoder, is the corresponding face spatial feature of the i-th layer, is the image-level output corresponding to the i-th layer.
[0063] Step S3: inputting the face and face spatial features obtained in step S1 and step S2 into a feature fusion super-resolution module to generate face fusion enhanced spatial features;
[0064] In an embodiment of the present application, the feature fusion super-resolution module is divided into a fusion factor generation stage, a pre-convolution stage, a matching weight generation stage, and a matching convolution fusion stage, wherein the fusion factor generation stage is divided into an identity feature generation sub-stage and a fusion factor conversion sub-stage.
[0065] 1) Face recognition network MagFace uses face images As input, the corresponding identity features are obtained. The process is shown as follows:
[0066]
[0067] Among them, MagFace() represents the face recognition network, X id , Y id Indicates the corresponding identity feature.
[0068] 2) Fusion factor conversion submodule Transfer with identity feature X id , Y id The corresponding fusion factor is generated as input. The fusion factor conversion submodule has 5 layers. The first 4 layers are composed of Linear, LayerNorm, and Relu layers in series, and the last layer is composed of Linear and Sigmoid layers in series. The process is shown as follows:
[0069]
[0070]
[0071]
[0072] Among them, Transfer() represents the fusion factor conversion submodule, represents the generated fusion factor group, Represents the fusion factor acting on the i-th layer of face feature fusion super-resolution network.
[0073] 3) Pre-convolution module based on face spatial features and the corresponding fusion factor The pre-convolution module is composed of a 3×3Conv layer and a LeakyRelu layer. The process is as follows:
[0074]
[0075]
[0076] Among them, LRelu() represents the LeakyRelu layer, con3() represents the 3×3Conv layer, Represents the preprocessing features of the corresponding layer.
[0077] 4) Matching weight generation module to preprocess features The corresponding correlation matching weight is obtained for the input. The matching weight generation module consists of a GroupNorm layer, a Sigmoid layer, and two 1×1Conv layers, and also includes feature concatenation, matrix deformation, and matrix multiplication operations. The process is expressed as follows:
[0078]
[0079]
[0080]
[0081] Among them, con1() represents the 1×1 Conv layer, GNorm() represents the GroupNormalization layer, Concat() represents the feature concatenation operation, Sigmoid() represents the Sigmoid layer, and Resize() represents the matrix deformation operation. Represents the matrix multiplication operation, W i Gives the generated relevance match weights.
[0082] 5) Matching convolution fusion module with face spatial features and preprocessing features and relevance matching weight W i The fusion enhanced spatial features are obtained for the input. The matching convolution fusion module consists of a 3×3 matching convolution layer based on channel correlation adjustment, a LeakyRelu layer and a 3×3Conv layer, and also includes feature splitting operations and residual addition operations. The process is expressed as follows:
[0083]
[0084] Among them, C out Indicates the number of channels of output features, I j and K (i,j) They represent the jth channel of the input feature and the jth channel of the i-th convolution kernel, respectively. Conv() represents a single-channel convolution operation. (i,j) represents the jth channel weight value of the ith weight, O i represents the i-th channel of the output feature, and this formula is the 3×3 matching convolution layer based on channel correlation adjustment.
[0085]
[0086]
[0087]
[0088] Among them, con3() represents a 3×3 Conv layer, LRelu() represents a LeakyRelu layer, Concat() represents a feature concatenation operation, com3() represents a 3×3 matching convolution layer based on channel correlation adjustment, and Split() represents a feature splitting operation. Enhance spatial features for fusion.
[0089] Step S4: inputting the face hidden features and face fusion enhanced spatial features obtained in step S2 and step S3 into the face reconstruction module to generate a high-quality fused super-resolution face;
[0090] In the embodiments of the present application:
[0091] 1) Feature encoding layer MLP uses face hidden feature X feature , Y feature As input, we can get the face hidden code that is more consistent with the initial feature distribution of the face reconstruction network StyleGAN. The feature encoding layer MLP is a 1-layer fully connected network layer. The process is as follows:
[0092] X code =MLP(X feature ),Ycode =MLP(Y feature )
[0093] Among them, MLP() represents the feature encoding layer, X code , Y code Hidden code for the generated face.
[0094] 2) Face reconstruction module uses face hidden code X code , Y code Fusion with face to enhance spatial features As input, we can get high-quality fused super-resolution faces. The face reconstruction module is a pre-trained StyleGAN network, which uses a progressive generation method to reconstruct high-quality face images based on facial features. The process is shown as follows:
[0095]
[0096]
[0097] Among them, StyleGAN i () represents the i-th layer generation subnetwork of StyleGAN, and is the corresponding image-level output of the i-th layer, and is the final output of the network, i.e., a high-quality fused super-resolution face image.
[0098] It should be noted that the first preset resolution and the second preset resolution can be set accordingly by those skilled in the art based on the classification standard of low resolution and high resolution, and no specific limitation is made here.
[0099] In step S103, a cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model.
[0100] As a possible way to achieve this, after the cross-camera face super-resolution network model is built, it is possible to use paired cross-camera low-resolution-high-resolution face image datasets for paired learning to super-resolve the cross-camera faces into high-resolution faces, and obtain fused and enhanced high-resolution faces.
[0101] Optionally, in one embodiment of the present application, a cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model, including: extracting corresponding face hidden features and face spatial features from the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating fusion factors based on the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; combining the face spatial features and the fusion factors to generate face fusion enhanced spatial features of the corresponding layer; converting the face hidden features into latent codes; reconstructing a fused super-resolution face image using the latent codes and the fusion enhanced face spatial features; calculating the total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset to obtain a calculation result, and using the calculation result to update the parameters of the cross-camera face super-resolution network model.
[0102] Among them, combining the face spatial features and the fusion factors to generate the face fusion enhanced spatial features of the corresponding layer includes: generating the corresponding identity features and the fusion factors based on the low-resolution face image; using the preset convolutional network block to preprocess the face spatial features and the fusion factors to obtain the preprocessed features; normalizing the preprocessed features to obtain the normalized features, and using the normalized features to obtain the corresponding matching weights; according to the matching weights, using the preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block to match and fuse the preprocessed features to obtain the face fusion enhanced spatial features.
[0103] like Figure 4 As shown, the cross-camera face super-resolution network model is a dual-branch generative adversarial network, whose structure may include: a face feature extraction module, a face feature fusion super-resolution module, a face feature reconstruction module and a high-resolution face discrimination module. The embodiment of the present application may utilize the dual-branch structure for joint training.
[0104] Among them, the face feature extraction module can adopt a seven-layer Unet structure.
[0105] The facial feature fusion super-resolution module can be obtained by step S102 and embedded in the cross-camera face super-resolution network.
[0106] The facial feature reconstruction module can be a pre-trained face reconstruction network StyleGAN.
[0107] The high-resolution face recognition module can adopt a discriminator structure consistent with StyleGAN.
[0108] The training process can include the following steps:
[0109] Step S1: Input the low-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset into the feature extraction network of the face feature extraction module to extract the face hidden features and face spatial features, which can be expressed as the following formula:
[0110]
[0111]
[0112]
[0113] Among them, Encoder() represents the feature encoder, X feature , Y feature is the corresponding hidden feature of the face, Decoder i () represents the i-th layer of the feature decoder, is the corresponding face spatial feature of the i-th layer, is the image-level output corresponding to the i-th layer.
[0114] Step S2: Input the low-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset into the fusion factor generation module of the face feature fusion super-resolution module in the cross-camera face super-resolution network model to generate a fusion factor, which can be expressed as the following formula:
[0115]
[0116]
[0117] Among them, FGen() represents the fusion factor generation module, Represents the fusion factor acting on the i-th layer of face feature fusion super-resolution network.
[0118] Step S3: Input the face spatial features and the matching fusion factors into the feature fusion super-resolution network of the corresponding layer to generate the fusion enhanced spatial features of the corresponding layer, which can be expressed as the following formula:
[0119]
[0120] Among them, Fusion i () is the i-th layer feature fusion super-resolution module, Enhance the spatial features of faces for the generated fusion.
[0121] Step S4: Input the hidden features of the face into the feature coding network and convert them into hidden codes, which can be expressed as the following formula:
[0122] X code =MLP(X feature ),Y code =MLP(Y feature )
[0123] Among them, MLP() represents the feature encoding layer, X code , Y code Hidden code for the generated face.
[0124] Step S5: Input the latent coding and fusion enhanced face spatial features into the face reconstruction network to reconstruct the fused super-resolution face image, which can be expressed as the following formula:
[0125]
[0126]
[0127] Among them, StyleGAN i () represents the i-th layer generation subnetwork of StyleGAN, and is the corresponding image-level output of the i-th layer, and is the final output of the network, i.e., a high-quality fused super-resolution face image.
[0128] Step S6: Calculate the total loss function and update the network parameters according to the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset.
[0129] Among them, the loss function can include: reconstruction loss, pixel loss, perceptual loss, identity loss, cross-identity loss, and update network parameters, which are specifically expressed as follows:
[0130]
[0131]
[0132] Among them, ∥AB∥1 represents L1 loss calculation, is the image-level output of the U-Net network layer i of the face feature extraction module in step S1, and They represent downsampled images iteratively obtained from real high-resolution faces, and DownSample() represents a bilinear 2x downsampling operation.
[0133] L pixel =||X output -X gt||1+||Y output -Y gt ||1
[0134] L per =||X percep -X g_percep ||1+||X style -X g_style ||1
[0135] +||Y percep -Y g_percep ||1+||Y style -Y g_style ||1
[0136] X percep ,X style =VGG(X output ),X g_percep ,X g_style =VGG(X gt )
[0137] Among them, X percep , X g_percep , Y percep , Y g_percep They represent the perceptual features generated by the output image of the X branch, the real high-resolution image of the X branch, the output image of the Y branch, and the high-resolution image of the Y branch, respectively. style , X g_style , Y style , Y g_style They represent the style features generated by the X-branch output image, the X-branch true high-resolution image, the Y-branch output image, and the Y-branch high-resolution image, respectively. VGG() represents the pre-trained VGG-19 network.
[0138] L id =||X id -X g_id ||1+||Y id -Y g_id ||1
[0139] X id =ArcFace(X input ),X g_id =ArcFace(X gt )
[0140] L cid =||X id -Y id ||1*(1-Sigmoid(||X g_id -Y g_id ||1))
[0141] Among them, X id , X g_id , Y id , Y g_id They represent the face recognition features generated by the output image of the X branch, the real high-resolution image of the X branch, the output image of the Y branch, and the high-resolution image of the Y branch, respectively. ArcFace() represents the pre-trained ArcFace network. Sigmoid() represents the Sigmoid activation function, which can map variables from the (-∞, +∞) interval to the (0, 1) interval. Since ||X g_id -Y g_id The value range of ||1 is [0,+∞), so the weight term 1-Sigmoid(||X g_id -Y g_id ||1) can be mapped to the interval (0,0.5]. In this case, the closer the distance between the two real high-resolution images is, the closer the value of the adjustment weight item is to 0.5, and the stronger the constraint effect of the cross-identity loss item is.
[0142] Step S7: Send the fused super-resolution face image and the corresponding high-resolution face to the high-resolution face discriminator, calculate the discrimination loss and update the network parameters, which is specifically expressed as follows:
[0143]
[0144]
[0145] Among them, ∥AB∥2 represents L2 loss calculation, and is the i-th layer fusion enhanced face spatial feature obtained in step S3, and and They are the real high-resolution images X gt and Y gt The reference space features are calculated by the training feature extraction network TU-Net, where TU-Net() represents the training feature extraction network TU-Net.
[0146] Step S8: Send the fused super-resolution face image and the corresponding high-resolution face to the high-resolution face discriminator, calculate the discrimination loss and update the network parameters respectively, which is specifically expressed as follows:
[0147]
[0148] Among them, D HR () represents a high-resolution face discriminator, whose main purpose is to compete with the cross-camera face fusion super-resolution model, so that the subject generation network can adjust the final output of its fusion super-resolution, and then obtain a high-quality face with a texture that is closer to the real distribution.
[0149] Optionally, in one embodiment of the present application, the calculation expression of the total loss function is:
[0150]
[0151] Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id , L adv They represent reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively. λ1, λ2, λ3, λ4 and λ5 represent the corresponding weights of reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively. L cid They represent the fusion loss and the cross-identity loss respectively, and λ6 and λ7 represent the corresponding weights of the fusion loss and the cross-identity loss respectively.
[0152] When training the network, the overall network parameters are updated according to the loss function to obtain a trained network model. The overall loss function is expressed as:
[0153]
[0154] Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id , L adv They represent reconstruction loss, pixel loss, perception loss, identity loss, and adversarial loss, respectively. They are commonly used loss items in face super-resolution tasks, and their corresponding weights are denoted as λ1, λ2, λ3, λ4, and λ5, respectively. L cid They represent fusion loss and cross-identity loss respectively, which are loss items specially designed for cross-camera face super-resolution problems, and the corresponding weights are denoted as λ6 and λ7. In the embodiment of the present application, the specific values used for each weight can be λ1=0.5, λ2=1, λ3=1, λ4=10, λ5=0.1, λ6=0.5, and λ7=5.
[0155] In step S104, the trained cross-camera face super-resolution network model is tested using a paired cross-camera low-resolution face image dataset to obtain a test face image, and the test face image is compared with the corresponding face image in the original high-definition face dataset to obtain a test result. When the test result meets a preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, so as to use the final cross-camera face super-resolution network model to perform cross-camera face fusion.
[0156] During the actual implementation process, the embodiments of the present application can use a paired cross-camera low-resolution face image dataset to test the trained cross-camera face super-resolution network model to obtain an output test face image, and then compare it with the corresponding face image in the original high-definition face dataset to determine whether the training result is qualified. If it is unqualified, the training is returned to the model. Otherwise, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model to achieve the purpose of utilizing the complementary information between cross-camera faces to obtain high-resolution fused faces with better quality and higher credibility.
[0157] By utilizing the channel correlation-based face fusion super-resolution network and embedding the facial feature fusion super-resolution module into the network, the low-resolution faces across cameras are fused and enhanced to high-resolution faces, successfully solving the problem of inconsistent cross-camera face degradation and severe degradation.
[0158] It should be noted that the preset convergence condition can be set by those skilled in the art according to actual conditions and is not specifically limited here.
[0159] According to the cross-camera face fusion super-resolution method based on channel correlation proposed in the embodiment of the present application, an original high-definition face dataset can be obtained, and the original high-definition face dataset can be used to construct a paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing, and the constructed cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model, and the trained cross-camera face super-resolution network model is tested using the paired cross-camera low-resolution face image dataset to obtain a test face image, and the test face image is compared with the corresponding face image in the original high-definition face dataset, so that when the test result meets the preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, and the final cross-camera face super-resolution network model is used to perform cross-camera face fusion, so as to utilize the complementarity between cross-camera low-resolution faces to reconstruct a high-quality and more reliable face image. Thereby, the technical problems in the related technologies, which are limited to blind restoration of single-frame facial images and focus on super-resolution of low-quality images with simple and known degradation processes, are solved; they are not suitable for cross-camera facial super-resolution conditions with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.
[0160] Next, a cross-camera face fusion super-resolution device based on channel correlation proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.
[0161] Figure 5 It is a block diagram of a cross-camera face fusion super-resolution device based on channel correlation according to an embodiment of the present application.
[0162] like Figure 5 As shown, the cross-camera face fusion super-resolution device 10 based on channel correlation is applied to the model building stage, wherein the device 10 includes:
[0163] Specifically, the acquisition module 101 is used to acquire an original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing.
[0164] The construction module 102 is used to construct a cross-camera face super-resolution network model, so as to utilize the cross-camera face super-resolution network model to fuse and enhance a low-resolution face image with a resolution less than a first preset resolution into a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution.
[0165] The training module 103 is used to train a cross-camera face super-resolution network model using a paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model.
[0166] The testing module 104 is used to test the trained cross-camera face super-resolution network model using a paired cross-camera low-resolution face image dataset to obtain a test face image, and compare the test face image with the corresponding face image in the original high-definition face dataset to obtain a test result, and when the test result meets a preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, so as to use the final cross-camera face super-resolution network model to perform cross-camera face fusion.
[0167] Optionally, in one embodiment of the present application, the training module 102 includes: an extraction unit, a first generation unit, a second generation unit, a reconstruction unit and an update unit.
[0168] Among them, the extraction unit is used to extract corresponding facial hidden features and facial spatial features from the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset.
[0169] The first generating unit is used to generate a fusion factor based on a low-resolution face image in a paired cross-camera low-resolution-high-resolution face image dataset.
[0170] The second generation unit is used to combine the face spatial features and the fusion factor to generate the face fusion enhanced spatial features of the corresponding layer; the conversion unit is used to convert the face latent features into latent codes.
[0171] The reconstruction unit is used to utilize hidden coding and fusion to enhance the spatial features of the face and reconstruct a fused super-resolution face image.
[0172] An updating unit is used to calculate a total loss function according to the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtain a calculation result, and use the calculation result to update the parameters of the cross-camera face super-resolution network model.
[0173] Optionally, in one embodiment of the present application, the calculation expression of the total loss function is:
[0174]
[0175] Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id, L adv They represent reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively. λ1, λ2, λ3, λ4 and λ5 represent the corresponding weights of reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively. L cid They represent the fusion loss and the cross-identity loss respectively, and λ6 and λ7 represent the corresponding weights of the fusion loss and the cross-identity loss respectively.
[0176] Optionally, in one embodiment of the present application, the second generation unit includes: a generation subunit, a first processing subunit, a second processing subunit and a fusion subunit.
[0177] Among them, the generation subunit is used to generate corresponding identity features and fusion factors based on the low-resolution face image.
[0178] The first processing subunit is used to preprocess the face spatial features and fusion factors using a preset convolutional network block to obtain preprocessed features.
[0179] The second processing subunit is used to normalize the preprocessing features to obtain normalized features, and use the normalized features to obtain corresponding matching weights.
[0180] The fusion subunit is used to match and fuse the preprocessed features according to the matching weights using the preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block to obtain the face fusion enhanced spatial features.
[0181] It should be noted that the aforementioned explanation of the embodiment of the cross-camera face fusion super-resolution method based on channel correlation is also applicable to the cross-camera face fusion super-resolution device based on channel correlation of this embodiment, and will not be repeated here.
[0182] According to the cross-camera face fusion super-resolution device based on channel correlation proposed in the embodiment of the present application, an original high-definition face dataset can be obtained, and the original high-definition face dataset can be used to construct a paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing, and the paired cross-camera low-resolution-high-resolution face image dataset is used to train the constructed cross-camera face super-resolution network model to obtain the trained cross-camera face super-resolution network model, and the trained cross-camera face super-resolution network model is tested using the paired cross-camera low-resolution face image dataset to obtain a test face image, and the test face image is compared with the corresponding face image in the original high-definition face dataset, so that when the test result meets the preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, and the final cross-camera face super-resolution network model is used to perform cross-camera face fusion, so as to utilize the complementarity between cross-camera low-resolution faces to reconstruct a high-quality and more reliable face image. Thereby, the technical problems in the related technologies, which are limited to blind restoration of single-frame facial images and focus on super-resolution of low-quality images with simple and known degradation processes, are solved; they are not suitable for cross-camera facial super-resolution conditions with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.
[0183] The above is a description of the cross-camera face fusion super-resolution method based on channel correlation in the model building stage of an embodiment of the present application. The following is a description of the cross-camera face fusion super-resolution method based on channel correlation in the model application stage of an embodiment of the present application.
[0184] Specifically, Figure 6 A flow chart of a cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of the present application.
[0185] like Figure 6 As shown, the cross-camera face fusion super-resolution method based on channel correlation is applied to the model use stage, wherein the method includes the following steps:
[0186] In step S601, multiple face images of the same target and the same modality are acquired from different cameras.
[0187] In the actual implementation process, when the embodiment of the present application is applied, multiple face images of the same target and the same modality from different cameras can be fused and enhanced to obtain a high-resolution face image.
[0188] Compared with related technologies, there is no need to utilize the identity correlation between faces of different modalities from the loss function level. By embedding a multi-camera face fusion super-resolution network and utilizing the feature correlation between different faces, the super-resolution quality is further improved, so that the embodiment of the present application does not require pre-migration during actual execution, which is more simplified.
[0189] In step S602, multiple facial images are input into a pre-built cross-camera face super-resolution network model to obtain reconstructed fused and enhanced face images with a resolution higher than a preset resolution threshold, wherein the cross-camera face super-resolution network model is trained by pairing a cross-camera low-resolution-high-resolution face image dataset.
[0190] According to the cross-camera face fusion super-resolution method based on channel correlation proposed in the embodiment of the present application, the complementary information between cross-camera faces can be used to obtain a high-resolution fused face with better quality and higher credibility, and the cross-camera low-resolution face fusion can be enhanced to a high-resolution face, successfully solving the problem of inconsistent cross-camera face degradation process and severe degradation degree. Thus, the technical problem that the related technology is limited to blind restoration of single-frame face images, and focuses on super-resolution of low-quality images with simple and known degradation process, is not applicable to cross-camera face super-resolution working conditions with inconsistent degradation process and severe degradation degree, and is difficult to apply to security and criminal investigation scenes is solved.
[0191] Next, a cross-camera face fusion super-resolution device based on channel correlation proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.
[0192] Figure 7 It is a block diagram of a cross-camera face fusion super-resolution device based on channel correlation according to an embodiment of the present application.
[0193] like Figure 7 As shown, the cross-camera face fusion super-resolution device 20 based on channel correlation is applied to the model use stage, wherein the device 20 includes: an acquisition module 201 and a reconstruction module 202.
[0194] Specifically, the acquisition module 201 is used to acquire multiple face images of the same target and the same modality from different cameras.
[0195] The reconstruction module 202 is used to input multiple facial images into a pre-built cross-camera facial super-resolution network model to obtain a reconstructed fusion-enhanced facial image with a resolution higher than a preset resolution threshold, wherein the cross-camera facial super-resolution network model is trained by pairing a cross-camera low-resolution-high-resolution facial image dataset.
[0196] It should be noted that the aforementioned explanation of the embodiment of the cross-camera face fusion super-resolution method based on channel correlation is also applicable to the cross-camera face fusion super-resolution device based on channel correlation of this embodiment, and will not be repeated here.
[0197] According to the cross-camera face fusion super-resolution device based on channel correlation proposed in the embodiment of the present application, the complementary information between cross-camera faces can be used to obtain a high-resolution fused face with better quality and higher credibility, and the cross-camera low-resolution face fusion can be enhanced to a high-resolution face, successfully solving the problem of inconsistent cross-camera face degradation process and severe degradation degree. Thus, the technical problem that the related technology is limited to blind restoration of single-frame face images, and focuses on super-resolution of low-quality images with simple and known degradation process, is not applicable to cross-camera face super-resolution working conditions with inconsistent degradation process and severe degradation degree, and is difficult to apply to security and criminal investigation scenes is solved.
[0198] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:
[0199] A memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802. When the processor 802 executes the program, the cross-camera face fusion super-resolution method based on channel correlation provided in the above embodiment is implemented.
[0200] Furthermore, the electronic device further comprises:
[0201] The communication interface 803 is used for communication between the memory 801 and the processor 802 .
[0202] The memory 801 is used to store computer programs that can be executed on the processor 802 .
[0203] The memory 801 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.
[0204] If the memory 801, the processor 802 and the communication interface 803 are implemented independently, the communication interface 803, the memory 801 and the processor 802 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0205] Optionally, in a specific implementation, if the memory 801, the processor 802 and the communication interface 803 are integrated on a chip, the memory 801, the processor 802 and the communication interface 803 can communicate with each other through an internal interface.
[0206] The processor 802 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0207] This embodiment also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned cross-camera face fusion super-resolution method based on channel correlation is implemented.
[0208] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the cross-camera face fusion super-resolution method based on channel correlation provided by an embodiment of the present invention.
[0209] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0210] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0211] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.
[0212] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.
[0213] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0214] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.
[0215] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0216] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.
Claims
1. A cross-camera face fusion super-resolution method based on channel correlation, characterized in that: Applied to the model building stage, wherein the method comprises the following steps: Obtain an original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; Constructing a cross-camera face super-resolution network model, and using the cross-camera face super-resolution network model to fuse and enhance a low-resolution face image with a resolution less than a first preset resolution into a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution; Using the paired cross-camera low-resolution-high-resolution face image dataset to train the cross-camera face super-resolution network model to obtain a trained cross-camera face super-resolution network model; The trained cross-camera face super-resolution network model is tested using the paired cross-camera low-resolution face image dataset to obtain a test face image, and the test face image is compared with the corresponding face image in the original high-definition face dataset to obtain a test result, and when the test result meets a preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, so as to use the final cross-camera face super-resolution network model to perform cross-camera face fusion.
2. The method according to claim 1, characterized in that The method of training the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model includes: Extracting corresponding facial hidden features and facial spatial features from the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; Generate a fusion factor based on the low-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset; Combining the face spatial feature and the fusion factor to generate the face fusion enhanced spatial feature of the corresponding layer; Converting the facial latent features into latent codes; Reconstructing a fused super-resolution face image by using the latent coding and the fusion enhanced face spatial features; A total loss function is calculated based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset to obtain a calculation result, and the calculation result is used to update the parameters of the cross-camera face super-resolution network model.
3. The method according to claim 2, characterized in that The calculation expression of the total loss function is: Among them, λ represents the empirical weight setting between various losses, L pixel , L per , L id , L adv denote reconstruction loss, pixel loss, perceptual loss, identity loss and adversarial loss respectively, λ1, λ2, λ3, λ4 and λ5 denote the corresponding weights of the reconstruction loss, the pixel loss, the perceptual loss, the identity loss and the adversarial loss respectively, L cid They represent the fusion loss and the cross-identity loss respectively, and λ6 and λ7 represent the corresponding weights of the fusion loss and the cross-identity loss respectively.
4. The method according to claim 2, characterized in that: The step of combining the face spatial feature and the fusion factor to generate the face fusion enhanced spatial feature of the corresponding layer includes: Generating corresponding identity features and fusion factors based on the low-resolution face image; Preprocessing the face spatial features and the fusion factors using a preset convolutional network block to obtain preprocessed features; Normalizing the preprocessing features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; According to the matching weight, the pre-processed features are matched and fused using a preset matching convolutional network block based on channel correlation adjustment to obtain the face fusion enhanced spatial features.
5. A cross-camera face fusion super-resolution method based on channel correlation, characterized in that: Applied to the model use phase, wherein the method comprises the following steps: Acquire multiple face images of the same target and modality from different cameras; The multiple facial images are input into a pre-built cross-camera face super-resolution network model to obtain reconstructed fused and enhanced facial images with a resolution higher than a preset resolution threshold, wherein the cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.
6. A cross-camera face fusion super-resolution device based on channel correlation, characterized in that: Applied to the model building stage, wherein the device comprises: An acquisition module is used to acquire an original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; A construction module is used to construct a cross-camera face super-resolution network model, so as to use the cross-camera face super-resolution network model to fuse and enhance a low-resolution face image with a resolution less than a first preset resolution into a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution; A training module, used for training the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain a trained cross-camera face super-resolution network model; A testing module is used to test the trained cross-camera face super-resolution network model using the paired cross-camera low-resolution face image dataset to obtain a test face image, and compare the test face image with the corresponding face image in the original high-definition face dataset to obtain a test result, and when the test result meets a preset convergence condition, use the trained cross-camera face super-resolution network model as the final cross-camera face super-resolution network model to perform cross-camera face fusion using the final cross-camera face super-resolution network model.
7. The device according to claim 6, characterized in that The training module includes: An extraction unit, configured to extract corresponding hidden facial features and spatial facial features from the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; A first generating unit is used to generate a fusion factor based on the low-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset; A second generating unit, configured to generate a face fusion enhanced spatial feature of a corresponding layer by combining the face spatial feature and the fusion factor; A conversion unit, used for converting the facial latent features into latent codes; A reconstruction unit, used to reconstruct a fused super-resolution face image by using the latent coding and the fused enhanced face spatial features; An updating unit is used to calculate a total loss function according to the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtain a calculation result, and use the calculation result to update the parameters of the cross-camera face super-resolution network model.
8. A cross-camera face fusion super-resolution device based on channel correlation, characterized in that: Applied to the model use stage, wherein the device comprises: An acquisition module is used to acquire multiple face images of the same target and modality from different cameras; A reconstruction module is used to input the multiple facial images into a pre-built cross-camera face super-resolution network model to obtain a reconstructed fusion-enhanced facial image with a resolution higher than a preset resolution threshold, wherein the cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.
9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-camera face fusion super-resolution method based on channel correlation as described in any one of claims 1-4 or 5.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the cross-camera face fusion super-resolution method based on channel correlation as described in any one of claims 1-4 or 5.
Citation Information
Patent Citations
Pedestrian face super-resolution reconstruction method for monitoring scenes
CN108765279A
Face image super-resolution method based on dense residual neural network
CN110610464A
Human face image super-resolution method based on attention mechanism
CN111080513A
Aircraft skin surface defect detection and classification method
CN111340754A
Face super-resolution method based on spatial feature transformation and cross-scale feature integration
CN112270644A
