Cross-camera face fusion super-resolution method and device based on channel correlation

Abstract

The application relates to a cross-camera face fusion super-resolution method and device based on channel correlation, wherein the method comprises the following steps: constructing a training set and a test set by using an original high-definition face data set; constructing a cross-camera face super-resolution network model; training the cross-camera face super-resolution network model by using the training set; testing the trained cross-camera face super-resolution network model by using the test set, obtaining a test face image, and comparing the test face image with a corresponding face image in the original high-definition face data set; and in the case that the test result meets a preset convergence condition, taking the trained cross-camera face super-resolution network model as a final cross-camera face super-resolution network model. Thus, the technical problem that the related art is limited to blind restoration of a single frame face image, is not applicable to cross-camera face super-resolution working conditions in which a degradation process is not uniform and a degradation degree is serious, and is difficult to be applied to security and criminal investigation scenes is solved.

Description

Technical Field

[0001] This application relates to the field of digital image technology, and in particular to a cross-camera face fusion super-resolution method and apparatus based on channel correlation. Background Technology

[0002] In security and criminal investigation scenarios, facial images captured by surveillance cameras have become an increasingly important clue in daily public security and criminal investigations. However, in actual surveillance scenarios, due to limitations such as camera cost and image storage cost, coupled with the combined effects of environmental factors such as lighting conditions, shooting angle, and shooting distance, the facial images actually captured and stored are often low-quality images with significant degradation. This undoubtedly increases the difficulty of identifying suspects using surveillance facial images in actual criminal investigation scenarios. Due to the high coverage of surveillance cameras, multiple different cameras often capture multiple facial images of suspects at different times in actual criminal investigation scenarios. This provides a possible way to address the problem of missing image information under severe degradation, thereby improving the image clarity and identification accuracy of super-resolution faces. However, in practice, faces captured by different cameras often exhibit inconsistent degradation processes and severe degradation degrees, making direct face fusion and super-resolution difficult.

[0003] With the development of deep learning in computer vision, significant progress has been made in the research of multi-frame image fusion super-resolution algorithms, yielding remarkable results. Among related technologies, one approach is to use face generation priors as the basis for reconstruction, introducing the StyleGAN network, which possesses powerful face generation capabilities, into the face-blind reconstruction task through a simple spatial feature transformation layer, thereby achieving high-quality face-blind reconstruction images. Another approach is to significantly improve super-resolution performance by combining additional information from multiple reference images and with the help of an attention mechanism. Furthermore, it is possible to achieve super-resolution of VR videos using a recurrent structure with an attention mechanism on a specially constructed dataset.

[0004] However, in related technologies, blind face restoration algorithms are limited to blind restoration of single-frame face images and cannot be directly used to solve the problem of cross-camera face super-resolution. Reference frame-based image super-resolution algorithms use high-quality images to improve the quality of low-resolution input images, but in security and criminal investigation scenarios, high-quality photos of relevant personnel cannot be used as reference images, which does not meet practical needs. Furthermore, related technologies mainly focus on super-resolution of low-quality images with simple and known degradation processes, and are not suitable for cross-camera face super-resolution situations where degradation processes are inconsistent or severe, requiring further improvement. Summary of the Invention

[0005] This application provides a cross-camera face fusion super-resolution method and apparatus based on channel correlation to solve the technical problems in related technologies, which are limited to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. They are not applicable to cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.

[0006] The first aspect of this application provides a cross-camera face fusion super-resolution method based on channel correlation, applied in the model building stage. The method includes the following steps: acquiring an original high-definition face dataset, and using the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; constructing a cross-camera face super-resolution network model to fuse and enhance low-resolution face images with a resolution less than a first preset resolution to high-resolution face images with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution. A preset resolution is used; the cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model; the trained cross-camera face super-resolution network model is tested using the paired cross-camera low-resolution face image dataset to obtain test face images, and the test face images are compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion.

[0007] Optionally, in one embodiment of this application, training the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model includes: extracting corresponding latent facial features and spatial facial features from low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating a fusion factor based on the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating face fusion enhancement spatial features for the corresponding layer by combining the face spatial features and the fusion factor; converting the latent facial features into hidden codes; reconstructing a fused super-resolution face image using the hidden codes and the fused enhanced face spatial features; calculating a total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtaining the calculation result, and updating the parameters of the cross-camera face super-resolution network model using the calculation result.

[0008] Optionally, in one embodiment of this application, the calculation expression for the total loss function is:

[0009]

[0010]

[0011] Where λ represents the empirical weighting of the various losses. L pixel L per L id L adv Let λ1, λ2, λ3, λ4, and λ5 represent the reconstruction loss, pixel loss, perception loss, identity loss, and adversarial loss, respectively, and let λ1, λ2, λ3, λ4, and λ5 represent the corresponding weights of the reconstruction loss, pixel loss, perception loss, identity loss, and adversarial loss, respectively. L cid λ6 and λ7 represent the fusion loss and cross-identity loss, respectively, and represent the weights of the fusion loss and cross-identity loss, respectively.

[0012] Optionally, in one embodiment of this application, the step of generating face fusion enhancement spatial features of the corresponding layer by combining the face spatial features and the fusion factor includes: generating corresponding identity features and fusion factors based on the low-resolution face image; preprocessing the face spatial features and the fusion factor using a preset convolutional network block to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and matching and fusing the preprocessed features using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block according to the matching weights to obtain the face fusion enhancement spatial features.

[0013] A second aspect of this application provides a cross-camera face fusion super-resolution method based on channel correlation, applied in the model usage stage. The method includes the following steps: acquiring multiple face images of the same target and the same modality from different cameras; inputting the multiple face images into a pre-constructed cross-camera face super-resolution network model to obtain a reconstructed, fused, and enhanced face image with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset.

[0014] A third aspect of this application provides a cross-camera face fusion super-resolution device based on channel correlation, applied in the model building stage. The device includes: an acquisition module for acquiring an original high-definition face dataset and using the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing; and a construction module for constructing a cross-camera face super-resolution network model to fuse and enhance low-resolution face images with a resolution less than a first preset resolution to high-resolution face images with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution. The system includes a resolution setting and a training module for training a cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model. A testing module is used to test the trained cross-camera face super-resolution network model using the paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are then compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet a preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion.

[0015] Optionally, in one embodiment of this application, the training module includes: an extraction unit, configured to extract corresponding latent facial features and spatial facial features from low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a first generation unit, configured to generate a fusion factor based on the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a second generation unit, configured to combine the spatial facial features and the fusion factor to generate corresponding layer's face fusion enhancement spatial features; a conversion unit, configured to convert the latent facial features into latent codes; a reconstruction unit, configured to reconstruct a fused super-resolution facial image using the latent codes and the fused enhanced facial spatial features; and an update unit, configured to calculate a total loss function based on the fused super-resolution facial image and the corresponding high-resolution facial image in the paired cross-camera low-resolution-high-resolution facial image dataset, obtain a calculation result, and update the parameters of the cross-camera face super-resolution network model using the calculation result.

[0016] Optionally, in one embodiment of this application, the calculation expression for the total loss function is:

[0017]

[0018] Where λ represents the empirical weighting of the various losses. L pixel L per L id L adv Let λ1, λ2, λ3, λ4, and λ5 represent the reconstruction loss, pixel loss, perception loss, identity loss, and adversarial loss, respectively, and let λ1, λ2, λ3, λ4, and λ5 represent the corresponding weights of the reconstruction loss, pixel loss, perception loss, identity loss, and adversarial loss, respectively. L cid λ6 and λ7 represent the fusion loss and cross-identity loss, respectively, and represent the weights of the fusion loss and cross-identity loss, respectively.

[0019] Optionally, in one embodiment of this application, the second generation unit includes: a generation subunit, configured to generate corresponding identity features and fusion factors based on the low-resolution face image; a first processing subunit, configured to preprocess the face spatial features and the fusion factors using a preset convolutional network block to obtain preprocessed features; a second processing subunit, configured to normalize the preprocessed features to obtain normalized features, and use the normalized features to obtain corresponding matching weights; and a fusion subunit, configured to match and fuse the preprocessed features according to the matching weights using a preset matching convolutional network block adjusted based on channel correlation and the preset convolutional network block to obtain the face fusion enhanced spatial features.

[0020] The fourth aspect of this application provides a cross-camera face fusion super-resolution device based on channel correlation, applied in the model usage stage. The device includes: an acquisition module for acquiring multiple face images of the same target and modality from different cameras; and a reconstruction module for inputting the multiple face images into a pre-constructed cross-camera face super-resolution network model to obtain a reconstructed and fused face image with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained from a paired cross-camera low-resolution-high-resolution face image dataset.

[0021] A fifth aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the channel-correlation-based cross-camera face fusion super-resolution method as described in the above embodiments.

[0022] A sixth aspect of this application provides a computer-readable storage medium storing computer instructions for causing the computer to perform the channel-correlation-based cross-camera face fusion super-resolution method as described in the above embodiments.

[0023] A seventh aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described cross-camera face fusion super-resolution method based on channel correlation.

[0024] This application embodiment can obtain the original high-definition face dataset, and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing. The constructed cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset, resulting in the trained cross-camera face super-resolution network model. The trained cross-camera face super-resolution network model is then tested using the paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are compared with the corresponding face images in the original high-definition face dataset. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model. This final cross-camera face super-resolution network model is then used for cross-camera face fusion, thereby utilizing the complementarity between cross-camera low-resolution faces to reconstruct high-quality, more reliable face images. This solves the technical problems of related technologies, which are limited to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are not applicable to cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.

[0025] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0027] Figure 1 This is a flowchart of a cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of this application;

[0028] Figure 2 This is a network diagram of a fusion factor generation module according to an embodiment of this application;

[0029] Figure 3 This is a network schematic diagram of a feature fusion super-resolution module according to an embodiment of this application;

[0030] Figure 4 This is a network diagram of a cross-camera face super-resolution network model according to an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of a cross-camera face fusion super-resolution device based on channel correlation provided in an embodiment of this application;

[0032] Figure 6 This is a flowchart of another cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of another cross-camera face fusion super-resolution device based on channel correlation provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0035] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0036] The following describes, with reference to the accompanying drawings, a method and apparatus for cross-camera face fusion super-resolution based on channel correlation, according to embodiments of this application. Addressing the limitations of the related technologies mentioned in the background section, which are restricted to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes, this application is unsuitable for cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, making it difficult to apply to security and criminal investigation scenarios. This application provides a method for cross-camera face fusion super-resolution based on channel correlation. In this method, an original high-definition face dataset can be acquired, and paired cross-camera low-resolution-high-resolution face image datasets for training and paired cross-camera low-resolution face image datasets for testing can be constructed using the original high-definition face dataset. A cross-camera face super-resolution network model is trained using an image dataset. This trained model is then tested using a paired cross-camera low-resolution face image dataset to obtain test face images. These test images are compared with corresponding face images from the original high-resolution face dataset. If the test results meet preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final model. This final model is then used for cross-camera face fusion, leveraging the complementarity between low-resolution faces from different cameras to reconstruct high-quality, more reliable face images. This solves the technical problems of related technologies, which are limited to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are unsuitable for cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, making them difficult to apply to security and criminal investigation scenarios.

[0037] Specifically, Figure 1 This is a flowchart illustrating a cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of this application.

[0038] like Figure 1 As shown, this cross-camera face fusion super-resolution method based on channel correlation is applied in the model building stage. The method includes the following steps:

[0039] In step S101, the original high-definition face dataset is obtained, and the original high-definition face dataset is used to construct a paired cross-camera low-resolution-high-resolution face image dataset for training the cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing.

[0040] In actual implementation, the embodiments of this application can preprocess the face images in the original high-definition face dataset to construct the paired cross-camera low-resolution-high-resolution face image dataset required for training the face super-resolution network model, and construct the paired cross-camera low-resolution face image dataset required for testing.

[0041] For example, in this embodiment, the RetinaFace model can be used to perform basic alignment of the original high-resolution face image dataset based on binocular calibration, and then bicubic interpolation is used to convert its resolution to 512×512. Afterwards, a simulated degradation model is used to generate low-quality-high-quality face image pairs; the dataset is divided into training and testing sets; based on identity information, paired cross-camera low-resolution-high-resolution face image datasets required for training the network and paired low-resolution face image datasets required for testing the network are formed. In this embodiment, [the text abruptly ends here, likely due to an incomplete translation or source material.] These are the low-resolution faces X and Y after the dataset processing, and the corresponding high-resolution faces X and Y, respectively.

[0042] In step S102, a cross-camera face super-resolution network model is constructed to fuse and enhance low-resolution face images with a resolution lower than a first preset resolution to a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution.

[0043] Cross-camera face super-resolution network models can be used to fuse and enhance low-resolution faces across cameras to high-resolution faces.

[0044] For example, such as Figure 2 and Figure 3 As shown, the cross-camera face super-resolution network model can structurally include: a feature extraction module, a feature fusion super-resolution module, and a feature reconstruction module.

[0045] In this embodiment, the basic features of two face images can be obtained by the feature extraction module. The obtained basic features are then input into the feature fusion super-resolution module to obtain the corresponding fused features. The fused features are then input into the feature reconstruction module to obtain two final output faces.

[0046] The feature extraction module of the cross-camera face super-resolution network model can adopt a 7-layer Unet structure, which can be subdivided into two parts: a feature encoder and a feature decoder, consisting of 7 downsampling modules and 7 upsampling modules. The cross-camera low-resolution face image is processed by the feature encoder to obtain the face latent features. The face latent features are processed by the feature decoder to obtain 7 layers of face spatial features and the corresponding image-level output.

[0047] The feature fusion super-resolution module of the cross-camera face super-resolution network can be divided into the fusion factor generation stage, the pre-convolution stage, the matching weight generation stage, and the matching convolution fusion stage during use.

[0048] The fusion factor generation stage is divided into an identity feature generation sub-stage and a fusion factor transformation sub-stage. The identity feature generation sub-stage can be performed by the face recognition network MagFace, while the fusion factor transformation sub-stage can be performed by the fusion factor transformation module Transfer. The fusion factor transformation sub-module consists of a fully connected layer, normalization, and activation functions ReLU and Sigmoid. The fusion factor generation sub-stage generates the corresponding identity features and fusion factors based on the input face image.

[0049] The pre-convolution stage can use a 3×3 convolutional network block to preprocess the face spatial features extracted by the feature extraction module and the fusion factors obtained in the fusion factor generation stage.

[0050] In the matching weight generation stage, GroupNormalization can be used to normalize the preprocessed features, and then the normalized features can be transformed and matrix multiplied to obtain the corresponding matching weights.

[0051] In the matching convolution fusion stage, based on the obtained matching weights, a 3×3 matching convolutional network block adjusted by channel correlation and a 3×3 convolutional network block are used to match and fuse the preprocessed features to obtain the final fused enhanced features.

[0052] Combination Figure 2 and Figure 3 As shown, the construction process may include the following steps:

[0053] Step S1: Cross-camera face super-resolution network model with low-resolution face images For input;

[0054] Step S2: Input the two face images obtained in step S1 into the feature extraction module to obtain the corresponding latent face features and face spatial features;

[0055] In the embodiments of this application:

[0056] 1) Feature encoder for face images and Using the given information as input, the corresponding latent facial features are obtained. The process is as follows:

[0057]

[0058] Where Encoder() represents the feature encoder, X feature Yfeature This corresponds to the hidden features of the face.

[0059] 2) The feature decoder takes the latent features of the face as input and obtains the spatial features of the face at various levels and the corresponding image-level output. The process is as follows:

[0060]

[0061]

[0062] Among them, Decoder i () represents the i-th layer of the feature decoder. For the corresponding i-th layer of face space features, This is the image-level output corresponding to the i-th layer.

[0063] Step S3: Input the face and face spatial features obtained in steps S1 and S2 into the feature fusion super-resolution module to generate face fusion enhanced spatial features;

[0064] In this embodiment, the feature fusion super-resolution module is divided into a fusion factor generation stage, a pre-convolution stage, a matching weight generation stage, and a matching convolution fusion stage. The fusion factor generation stage is further divided into an identity feature generation sub-stage and a fusion factor transformation sub-stage.

[0065] 1) MagFace, a facial recognition network, uses facial images... Given the input, the corresponding identity features are obtained. The process is as follows:

[0066]

[0067] Where MagFace() represents a face recognition network, X id Y id This indicates the corresponding identity characteristics.

[0068] 2) The fusion factor conversion submodule Transfer uses identity feature X id Y id The input is used to generate the corresponding fusion factor. The fusion factor transformation submodule consists of 5 layers. The first 4 layers are composed of a cascaded Linear, LayerNorm, and ReLU layer, and the last layer is composed of a cascaded Linear and Sigmoid layer. The process is represented as follows:

[0069]

[0070]

[0071]

[0072] The Transfer() function represents the fusion factor conversion submodule. This represents the generated fusion factor set. This represents the fusion factor applied to the i-th layer face feature fusion super-resolution network.

[0073] 3) The preconvolution module uses facial spatial features and the corresponding fusion factor The input is used to obtain preprocessed features. The preconvolutional module consists of a 3×3 Conv layer and a LeakyReLU layer. The process is as follows:

[0074]

[0075]

[0076] Where LRelu() represents a LeakyRelu layer, and con3() represents a 3×3 Conv layer. This indicates the preprocessing features of the corresponding layer.

[0077] 4) Matching weight generation module to preprocess features The corresponding relevance matching weights are obtained from the input. The matching weight generation module consists of a GroupNorm layer, a Sigmoid layer, and two 1×1 Conv layers, and also includes feature concatenation, matrix transformation, and matrix multiplication operations. Its process is represented as follows:

[0078]

[0079]

[0080]

[0081] Where con1() represents a 1×1 Conv layer, GNorm() represents a GroupNormalization layer, Concat() represents a feature concatenation operation, Sigmoid() represents a Sigmoid layer, and Resize() represents a matrix transformation operation. W represents the matrix multiplication operation. i Weights are matched to the generated relevance.

[0082] 5) Matching convolutional fusion module based on facial spatial features and preprocessing features And relevance matching weight W i The input is used to obtain fused and enhanced spatial features. The matched convolutional fusion module consists of a 3×3 matched convolutional layer adjusted based on channel correlation, a LeakyReLU layer, and a 3×3 Conv layer, and also includes feature splitting and residual addition operations. The process is represented as follows:

[0083]

[0084] Among them, C out I represents the number of channels in the output feature. j and K (i,j) These represent the j-th channel of the input feature and the j-th channel of the i-th convolutional kernel, respectively. Conv() represents a single-channel convolution operation, and W... (i,j) This represents the channel weight value of the i-th weight, O i This represents the i-th channel of the output feature, and the formula is the 3×3 matched convolutional layer based on channel correlation adjustment.

[0085]

[0086]

[0087]

[0088] In this context, con3() represents a 3×3 Conv layer, LRelu() represents a LeakyRelu layer, Concat() represents a feature concatenation operation, com3() represents a 3×3 matched convolutional layer adjusted based on channel correlation, and Split() represents a feature splitting operation. To enhance spatial features through integration.

[0089] Step S4: Input the hidden facial features and face fusion enhancement spatial features obtained in steps S2 and S3 into the face reconstruction module to generate a high-quality fused super-resolution face;

[0090] In the embodiments of this application:

[0091] 1) Feature encoding layer MLP uses face latent features X feature Y feature The input is used to obtain a hidden face code that better matches the initial feature distribution of the StyleGAN face reconstruction network. The feature encoding layer MLP is a single fully connected network layer. The process is as follows:

[0092] X code =MLP(X feature ),Ycode =MLP(Y feature )

[0093] Where MLP() represents the feature encoding layer, X code Y code Hidden encoding for the generated face.

[0094] 2) The face reconstruction module uses face hidden coding X code Y code Enhancing spatial features through face fusion The input is used to obtain a high-quality fused super-resolution face. The face reconstruction module is a pre-trained StyleGAN network, which uses a progressive generation method to reconstruct high-quality face images based on facial features. The process is as follows:

[0095]

[0096]

[0097] Among them, StyleGAN i () represents the i-th layer generator subnetwork of StyleGAN, and This is the corresponding i-th layer image level output, and The final output of the network is a high-quality fused super-resolution face image.

[0098] It should be noted that the first preset resolution and the second preset resolution can be set by those skilled in the art based on the criteria for classifying low resolution and high resolution, and no specific restrictions are imposed here.

[0099] In step S103, a cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model.

[0100] As one possible approach, after constructing a cross-camera face super-resolution network model, it can utilize paired cross-camera low-resolution-high-resolution face image datasets for paired learning, thereby super-resolutioning cross-camera faces into high-resolution faces and obtaining fused and enhanced high-resolution faces.

[0101] Optionally, in one embodiment of this application, a cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model. This includes: extracting corresponding latent facial features and spatial facial features from low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating a fusion factor based on the low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; combining the spatial facial features and the fusion factor to generate face fusion enhancement spatial features for the corresponding layer; converting the latent facial features into latent codes; reconstructing a fused super-resolution face image using the latent codes and the fused enhanced face spatial features; calculating the total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtaining the calculation result, and updating the parameters of the cross-camera face super-resolution network model using the calculation result.

[0102] The process of generating face fusion enhancement spatial features for the corresponding layer by combining face spatial features and fusion factors includes: generating corresponding identity features and fusion factors based on low-resolution face images; preprocessing the face spatial features and fusion factors using preset convolutional network blocks to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and matching and fusing the preprocessed features using preset matching convolutional network blocks based on channel correlation adjustment and preset convolutional network blocks according to the matching weights to obtain face fusion enhancement spatial features.

[0103] like Figure 4 As shown, the cross-camera face super-resolution network model is a two-branch generative adversarial network. Its structure may include: a face feature extraction module, a face feature fusion super-resolution module, a face feature reconstruction module, and a high-resolution face discrimination module. The embodiments of this application can use the two-branch structure for joint training.

[0104] The facial feature extraction module can adopt a seven-layer Unet structure.

[0105] The face feature fusion super-resolution module can be obtained from step S102 and embedded in the cross-camera face super-resolution network.

[0106] The face feature reconstruction module can be a pre-trained face reconstruction network, StyleGAN.

[0107] The high-resolution face discrimination module can adopt the same discriminator structure as StyleGAN.

[0108] The training process may include the following steps:

[0109] Step S1: Input the low-resolution face images from the paired cross-camera low-resolution-high-resolution face image dataset into the feature extraction network of the face feature extraction module to extract latent face features and face spatial features, which can be expressed as the following formula:

[0110]

[0111]

[0112]

[0113] Where Encoder() represents the feature encoder, X feature Y feature For the corresponding hidden facial features, Decoder i () represents the i-th layer of the feature decoder. For the corresponding i-th layer of face space features, This is the image-level output corresponding to the i-th layer.

[0114] Step S2: Input the low-resolution face images from the paired cross-camera low-resolution-high-resolution face image dataset into the fusion factor generation module of the face feature fusion super-resolution module in the cross-camera face super-resolution network model to generate fusion factors, which can be expressed as the following formula:

[0115]

[0116]

[0117] Where FGen() represents the fusion factor generation module, This represents the fusion factor applied to the i-th layer face feature fusion super-resolution network.

[0118] Step S3: Input the facial spatial features and their matching fusion factors into the feature fusion super-resolution network of the corresponding layer to generate the fused enhanced spatial features of the corresponding layer, which can be expressed as the following formula:

[0119]

[0120] Among them, Fusion i () represents the feature fusion super-resolution module of the i-th layer. To enhance the spatial features of the generated fused human face.

[0121] Step S4: Input the latent facial features into the feature encoding network and convert them into latent codes, which can be expressed as the following formula:

[0122] X code =MLP(X feature ),Y code =MLP(Y feature )

[0123] Where MLP() represents the feature encoding layer, X code Y code Hidden encoding for the generated face.

[0124] Step S5: Input the hidden coding and fused enhanced face spatial features into the face reconstruction network to reconstruct the fused super-resolution face image, which can be expressed as the following formula:

[0125]

[0126]

[0127] Among them, StyleGAN i () represents the i-th layer generator subnetwork of StyleGAN, and This is the corresponding i-th layer image level output, and The final output of the network is a high-quality fused super-resolution face image.

[0128] Step S6: Calculate the total loss function and update the network parameters based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset.

[0129] The loss function can include: reconstruction loss, pixel loss, perceptual loss, identity loss, and cross-identity loss, and updates the network parameters, as shown below:

[0130]

[0131]

[0132] Where ∥AB∥1 represents the calculation of L1 loss, This refers to the image-level output of the i-th layer of the U-Net network in step S1, where the face feature extraction module is located. These represent downsampled images obtained iteratively from real high-resolution faces, with DownSample() representing a bilinear 2x downsampling operation.

[0133] L pixel =||X output -X gt||1+||Y output -Y gt ||1

[0134] L per =||X percep -X g_percep ||1+||X style -X g_style ||1

[0135] +||Y percep -Y g_percep ||1+||Y style -Y g_style ||1

[0136] X percep ,X style =VGG(X output ),X g_percep ,X g_style =VGG(X gt )

[0137] Among them, X percep X g_percep Y percep Y g_percep These represent the perceptual features generated from the X-branch output image, the X-branch true high-resolution image, the Y-branch output image, and the Y-branch high-resolution image, respectively. style X g_style Y style Y g_style represents the style features generated from the output image of the X branch, the real high-resolution image of the X branch, the output image of the Y branch, and the high-resolution image of the Y branch, respectively, and VGG() represents the pre-trained VGG-19 network.

[0138] L id =||X id -X g_id ||1+||Y id -Y g_id ||1

[0139] X id =ArcFace(X input ),X g_id =ArcFace(X gt )

[0140] L cid =||X id -Y id ||1*(1-Sigmoid(||X g_id -Y g_id ||1))

[0141] Among them, X id X g_id Y id Y g_id Let X represent the face recognition features generated from the output image of the X branch, the real high-resolution image of the X branch, the output image of the Y branch, and the high-resolution image of the Y branch, respectively. ArcFace() represents the pre-trained ArcFace network, and Sigmoid() represents the Sigmoid activation function, which maps variables from the (-∞, +∞) interval to the (0, 1) interval. Since ||X... g_id -Y g_id The value of ||1 is in the range [0, +∞), therefore the adaptive adjustment of the weight term 1-Sigmoid(||X) g_id -Y g_id ||1) can be mapped to the interval (0, 0.5]. In this case, the closer the distance between two real high-resolution images, the closer the value of the adjustment weight term is to 0.5, and the stronger the constraint effect of the cross identity loss term.

[0142] Step S7: Feed the fused super-resolution face image and the corresponding high-resolution face into the high-resolution face discriminator, calculate the discrimination loss and update the network parameters, as shown below:

[0143]

[0144]

[0145] Where ∥AB∥2 represents the calculation of L2 loss, and It is the i-th layer of fused and enhanced facial spatial features obtained in step S3, and and The real high-resolution image X gt and Y gt The reference space features are calculated by the feature extraction network TU-Net used for training. TU-Net() represents the feature extraction network TU-Net used for training.

[0146] Step S8: Feed the fused super-resolution face image and the corresponding high-resolution face into the high-resolution face discriminator, calculate the discrimination loss and update the network parameters respectively, as shown below:

[0147]

[0148] Among them, D HR () represents a high-resolution face discriminator, whose main purpose is to compete with the cross-camera face fusion super-resolution model, thereby enabling the subject generation network to adjust the final output of its fusion super-resolution model, and thus obtain a high-quality face with texture representation that is closer to the real distribution.

[0149] Optionally, in one embodiment of this application, the calculation expression for the total loss function is:

[0150]

[0151] Where λ represents the empirical weighting of the various losses. L pixel L per L id L adv Let λ1, λ2, λ3, λ4, and λ5 represent the reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively, and let λ1, λ2, λ3, λ4, and λ5 represent the corresponding weights of the reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively. L cid Let λfusion and λcross-identity represent the fusion loss and cross-identity loss, respectively, and let λ6 and λ7 represent the weights of the fusion loss and cross-identity loss, respectively.

[0152] During network training, the overall network parameters are updated according to the loss function to obtain the trained network model. The overall loss function is expressed as:

[0153]

[0154] Where λ represents the empirical weighting of the various losses, L pixel L per L id L adv λ1, λ2, λ3, λ4, and λ5 represent reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively, and are commonly used loss terms in face super-resolution tasks. Their corresponding weights are denoted as λ1, λ2, λ3, λ4, and λ5, respectively. L cid λ1 and λ2 represent the fusion loss and cross-identity loss, respectively, which are loss terms specifically designed for the cross-camera face super-resolution problem, and their corresponding weights are denoted as λ6 and λ7, respectively. In the embodiments of this application, the specific values ​​used for each weight can be λ1 = 0.5, λ2 = 1, λ3 = 1, λ4 = 10, λ5 = 0.1, λ6 = 0.5, and λ7 = 5.

[0155] In step S104, the trained cross-camera face super-resolution network model is tested using a paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are then compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion.

[0156] In actual implementation, the embodiments of this application can use a paired cross-camera low-resolution face image dataset to test the trained cross-camera face super-resolution network model, and obtain the output test face image. By comparing it with the corresponding face image in the original high-definition face dataset, it can be determined whether the training result is qualified. If it is not qualified, it is returned to training again. Otherwise, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model, so as to achieve the purpose of obtaining a higher quality and more reliable high-resolution fused face by utilizing the complementary information between cross-camera faces.

[0157] By utilizing a channel-correlation-based face fusion super-resolution network and embedding a face feature fusion super-resolution module into the network, low-resolution faces from different cameras are enhanced to high-resolution faces, successfully solving the problems of inconsistent face degradation processes and severe degradation levels across different cameras.

[0158] It should be noted that the preset convergence conditions can be set by those skilled in the art according to the actual situation, and no specific restrictions are imposed here.

[0159] According to the channel correlation-based cross-camera face fusion super-resolution method proposed in this application, an original high-definition face dataset can be obtained. A paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing are constructed using the original high-definition face dataset. A cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset, resulting in a trained cross-camera face super-resolution network model. The trained cross-camera face super-resolution network model is then tested using the paired cross-camera low-resolution face image dataset to obtain test face images. These test face images are compared with the corresponding face images in the original high-definition face dataset. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model. This final cross-camera face super-resolution network model is then used for cross-camera face fusion, thereby utilizing the complementarity between low-resolution faces from different cameras to reconstruct high-quality, more reliable face images. This solves the technical problems of related technologies, which are limited to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are not applicable to cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.

[0160] Next, referring to the accompanying drawings, a cross-camera face fusion super-resolution device based on channel correlation is described according to an embodiment of this application.

[0161] Figure 5 This is a block diagram of a cross-camera face fusion super-resolution device based on channel correlation according to an embodiment of this application.

[0162] like Figure 5 As shown, the cross-camera face fusion super-resolution device 10 based on channel correlation is applied in the model building stage. The device 10 includes:

[0163] Specifically, module 101 is used to acquire the original high-definition face dataset and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training the cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing.

[0164] The construction module 102 is used to construct a cross-camera face super-resolution network model, so as to use the cross-camera face super-resolution network model to fuse and enhance low-resolution face images with a resolution lower than a first preset resolution to a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution.

[0165] Training module 103 is used to train a cross-camera face super-resolution network model using paired cross-camera low-resolution-high-resolution face image datasets, and obtain the trained cross-camera face super-resolution network model.

[0166] The testing module 104 is used to test the trained cross-camera face super-resolution network model using a paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are then compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion.

[0167] Optionally, in one embodiment of this application, the training module 102 includes: an extraction unit, a first generation unit, a second generation unit, a reconstruction unit, and an update unit.

[0168] The extraction unit is used to extract corresponding latent facial features and spatial facial features from low-resolution facial images in a paired cross-camera low-resolution-high-resolution facial image dataset.

[0169] The first generation unit is used to generate fusion factors based on low-resolution face images from a paired cross-camera low-resolution-high-resolution face image dataset.

[0170] The second generation unit is used to combine facial spatial features and fusion factors to generate facial fusion enhancement spatial features for the corresponding layer; the transformation unit is used to convert facial latent features into latent codes.

[0171] The reconstruction unit is used to reconstruct a fused super-resolution face image by utilizing implicit coding and fusion to enhance facial spatial features.

[0172] The update unit is used to calculate the total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtain the calculation result, and use the calculation result to update the parameters of the cross-camera face super-resolution network model.

[0173] Optionally, in one embodiment of this application, the calculation expression for the total loss function is:

[0174]

[0175] Where λ represents the empirical weighting of the various losses. L pixel L per L idL adv Let λ1, λ2, λ3, λ4, and λ5 represent the reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively, and let λ1, λ2, λ3, λ4, and λ5 represent the corresponding weights of the reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively. L cid Let λfusion and λcross-identity represent the fusion loss and cross-identity loss, respectively, and let λ6 and λ7 represent the weights of the fusion loss and cross-identity loss, respectively.

[0176] Optionally, in one embodiment of this application, the second generation unit includes: a generation subunit, a first processing subunit, a second processing subunit, and a fusion subunit.

[0177] The generation subunit is used to generate corresponding identity features and fusion factors based on low-resolution face images.

[0178] The first processing subunit is used to preprocess the face spatial features and fusion factors using a preset convolutional network block to obtain preprocessed features.

[0179] The second processing subunit is used to normalize the preprocessed features to obtain normalized features, and then use the normalized features to obtain the corresponding matching weights.

[0180] The fusion subunit is used to match and fuse preprocessed features based on matching weights and using a preset matching convolutional network block and a preset convolutional network block adjusted based on channel correlation, so as to obtain face fusion enhanced spatial features.

[0181] It should be noted that the foregoing explanation of the embodiment of the cross-camera face fusion super-resolution method based on channel correlation also applies to the cross-camera face fusion super-resolution device based on channel correlation in this embodiment, and will not be repeated here.

[0182] According to the channel correlation-based cross-camera face fusion super-resolution device proposed in this application, an original high-definition face dataset can be acquired. A paired cross-camera low-resolution-high-resolution face image dataset for training and a paired cross-camera low-resolution face image dataset for testing are constructed using the original high-definition face dataset. A cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset, resulting in a trained cross-camera face super-resolution network model. The trained cross-camera face super-resolution network model is then tested using the paired cross-camera low-resolution face image dataset to obtain test face images. These test face images are compared with the corresponding face images in the original high-definition face dataset. If the test results meet a preset convergence condition, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model. This final model is then used for cross-camera face fusion, thereby utilizing the complementarity between low-resolution faces from different cameras to reconstruct high-quality, more reliable face images. This solves the technical problems of related technologies, which are limited to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are not applicable to cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, and are difficult to apply to security and criminal investigation scenarios.

[0183] The above describes the channel-correlation-based cross-camera face fusion super-resolution method of this application in the model building stage. The following describes the channel-correlation-based cross-camera face fusion super-resolution method of this application in the model application stage.

[0184] Specifically, Figure 6 This is a flowchart illustrating a cross-camera face fusion super-resolution method based on channel correlation provided in an embodiment of this application.

[0185] like Figure 6 As shown, this cross-camera face fusion super-resolution method based on channel correlation is applied in the model usage stage. The method includes the following steps:

[0186] In step S601, multiple face images of the same target and the same modality are acquired from different cameras.

[0187] In actual implementation, when this application embodiment is applied, it can fuse and enhance multiple face images of the same target and the same modality from different cameras to obtain high-resolution face images.

[0188] Compared to related technologies, this method does not require utilizing the identity correlation between different modal faces at the loss function level. By embedding a multi-camera face fusion super-resolution network, it utilizes the feature correlation between different faces to further improve the super-resolution quality. This makes the embodiments of this application simpler in actual execution without the need for pre-transfer.

[0189] In step S602, multiple face images are input into a pre-built cross-camera face super-resolution network model to obtain reconstructed and enhanced face images with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.

[0190] The cross-camera face fusion super-resolution method based on channel correlation proposed in this application can utilize complementary information between faces from different cameras to obtain higher-quality, more reliable high-resolution fused faces. It enhances low-resolution faces from different cameras to high-resolution faces, successfully solving the problems of inconsistent degradation processes and severe degradation in cross-camera face fusion. This addresses the limitations of related technologies, which are restricted to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are unsuitable for cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, making them difficult to apply to security and criminal investigation scenarios.

[0191] Next, referring to the accompanying drawings, a cross-camera face fusion super-resolution device based on channel correlation is described according to an embodiment of this application.

[0192] Figure 7 This is a block diagram of a cross-camera face fusion super-resolution device based on channel correlation according to an embodiment of this application.

[0193] like Figure 7 As shown, the cross-camera face fusion super-resolution device 20 based on channel correlation is applied in the model usage stage. The device 20 includes an acquisition module 201 and a reconstruction module 202.

[0194] Specifically, the acquisition module 201 is used to acquire multiple face images of the same target and the same modality from different cameras.

[0195] The reconstruction module 202 is used to input multiple face images into a pre-built cross-camera face super-resolution network model to obtain a reconstructed and enhanced face image with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained by a paired cross-camera low-resolution-high-resolution face image dataset.

[0196] It should be noted that the foregoing explanation of the embodiment of the cross-camera face fusion super-resolution method based on channel correlation also applies to the cross-camera face fusion super-resolution device based on channel correlation in this embodiment, and will not be repeated here.

[0197] The cross-camera face fusion super-resolution device based on channel correlation proposed in this application can utilize complementary information between faces from different cameras to obtain higher-quality, more reliable high-resolution fused faces. It enhances low-resolution faces from different cameras to high-resolution faces, successfully solving the problems of inconsistent and severe degradation processes in cross-camera face fusion. This addresses the limitations of related technologies, which are restricted to blind restoration of single-frame face images and focus on super-resolution of low-quality images with simple and known degradation processes. These technologies are unsuitable for cross-camera face super-resolution scenarios with inconsistent degradation processes and severe degradation, making them difficult to apply to security and criminal investigation scenarios.

[0198] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include:

[0199] The system includes a memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802. When the processor 802 executes the program, it implements the channel-correlation-based cross-camera face fusion super-resolution method provided in the above embodiments.

[0200] Furthermore, electronic devices also include:

[0201] Communication interface 803 is used for communication between memory 801 and processor 802.

[0202] The memory 801 is used to store computer programs that can run on the processor 802.

[0203] The memory 801 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0204] If the memory 801, processor 802, and communication interface 803 are implemented independently, then the communication interface 803, memory 801, and processor 802 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0205] Optionally, in a specific implementation, if the memory 801, processor 802, and communication interface 803 are integrated on a single chip, then the memory 801, processor 802, and communication interface 803 can communicate with each other through an internal interface.

[0206] The processor 802 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0207] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described cross-camera face fusion super-resolution method based on channel correlation.

[0208] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-camera face fusion super-resolution method based on channel correlation provided in this embodiment of the invention.

[0209] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0210] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0211] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0212] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0213] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0214] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0216] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A cross-camera face fusion super-resolution method based on channel correlation, characterized in that, Applied to the model building phase, the method includes the following steps: Obtain the original high-resolution face dataset, and use the original high-resolution face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing. A cross-camera face super-resolution network model is constructed to fuse and enhance low-resolution face images with a resolution lower than a first preset resolution to high-resolution face images with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution. The cross-camera face super-resolution network model is trained using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model. The trained cross-camera face super-resolution network model is tested using the paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are then compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion. The step of training the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset to obtain the trained cross-camera face super-resolution network model includes: extracting corresponding latent facial features and spatial facial features from low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating fusion factors based on low-resolution face images in the paired cross-camera low-resolution-high-resolution face image dataset; generating face fusion enhancement spatial features for the corresponding layer by combining the face spatial features and the fusion factors; converting the latent facial features into hidden codes; reconstructing a fused super-resolution face image using the hidden codes and the fused enhanced face spatial features; calculating the total loss function based on the fused super-resolution face image and the corresponding high-resolution face image in the paired cross-camera low-resolution-high-resolution face image dataset, obtaining the calculation result, and updating the parameters of the cross-camera face super-resolution network model using the calculation result; The step of generating face fusion enhancement spatial features for the corresponding layer by combining the face spatial features and the fusion factor includes: generating corresponding identity features and fusion factors based on the low-resolution face image; preprocessing the face spatial features and the fusion factor using a preset convolutional network block to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and matching and fusing the preprocessed features using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block according to the matching weights to obtain the face fusion enhancement spatial features.

2. The method according to claim 1, characterized in that, The expression for calculating the total loss function is as follows: in, This indicates the empirical weighting settings between the various losses. , , , , These represent reconstruction loss, pixel loss, perceptual loss, identity loss, and adversarial loss, respectively. These represent the weights of the reconstruction loss, the pixel loss, the perception loss, the identity loss, and the adversarial loss, respectively. , These represent the fusion loss and cross-identity loss, respectively. These represent the weights of the fusion loss and the cross-identity loss, respectively.

3. A cross-camera face fusion super-resolution method based on channel correlation, characterized in that, When applied to the model usage phase, the method includes the following steps: Acquire multiple facial images of the same target and modality from different cameras; The multiple face images are input into a pre-constructed cross-camera face super-resolution network model to obtain reconstructed, fused, and enhanced face images with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset, which includes: The system extracts corresponding latent facial features and spatial facial features from low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; generates fusion factors based on the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; combines the spatial facial features and the fusion factors to generate corresponding layer-level face fusion enhancement spatial features; converts the latent facial features into hidden codes; reconstructs a fused super-resolution facial image using the hidden codes and the fused enhanced facial spatial features; calculates the total loss function based on the fused super-resolution facial image and the corresponding high-resolution facial image in the paired cross-camera low-resolution-high-resolution facial image dataset, obtains the calculation results, and updates the parameters of the cross-camera face super-resolution network model using the calculation results. The step of generating face fusion enhancement spatial features for the corresponding layer by combining the face spatial features and the fusion factor includes: generating corresponding identity features and fusion factors based on the low-resolution face image; preprocessing the face spatial features and the fusion factor using a preset convolutional network block to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and matching and fusing the preprocessed features using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block according to the matching weights to obtain the face fusion enhancement spatial features.

4. A cross-camera face fusion super-resolution device based on channel correlation, characterized in that, Applied to the model building phase, wherein the apparatus includes: The acquisition module is used to acquire the original high-definition face dataset and use the original high-definition face dataset to construct a paired cross-camera low-resolution-high-resolution face image dataset for training a cross-camera face super-resolution network model and a paired cross-camera low-resolution face image dataset for model testing. A construction module is used to construct a cross-camera face super-resolution network model, so as to use the cross-camera face super-resolution network model to fuse and enhance low-resolution face images with a resolution less than a first preset resolution to a high-resolution face image with a resolution greater than a second preset resolution, wherein the second preset resolution is greater than the first preset resolution; The training module is used to train the cross-camera face super-resolution network model using the paired cross-camera low-resolution-high-resolution face image dataset, and obtain the trained cross-camera face super-resolution network model. The testing module is used to test the trained cross-camera face super-resolution network model using the paired cross-camera low-resolution face image dataset to obtain test face images. The test face images are then compared with the corresponding face images in the original high-resolution face dataset to obtain test results. If the test results meet the preset convergence conditions, the trained cross-camera face super-resolution network model is used as the final cross-camera face super-resolution network model for cross-camera face fusion. The training module includes: an extraction unit for extracting corresponding latent facial features and spatial facial features from low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a first generation unit for generating a fusion factor based on the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; a second generation unit for generating face fusion enhancement spatial features for the corresponding layer by combining the facial spatial features and the fusion factor; a transformation unit for converting the latent facial features into hidden codes; a reconstruction unit for reconstructing a fused super-resolution facial image using the hidden codes and the fused enhanced facial spatial features; and an update unit for calculating a total loss function based on the fused super-resolution facial image and the corresponding high-resolution facial image in the paired cross-camera low-resolution-high-resolution facial image dataset, obtaining the calculation result, and updating the parameters of the cross-camera face super-resolution network model using the calculation result. The second generation unit includes: a generation subunit for generating corresponding identity features and fusion factors based on the low-resolution face image; a first processing subunit for preprocessing the face spatial features and fusion factors using a preset convolutional network block to obtain preprocessed features; a second processing subunit for normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and a fusion subunit for matching and fusing the preprocessed features according to the matching weights using a preset matching convolutional network block adjusted based on channel correlation and the preset convolutional network block to obtain the face fusion enhanced spatial features.

5. A cross-camera face fusion super-resolution device based on channel correlation, characterized in that, Applied to the model usage phase, wherein the apparatus includes: The acquisition module is used to acquire multiple face images of the same target and the same modality from different cameras; A reconstruction module is used to input the multiple face images into a pre-constructed cross-camera face super-resolution network model to obtain reconstructed, fused, and enhanced face images with a resolution higher than a preset resolution threshold. The cross-camera face super-resolution network model is trained using a paired cross-camera low-resolution-high-resolution face image dataset, and includes: The system extracts corresponding latent facial features and spatial facial features from low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; generates fusion factors based on the low-resolution facial images in the paired cross-camera low-resolution-high-resolution facial image dataset; combines the spatial facial features and the fusion factors to generate corresponding layer-level face fusion enhancement spatial features; converts the latent facial features into hidden codes; reconstructs a fused super-resolution facial image using the hidden codes and the fused enhanced facial spatial features; calculates the total loss function based on the fused super-resolution facial image and the corresponding high-resolution facial image in the paired cross-camera low-resolution-high-resolution facial image dataset, obtains the calculation results, and updates the parameters of the cross-camera face super-resolution network model using the calculation results. The step of generating face fusion enhancement spatial features for the corresponding layer by combining the face spatial features and the fusion factor includes: generating corresponding identity features and fusion factors based on the low-resolution face image; preprocessing the face spatial features and the fusion factor using a preset convolutional network block to obtain preprocessed features; normalizing the preprocessed features to obtain normalized features, and using the normalized features to obtain corresponding matching weights; and matching and fusing the preprocessed features using a preset matching convolutional network block based on channel correlation adjustment and the preset convolutional network block according to the matching weights to obtain the face fusion enhancement spatial features.

6. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the channel-correlation-based cross-camera face fusion super-resolution method as described in any one of claims 1-2 or 3.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the cross-camera face fusion super-resolution method based on channel correlation as described in any one of claims 1-2 or 3.

Images