Two-stage multi-scale adaptive low-resolution face recognition method and its application

Abstract

This invention discloses a two-stage, multi-scale, adaptive low-resolution face recognition method and application. The method comprises the following steps: 1. preprocessing a high-resolution face database and a real low-resolution face image database; 2. constructing high- and low-resolution representation learning networks and a momentum network; 3. pretraining the high-resolution representation learning network; and 4. training the constructed network model in a two-stage approach and using the trained model to extract representations from the low-resolution face images to be tested. The method can adaptively accept inputs of images with multiple resolutions, achieving better face recognition results in real low-resolution scenarios.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a two-stage multi-scale adaptive low-resolution face recognition method and application. Background Art

[0002] Face recognition, a key research topic in computer vision, has received extensive research and attention in recent years, with widespread applications in security, payment, and identity verification. Current face recognition methods target high-quality, high-resolution facial images. However, in real-world applications, the quality of facial images collected is uncertain, and the resolution varies. Only low-resolution adaptive face recognition can meet these needs.

[0003] Existing low-resolution face recognition methods can be divided into two main categories: super-resolution-based methods and subspace alignment-based methods. Super-resolution-based methods utilize super-resolution techniques to reconstruct facial details from low-resolution images and then feed the synthesized high-resolution images into conventional face recognition models. However, due to the significant domain difference between the synthesized high-resolution images and the original high-resolution images, super-resolution-based methods may introduce noise, resulting in poor generalization performance for face recognition. Furthermore, these methods suffer from high computational complexity and high memory consumption, making them unsuitable for real-time deployment. Subspace alignment-based methods map representations from different resolutions into a common subspace and minimize the distance between them. However, previous subspace alignment-based methods primarily align high- and low-resolution representations at the output layer, neglecting the guiding role of feature maps from intermediate layers and the class correlation between multiple low-resolution representations, which involves the geometric details of the face. Furthermore, these previous methods randomly downsample to a low-resolution, resulting in a chaotic resolution distribution.

[0004] In addition, existing low-resolution face recognition methods are mainly trained on pairs of low-resolution and high-resolution images. The low-resolution images are obtained by linear interpolation and downsampling using high-resolution images, and have distribution and noise differences from real low-resolution images. Therefore, the existing methods have poor performance in low-resolution face recognition in real scenes. Summary of the Invention

[0005] In order to overcome the shortcomings of the existing technology, the present invention proposes a two-stage multi-scale adaptive low-resolution face recognition method and application, in order to adaptively represent and extract images with multiple resolutions, thereby improving the face recognition effect in real low-resolution scenes.

[0006] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0007] The present invention provides a two-stage multi-scale adaptive low-resolution face recognition method characterized by the following steps: Step 1, pre-processing the high-resolution face database and the real low-resolution face image database respectively:

[0008] Step 1.1: Preprocessing of high-resolution face database:

[0009] The MTCNN neural network algorithm is used to analyze the high-resolution face database Perform face detection and correction on high-resolution face images and crop them to the input size to obtain a normalized high-resolution face image database;

[0010] Any high-resolution sample in the normalized high-resolution face image database is denoted as , the category of its label is recorded as ,and , Represents the total number of categories, and is interpolated by bicubic interpolation. Downsample to obtain four low-resolution samples and then upsample back to the original input size to obtain four synthesized low-resolution samples ;Depend on Construct a set of synthetic face images to obtain a synthetic face database ;

[0011] Step 1.2: Preprocessing of the real low-resolution face image database:

[0012] The real low-resolution face image database is interpolated by bicubic interpolation. The image size is cropped to the input size, and any low-resolution sample after cropping is recorded as , whose label category is ,Depend on Construct a set of real low-resolution face images, thereby obtaining a real low-resolution face image database ;

[0013] Step 2: Build a high-resolution representation learning network based on the ResNet-50 network , low-resolution representation learning network and Momentum Network ;

[0014] The ResNet-50 network includes an input layer, stacked blocks and output layer, where the input layer includes convolution layer, instance normalization layer, ReLU activation function, and maximum pooling layer. Each stacking block is connected by The model is composed of residual blocks, each of which consists of a convolutional layer, an instance normalization layer, and a ReLU activation function. The output layer includes an average pooling layer and a fully connected layer.

[0015] Let the high-resolution representation learning network All convolutional layers and fully connected layers in

[15] use Gaussian distribution for weight initialization;

[0016] Step 3: Pre-training high-resolution representation learning network :

[0017] Step 3.1: The synthesized face database Any high-resolution sample Input to the high-resolution representation learning network Processed in the ;

[0018] Formula (1) is used to establish the face recognition loss function based on additive angular margin for high-resolution samples. :

[0019] (1)

[0020] In formula (1), express and categories are The angle between the weights of the corresponding last fully connected layer; is the angular interval of category addition; express and categories are The angle between the weights of the corresponding last fully connected layer; Represents the synthetic face database Calculate the loss function for all high-resolution representations in and sum them up;

[0021] Step 3.2: Use SGD method to learn high-resolution representation network Perform pre-training and calculate the loss function Used to update network parameters. When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining a high-resolution representation learning network after training. and its optimal parameters;

[0022] Leverage trained high-resolution representation learning network The optimal parameters for low-resolution representation learning networks Initialize the network parameters;

[0023] Step 4: Use a two-stage approach to train the initialized low-resolution representation learning network , and use the low-resolution representation learning network after second-order training Extract representations of low-resolution face images to be tested:

[0024] Step 4.1: Multi-scale distillation stage:

[0025] Step 4.1.1: The synthesized face database Any high-resolution sample Input to the trained high-resolution representation learning network and respectively from Stacked blocks The features are extracted from the output of the output layer and the output layer, and the corresponding High-resolution feature maps at different scales and trained high-resolution representations ; Indicates the The output of the stacking block High-resolution feature maps at different scales;

[0026] Will The corresponding 4 synthetic low-resolution samples Input the initialized low-resolution representation learning network respectively and respectively from Stacked blocks The features are extracted from the output of the output layer and the output layer, and four types of low-resolution feature maps of four scales and four low-resolution representations, and one of them The low-resolution feature map of each scale and a low-resolution representation are respectively denoted as and ; Indicates the A first-order output of a stacked block Low-resolution feature maps at different scales;

[0027] Step 4.1.2: Use formula (2) to establish the face recognition loss function based on additive angular margin for low-resolution samples. :

[0028] (2)

[0029] In formula (2), express and categories are The angle between the weights of the corresponding last fully connected layer; express and categories are The angle between the weights of the corresponding last fully connected layer; Represents the synthetic face database Calculate the loss function for all low-resolution representations in and sum them up;

[0030] Step 4.1.3: Use formula (3) to establish the loss function of pixel value error :

[0031] (3)

[0032] In formula (3), Indicates traversing all high and low resolution The feature maps of each scale are summed up. Indicates traversing all high and low resolution representations and summing them up. represents the L2 norm;

[0033] Step 4.1.4: Any one of the feature maps at high resolution and four low resolutions After remodeling operation Then, we get a two-dimensional vector ; Represent the height, width and channel dimensions of the feature map respectively;

[0034] Use formula (4) to build affinity matrix :

[0035] (4)

[0036] In formula (4), is the softmax function, matrix multiplication, is the transpose of the matrix;

[0037] Use formula (5) to establish the loss function of affinity matrix distillation :

[0038] (5)

[0039] In formula (5), represents the L1 norm;

[0040] Step 4.1.5: High-resolution representation after training Corresponding high-resolution samples In a mini-batch of data, the low-resolution representation of the same category is recorded as , the low-resolution representations of different categories are , and constitute a positive match Represents joint distribution The samples in constitute negative matches Representing marginal distribution Samples in

[0041] Use formula (6) to establish the loss function that maximizes mutual information :

[0042] (6)

[0043] In formula (6), Represents softplus operation; express expectations;

[0044] Step 4.1.6: Use formula (7) to construct the overall objective function of the first stage :

[0045] (7)

[0046] In formula (7), , and are all weight factors;

[0047] Step 4.1.7: Use SGD to learn the initialized low-resolution representation network Perform the first stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the first stage of training. and its optimal parameters;

[0048] Step 4.2, multi-resolution clustering stage: Step 4.2.1, use the low-resolution representation learning network after the first stage training The optimal parameters for the momentum network Initialize parameters;

[0049] Using formula (8), a key memory is established to store four low-resolution representations. and a category memory for storing the corresponding labels :

[0050] (8)

[0051] In formula (8), is the length of the repository; Represents key memory Middle Four low-resolution representations, Represents class memory Middle tags;

[0052] The synthesized face database Randomly select samples, where any sample Four low-resolution samples of Input to the low-resolution representation learning network after the first stage training , correspondingly obtain the low-resolution representation learning network after the first stage of training Four low-resolution representations of the output , storing the four low-resolution representations into the key memory In the example, the label representing the corresponding is stored in the category memory middle;

[0053] Step 4.2.2, as described Samples within samples Four low-resolution samples of Input into the low-resolution representation learning network after the first stage training respectively And the initialized momentum network In the example, the low-resolution representation learning network after the first-order training is obtained. Four low-resolution representations of the output and Momentum Network Four low-resolution representations of the output , in turn Any representation As an anchor point, let the opposite of the anchor point be the momentum network Characterization of output at different resolutions and key memory Four low-resolution representations of the same category ;in, Representing Momentum Network Output division For any representation other than Four low-resolution representations of different categories ;

[0054] Use Equations (9)-(11) to establish a multi-resolution loss function :

[0055] (9)

[0056] (10) (11)

[0057] In formula (9)-formula (11), represents a synthetic low-resolution representation, Represents a set of synthetic low-resolution face image representations, including four low-resolution representations; is the temperature parameter; Represents key storage Middle The first Characterization of resolution, Momentum Network Output Resolution characterization, Represents key storage Middle The first Characterization of resolution; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; Indicates anchor point Calculate the sum of similarities with the low-resolution representations of all positive and negative pairs;

[0058] Step 4.2.3: Use SGD method to learn the low-resolution representation network after the first stage training Perform the second stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the second stage of training. ;

[0059] Step 4.2.4: Momentum Network Characterization of four output resolutions and the category of the tag Stored in key memory and category memory In order to update the dynamic queue;

[0060] Use formula (12) to update the initialized momentum network Network parameters , get the updated network parameters :

[0061] (12)

[0062] In formula (12), is the low-resolution representation learning network after the second stage training Parameters, is the momentum coefficient;

[0063] Step 4.3: Fine-tuning of real low-resolution face images:

[0064] Step 4.3.1: The real low-resolution face image database Any low-resolution sample in Input to the low-resolution representation learning network after second-stage training In the low-resolution representation , and in the new key memory Save in the new category memory Save the category of the tag ,in, Represents key storage Middle A low-resolution representation; Represents class memory Middle Tags, which will represent As a new anchor point, The opposite of , The negative pair of ;

[0065] Use formula (13) to establish the contrast loss function of real low resolution :

[0066] (13)

[0067] In formula (13), represents the true low-resolution representation, Represents a set of real low-resolution face image representations; Represents key storage Middle A low-resolution representation;

[0068] Step 4.3.2: Use SGD method to learn the low-resolution representation network after the second stage training Perform fine-tuning and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the optimal low-resolution representation learning network. ;

[0069] Step 4.4: Use the optimal low-resolution representation to learn the network The low-resolution face image to be tested is processed to obtain a representation of the low-resolution face image, and feature matching is performed with paired face images to achieve face verification, or feature matching is performed between the low-resolution face image to be tested and face images in the database to achieve face recognition.

[0070] The present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the low-resolution face recognition method, and the processor is configured to execute the program stored in the memory.

[0071] The present invention provides a computer-readable storage medium having a computer program stored thereon. The computer-readable storage medium is characterized in that when the computer program is executed by a processor, the steps of the low-resolution face recognition method are executed.

[0072] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0073] 1. By proposing a two-stage resolution-adaptive training strategy, the present invention can adaptively accept the input of multiple low-resolution facial images. The first-stage training transfers the structural and semantic attributes of high-resolution facial images, and the second-stage training obtains more compact clustering. This solves the problems of existing low-resolution face recognition methods being limited by the chaotic input resolution distribution and the large time and space costs, and realizes face recognition with more robust resolution.

[0074] 2. By proposing a multi-scale feature map distillation strategy, the present invention can effectively distill geometric information such as contours and appearance in high-resolution images into a low-resolution representation learning network, helping low-resolution images to supplement missing facial details. This makes up for the defect that previous methods only consider the representation of the output layer and ignore the feature maps of the intermediate layer. It can effectively enhance the guidance effect brought by high-resolution images and learn more discriminative features.

[0075] 3. The present invention designs a multi-resolution contrast loss function, which can adjust the representation distribution of multiple low-resolution images so that the representations of the same type are brought closer and the representations of different types are pushed away, thereby generating more compact clusters and further improving the face recognition effect of this method.

[0076] 4. By proposing a method for fine-tuning on real low-resolution facial images, the present invention can make the trained network adapt to both synthetic facial data and real facial data, solve the problem of the difference between synthetic low-resolution images and real low-resolution images, and improve its usability in actual environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 Schematic diagram of the process of the present invention;

[0078] Figure 2 This is a structural diagram of the multi-scale distillation stage model of the present invention;

[0079] Figure 3 This is a structural diagram of the multi-resolution clustering stage model of the present invention. DETAILED DESCRIPTION

[0080] In this embodiment, Figure 1 As shown in the figure, a two-stage multi-scale adaptive low-resolution face recognition method is performed in the following steps:

[0081] Step 1: Preprocess the high-resolution face database and the real low-resolution face image database separately:

[0082] Step 1.1: Preprocessing of high-resolution face database:

[0083] Use MTCNN (Multi-task convolutional neural network) neural network algorithm to analyze the high-resolution face database Perform face detection and correction on high-resolution face images and crop them to the input size to obtain a normalized high-resolution face image database; in this embodiment, the pixel size of all normalized face images is 112×112;

[0084] Any high-resolution sample in the normalized high-resolution face image database is denoted as , the category of its label is recorded as ,and , Represents the total number of categories, and is interpolated by bicubic interpolation. Downsampling is performed to obtain four low-resolution samples, which are then upsampled back to the original input size to unify the input size of the model, thereby obtaining four synthesized low-resolution samples. In this embodiment, the four low resolutions are 56×56, 28×28, 14×14 and 7×7; Construct a set of synthetic face images to obtain a synthetic face database ;

[0085] In this embodiment, the MS1M-ArcFace face image database is used; the MS1M-ArcFace face image database contains 582,265,3 face images collected from 857,42 volunteers;

[0086] Step 1.2: Preprocessing of the real low-resolution face image database:

[0087] Since the resolution of the real low-resolution face image is unknown, the real low-resolution face image database is interpolated by bicubic interpolation. The image size is cropped to the input size, and any low-resolution sample after cropping is recorded as , whose label category is ,Depend on Construct a set of real low-resolution face images, thereby obtaining a real low-resolution face image database ;

[0088] In this embodiment, the SCFace face image database is used; the SCFace face image database contains 416,00 face images collected from 130 volunteers, captured by five video surveillance cameras from three different distances (4.20 meters, 2.60 meters, and 1.00 meters), and 80 people's face images are randomly selected from them to form a real low-resolution face image database.

[0089] Step 2: Build a high-resolution representation learning network based on the ResNet-50 network , low-resolution representation learning network and Momentum Network ;

[0090] The ResNet-50 network includes the input layer, stacked blocks and an output layer. In this embodiment, Set to 4; the input layer includes a convolution layer with a convolution kernel size of 7×7 and a stride of 2, an instance normalization layer, a ReLU activation function, and a maximum pooling layer with a pooling size of 7×7 and a stride of 2. Each stacking block is connected by The network is composed of residual blocks, each of which consists of a convolutional layer, an instance normalization layer, and a ReLU activation function. In this embodiment, the four residual blocks are composed of 3, 4, 6, and 3 convolutional blocks, respectively. Each convolutional block includes three convolutional layers with convolution kernel sizes of 1×1, 3×3, and 1×1, respectively. The output layer includes an average pooling layer and a fully connected layer, which is mapped to a 512-dimensional representation through the fully connected layer.

[0091] Let the high-resolution representation learning network All convolutional layers and fully connected layers in

[15] use Gaussian distribution for weight initialization;

[0092] Step 3: Pre-training high-resolution representation learning network :

[0093] Step 3.1: Synthesized face database Any high-resolution sample Input to the high-resolution representation learning network Processed in the ;

[0094] Formula (1) is used to establish the face recognition loss function based on additive angular margin for high-resolution samples. :

[0095] (1)

[0096] In formula (1), express and categories are The angle between the weights of the corresponding last fully connected layer; is the angular interval of the category addition, in this embodiment, Set to 0.5; express and categories are The angle between the weights of the corresponding last fully connected layer; For all high-resolution characterizations Calculate the loss function and sum it; through the face recognition loss function The optimization of high-resolution representation learning network Can generate discriminative face representations;

[0097] Step 3.2: Use SGD (Stochastic Gradient Descent) method to learn high-resolution representation network Perform pre-training and calculate the loss function Used to update network parameters. When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining a high-resolution representation learning network after training. and its optimal parameters; in this embodiment, the number of training iterations is set to 20;

[0098] Leverage trained high-resolution representation learning network The optimal parameters for low-resolution representation learning networks Initialize the network parameters; after pre-training, the high-resolution representation learning network The parameters of are fixed, and then only the low-resolution representation learning network is trained ,This approach can maintain the guiding effect of high-resolution ,images;

[0099] Step 4: Use a two-stage approach to train the initialized low-resolution representation learning network , and use the trained low-resolution representation to learn the network Extract representations of low-resolution face images to be tested:

[0100] Step 4.1: Multi-scale distillation stage, which aligns feature maps and representations of images of different resolutions at multiple scales:

[0101] Step 4.1.1, as Figure 2 As shown, the synthetic face database Any high-resolution sample Input to the trained high-resolution representation learning network and respectively from Stacked blocks The features are extracted from the output of the output layer and the output layer, and the corresponding High-resolution feature maps at different scales and trained high-resolution representations ; Indicates the The output of the stacking block High-resolution feature maps of different scales; in this embodiment, the sizes of the four high-resolution feature maps of different scales are 56×56, 28×28, 14×14 and 7×7, and the size of the high-resolution representation is 1×512;

[0102] Will The corresponding 4 synthetic low-resolution samples Input the initialized low-resolution representation learning network respectively and respectively from Stacked blocks The features are extracted from the output of the output layer and the output layer, and four types of In this embodiment, the sizes of the four low-resolution feature maps of different scales are 56×56, 28×28, 14×14 and 7×7, the size of the low-resolution representation is 1×512, and one of them is The low-resolution feature map of each scale and a low-resolution representation are respectively denoted as and ; Indicates the A first-order output of a stacked block Low-resolution feature maps at different scales;

[0103] Step 4.1.2: Use formula (2) to establish the face recognition loss function based on additive angular margin for low-resolution samples. :

[0104] (2)

[0105] In formula (2), express and categories are The angle between the weights of the corresponding last fully connected layer; express and categories are The angle between the weights of the corresponding last fully connected layer; For all low-resolution representations Calculate the loss function and sum it;

[0106] Step 4.1.3: In order to extract local information in pixel units from high-resolution images to low-resolution images, we minimize the Euclidean distance between pairs of multi-scale feature maps and between representations. Specifically, we use Equation (3) to establish the loss function of pixel value error: :

[0107] (3)

[0108] In formula (3), Indicates traversing all high and low resolution The feature maps of each scale are summed up. Indicates traversing all high and low resolution representations and summing them up. represents the L2 norm;

[0109] Step 4.1.4: As the resolution decreases, the texture and components of the face (eyes, eyebrows, mouth, etc.) become blurred. However, the shape and contour information of the low-resolution image remain unchanged, and its structure is consistent with the high-resolution image. The affinity matrix, a non-local operation, can expand the receptive field and calculate the relationship between pixels, thereby reflecting the global correlation between pixels and the resolution-invariant structural pattern; specifically, any feature map of the high-resolution and four low-resolution feature maps is added. After remodeling operation Then, we get a two-dimensional vector ; Represent the height, width and channel dimensions of the feature map respectively;

[0110] Use formula (4) to build affinity matrix :

[0111] (4)

[0112] In formula (4), is the softmax function, matrix multiplication, is the transpose of the matrix;

[0113] Use formula (5) to establish the loss function of affinity matrix distillation :

[0114] (5)

[0115] In formula (5), represents the L1 norm;

[0116] Step 4.1.5: Maximizing the mutual information between different resolutions can enhance the dependence between the distributions of multi-resolution representations and learn resolution-robust representations. We maximize the mutual information between cross-resolutions to promote the consistency of semantic attributes of high-resolution and low-resolution representations. Since the direct calculation of mutual information is challenging, this system uses the Jensen-Shannon mutual information estimator, which maintains a good balance between computational complexity and performance. Specifically, the trained high-resolution representation Corresponding high-resolution samples In a mini-batch of data, the low-resolution representation of the same category is recorded as , the low-resolution representations of different categories are , and constitute a positive match Represents joint distribution The samples in constitute negative matches Representing marginal distribution Samples in

[0117] Use formula (6) to establish the loss function that maximizes mutual information :

[0118] (6)

[0119] In formula (6), Represents softplus operation; express expectations;

[0120] Step 4.1.6: Use formula (7) to construct the overall objective function of the first stage :

[0121] (7)

[0122] In formula (7), , and are all weight factors; in this embodiment, , and Set to 0.1, 100 and 10 respectively;

[0123] Step 4.1.7: Use SGD to learn the initialized low-resolution representation network Perform the first stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the first stage of training. and its optimal parameters; in this embodiment, the number of training iterations is set to 16;

[0124] Step 4.2, multi-resolution clustering stage, which adjusts the representation of multiple low-resolution images to generate more compact clusters: Step 4.2.1, using the resolution network trained in the first stage The optimal parameters for the momentum network Initialize parameters;

[0125] Using formula (8), a key memory is established to store four low-resolution representations. and a category memory for storing the corresponding labels :

[0126] (8)

[0127] In formula (8), is the length of the repository; Represents key memory Middle Four low-resolution representations, Represents class memory Middle tags;

[0128] Synthetic face database Randomly select samples, where any sample Four low-resolution samples of Input to the low-resolution representation learning network after the first stage training , correspondingly obtain the low-resolution representation learning network after the first stage of training Four low-resolution representations of the output , storing the four low-resolution representations into the key memory In the example, the label representing the corresponding is stored in the category memory middle;

[0129] Step 4.2.2, as Figure 3 As shown, the synthetic face database Medium sample Four low-resolution samples of Input into the low-resolution representation learning network after the first stage training respectively And the initialized momentum network In the first stage, the low-resolution representation learning network is obtained. Four low-resolution representations of the output and Momentum Network Four low-resolution representations of the output , in turn Any representation As an anchor point, let the opposite of the anchor point be the momentum network Characterization of output at different resolutions (In this embodiment, one of the four low-resolution representations is used as an anchor point, and the other three resolutions are its opposites) and key memory Four low-resolution representations of the same category (In this embodiment, the same category represents the same person); wherein, Representing Momentum Network Output division For any representation other than Four low-resolution representations of different categories ,In this embodiment, different categories represent different people;

[0130] Use Equations (9)-(11) to establish a multi-resolution loss function :

[0131] (9)

[0132] (10)

[0133] (11)

[0134] In formula (9)-formula (11), represents a synthetic low-resolution representation, Represents a set of synthetic low-resolution face image representations, including four low-resolution representations; is the temperature parameter; Represents key storage Middle The first Characterization of resolution, Momentum Network Output Resolution characterization, Represents key storage Middle The first Characterization of resolution; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; Indicates anchor point The sum of similarities is calculated with the low-resolution representations of all positive and negative pairs; the multi-resolution contrast loss expands the conventional two views into multiple views, bringing multiple low-resolution representations from the same sample and the same person closer by maximizing intra-class compactness, while maximizing inter-class differences to pull multiple low-resolution representations from different people farther apart;

[0135] Step 4.2.3: Use SGD method to learn the low-resolution representation network after the first stage training Conduct the second stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the second stage of training. ; In this embodiment, the number of training iterations is set to 4;

[0136] Step 4.2.4: Momentum Network Characterization of four output resolutions and the category of the tag Stored in key memory and category memory In order to update the dynamic queue;

[0137] Use formula (1) to update the initialized momentum network Network parameters , get the updated network parameters :

[0138] (12)

[0139] In formula (12), is the low-resolution representation learning network after the second stage training Parameters, is the momentum coefficient; in this embodiment, the momentum network Does not participate in training, but learns the network through low-resolution representation The parameter momentum update of this approach maintains the key memory The representation distribution consistency in

[0140] Step 4.3: Fine-tuning of real low-resolution face images:

[0141] Step 4.3.1. Real low-resolution face image database Any low-resolution sample in Input to the low-resolution representation learning network after second-stage training In the low-resolution representation , and in the new key memory Save in the new category memory Save the category of the tag ,in, Represents key storage Kind A low-resolution representation; Represents class memory Middle labels, since the true resolution of these samples is unknown, different images of the same person are considered to be multi-resolution; As a new anchor point; The opposite of , The negative pair of ;

[0142] Use formula (13) to establish the contrast loss function of real low resolution :

[0143] (13)

[0144] In formula (13), represents the true low-resolution representation, Represents a set of real low-resolution face image representations; Represents key storage Middle A low-resolution representation;

[0145] Step 4.3.2: Use SGD method to learn the low-resolution representation network after the second stage training Perform fine-tuning and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the optimal low-resolution representation learning network. ; In this embodiment, the number of training iterations is set to 4;

[0146] Step 4.4: Use the optimal low-resolution representation to learn the network The low-resolution face image to be tested is processed to obtain a representation of the low-resolution face image, and feature matching is performed with paired face images to achieve face verification, or feature matching is performed between the low-resolution face image to be tested and face images in the database to achieve face recognition.

[0147] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the above-mentioned low-resolution face recognition method, and the processor is configured to execute the program stored in the memory.

[0148] In this embodiment, a computer-readable storage medium stores a computer program, which executes the steps of the above-mentioned low-resolution face recognition method when executed by a processor.

Claims

1. A two-stage multi-scale adaptive low-resolution face recognition method, characterized by Proceed as follows: Step 1: Preprocess the high-resolution face database and the real low-resolution face image database separately: Step 1.1: Preprocessing of high-resolution face database: The MTCNN neural network algorithm is used to analyze the high-resolution face database Perform face detection and correction on a high-resolution face image and crop the image to the input size to obtain a normalized high-resolution face image database. Any high-resolution sample in the normalized high-resolution face image database is denoted as , the category of its label is recorded as ,and , Represents the total number of categories, and is interpolated by bicubic interpolation. Downsample to obtain four low-resolution samples and then upsample back to the original input size to obtain four synthesized low-resolution samples ;Depend on Construct a set of synthetic face images to obtain a synthetic face database ; Step 1.2: Preprocessing of the real low-resolution face image database: The real low-resolution face image database is interpolated by bicubic interpolation. The image size is cropped to the input size, and any low-resolution sample after cropping is recorded as , whose label category is ,Depend on Construct a set of real low-resolution face images, thereby obtaining a real low-resolution face image database ; Step 2: Build a high-resolution representation learning network based on the ResNet-50 network , low-resolution representation learning network and Momentum Network ; The ResNet-50 network includes an input layer, stacked blocks and output layer, where the input layer includes convolutional layer, instance normalization layer, ReLU activation function, and maximum pooling layer. Each stacking block is connected by The model is composed of residual blocks, each of which consists of a convolutional layer, an instance normalization layer, and a ReLU activation function. The output layer includes an average pooling layer and a fully connected layer. Let the high-resolution representation learning network All convolutional layers and fully connected layers in [15] use Gaussian distribution for weight initialization; Step 3: Pre-training high-resolution representation learning network : Step 3.1: The synthesized face database Any high-resolution sample Input to the high-resolution representation learning network Processed in the ; Formula (1) is used to establish the face recognition loss function based on additive angular margin for high-resolution samples. : (1) In formula (1), express and categories are The angle between the weights of the corresponding last fully connected layer; is the angular interval of category addition; express and categories are The angle between the weights of the corresponding last fully connected layer; Represents the synthetic face database Calculate the loss function for all high-resolution representations in and sum them up; Step 3.2: Use SGD method to learn high-resolution representation network Perform pre-training and calculate the loss function Used to update network parameters. When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining a high-resolution representation learning network after training. and its optimal parameters; Leverage trained high-resolution representation learning network The optimal parameters for low-resolution representation learning networks Initialize the network parameters; Step 4: Use a two-stage approach to train the initialized low-resolution representation learning network , and use the low-resolution representation learning network after second-order training Extract representations of low-resolution face images to be tested: Step 4.1: Multi-scale distillation stage: Step 4.1.1: The synthesized face database Any high-resolution sample Input to the trained high-resolution representation learning network and respectively from Stacked blocks The features are extracted from the output of the output layer and the output of the output layer, and the corresponding High-resolution feature maps at different scales and trained high-resolution representations ; Indicates the The output of the stacking block High-resolution feature maps at different scales; Will The corresponding 4 synthetic low-resolution samples Input the initialized low-resolution representation learning network respectively and respectively from Stacked blocks The features are extracted from the output of the output layer and the output layer, and four types of low-resolution feature maps of four scales and four low-resolution representations, and one of them The low-resolution feature map of each scale and a low-resolution representation are respectively denoted as and ; Indicates the A first-order output of a stacked block Low-resolution feature maps at different scales; Step 4.1.2: Use formula (2) to establish the face recognition loss function based on additive angular margin for low-resolution samples. : (2) In formula (2), express and categories are The angle between the weights of the corresponding last fully connected layer; express and categories are The angle between the weights of the corresponding last fully connected layer; Represents the synthetic face database Calculate the loss function for all low-resolution representations in and sum them up; Step 4.1.3: Use formula (3) to establish the loss function of pixel value error : (3) In formula (3), Indicates traversing all high and low resolution The feature maps of each scale are summed up. Indicates traversing all high and low resolution representations and summing them up. represents the L2 norm; Step 4.1.4: Any one of the feature maps at high resolution and four low resolutions After remodeling operation Then, we get a two-dimensional vector ; Represent the height, width and channel dimensions of the feature map respectively; Use formula (4) to build affinity matrix : (4) In formula (4), is the softmax function, matrix multiplication, is the transpose of the matrix; Use formula (5) to establish the loss function of affinity matrix distillation : (5) In formula (5), represents the L1 norm; Step 4.1.5: High-resolution representation after training Corresponding high-resolution samples In a mini-batch of data, the low-resolution representation of the same category is recorded as , the low-resolution representations of different categories are , and constitute a positive match Represents joint distribution The samples in constitute negative matches Representing marginal distribution Samples in Use formula (6) to establish the loss function that maximizes mutual information : (6) In formula (6), Represents softplus operation; express expectations; Step 4.1.6: Use formula (7) to construct the overall objective function of the first stage : (7) In formula (7), , and are all weight factors; Step 4.1.7: Use SGD to learn the initialized low-resolution representation network Perform the first stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the first stage of training. and its optimal parameters; Step 4.2, multi-resolution clustering stage: Step 4.2.1, use the low-resolution representation learning network after the first stage training The optimal parameters for the momentum network Initialize parameters; Using formula (8), a key memory is established to store four low-resolution representations. and a category memory for storing the corresponding labels : (8) In formula (8), is the length of the repository; Represents key memory Middle Four low-resolution representations, Represents class memory Middle tags; The synthesized face database Randomly select samples, where any sample Four low-resolution samples of Input to the low-resolution representation learning network after the first stage training , correspondingly obtain the low-resolution representation learning network after the first stage of training Four low-resolution representations of the output , storing the four low-resolution representations into the key memory In the example, the label representing the corresponding is stored in the category memory middle; Step 4.2.2, as described Samples within samples Four low-resolution samples of Input into the low-resolution representation learning network after the first stage training respectively And the initialized momentum network In the first stage, the low-resolution representation learning network is obtained. Four low-resolution representations of the output and Momentum Network Four low-resolution representations of the output , in turn Any representation As an anchor point, let the opposite of the anchor point be the momentum network Characterization of output at different resolutions and key memory Four low-resolution representations of the same category ;in, Representing Momentum Network Output division For any representation other than Four low-resolution representations of different categories ; Use Equations (9)-(11) to establish a multi-resolution loss function : (9) (10) (11) In formula (9)-formula (11), represents a synthetic low-resolution representation, Represents a set of synthetic low-resolution face image representations, including four low-resolution representations; is the temperature parameter; Represents key storage Middle The first Characterization of resolution, Momentum Network Output Resolution characterization, Represents key storage Middle The first Characterization of resolution; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; express and Are they the same? If so, let Otherwise, let ; Indicates anchor point Calculate the sum of similarities with the low-resolution representations of all positive and negative pairs; Step 4.2.3: Use SGD method to learn the low-resolution representation network after the first stage training Perform the second stage of training and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the low-resolution representation learning network after the second stage of training. ; Step 4.2.4: Momentum Network Characterization of four output resolutions and the category of the tag Stored in key memory and category memory In order to update the dynamic queue; Use formula (12) to update the initialized momentum network Network parameters , get the updated network parameters : (12) In formula (12), is the low-resolution representation learning network after the second stage training Parameters, is the momentum coefficient; Step 4.3: Fine-tuning of real low-resolution face images: Step 4.3.1: The real low-resolution face image database Any low-resolution sample in Input to the low-resolution representation learning network after second-stage training In the low-resolution representation , and in the new key memory Save in the new category memory Save the category of the tag ,in, Represents key storage Middle A low-resolution representation; Represents class memory Middle Tags, which will represent As a new anchor point, The opposite of , The negative pair of ; Use formula (13) to establish the contrast loss function of real low resolution : (13) In formula (13), represents the true low-resolution representation, Represents a set of real low-resolution face image representations; Represents key storage Middle A low-resolution representation; Step 4.3.2: Use SGD method to learn the low-resolution representation network after the second stage training Perform fine-tuning and calculate the loss function When the number of training iterations reaches the set number or the loss error is less than the set threshold, the training stops, thus obtaining the optimal low-resolution representation learning network. ; Step 4.4: Learning the network using the optimal low-resolution representation The low-resolution face image to be tested is processed to obtain a representation of the low-resolution face image, and feature matching is performed with paired face images to achieve face verification, or feature matching is performed between the low-resolution face image to be tested and face images in the database to achieve face recognition.

2. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program for supporting a processor to execute the method according to claim 1 , and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 1 are performed.

Images