Multi-camera heterogeneous face fusion recognition method and device

Abstract

The application discloses a multi-camera heterogeneous face fusion recognition method and device, and relates to the technical field of digital images. The method comprises the following steps: constructing aligned image data sets of low-quality face style graphs and high-quality face content graphs required for training a style transfer model; constructing a multi-camera heterogeneous face fusion network based on progressive style transfer to obtain multiple degraded style transfer models; inputting a high-quality query image into the degraded style transfer model obtained after training to obtain multiple low-quality face images of different modalities and generate a face fusion feature; inputting the obtained multiple low-quality face images of different modalities into the multi-camera heterogeneous face fusion network to generate another face fusion feature; and performing similarity calculation on the two generated face fusion features to find a matching face. The application can be unsupervisedly trained on real data to adapt to real scenes, so that the recognition rate is better.

Description

Technical Field

[0001] The present invention relates to the field of digital image technology, and in particular to a multi-camera heterogeneous face fusion recognition method and device. Background Art

[0002] With the diversification of surveillance camera types, it's common for the same person to be captured at different times and locations by different types of cameras, resulting in heterogeneous facial images (e.g., visible light and infrared faces). Visible light and infrared faces of the same person share identity consistency and modality complementarity. Fusion of their features as a query image to search for query images is a potential approach to improving heterogeneous face recognition. However, the degradation process of low-quality visible light and infrared faces captured in practice is unknown. Improving the recognition accuracy of heterogeneous face fusion without knowing the image degradation process is a pressing issue.

[0003] Existing heterogeneous face recognition methods mainly target high-quality heterogeneous face images. However, in actual monitoring scenarios, due to the interference of multiple degradation factors, the modal changes of heterogeneous faces captured in actual environments are complex and of low quality. It is difficult to find a facial feature that is robust to multiple modalities. In addition, the query image is a modality. It can be seen that existing methods do not consider low-quality heterogeneous face fusion recognition. Summary of the Invention

[0004] In response to the defects in the prior art, the purpose of the present invention is to provide a multi-camera heterogeneous face fusion recognition method and device, which can perform unsupervised training on real data to adapt to real scenes, thereby improving the recognition rate.

[0005] To achieve the above objectives, the present invention provides a multi-camera heterogeneous face fusion recognition method, which specifically includes the following steps:

[0006] Based on the low-quality and high-quality face images in the original surveillance image dataset, an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model is constructed;

[0007] Constructing a multi-camera heterogeneous face fusion network based on progressive style transfer, and using the aligned image dataset to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models;

[0008] Based on the degraded style transfer model obtained after training, a high-quality query image is input to obtain multiple low-quality face images of different modalities, and the low-quality face images in the original surveillance image dataset are passed through the multi-camera heterogeneous face fusion network to generate a face fusion feature;

[0009] The obtained multiple low-quality face images of different modalities are passed through the multi-camera heterogeneous face fusion network to generate another face fusion feature, and the similarity of the two generated face fusion features is calculated to find a matching face.

[0010] Based on the above technical solution, the method constructs an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model based on low-quality face images and high-quality face images in the original surveillance image dataset. The specific steps include:

[0011] The facial images of different modalities and the corresponding high-quality facial images in the original surveillance image dataset are intercepted using the RetinaFace-R50 face detection model;

[0012] The captured low-quality face images are manipulated by implementing the BICUBIC function in the Image.resize() function of the PIL library, and the low-quality face images are upsampled and downsampled to the target resolution to construct a low-quality face image dataset for feeding into the feature extraction sub-network.

[0013] Use the low-quality face image dataset as the query image and select an image from the low-quality face image dataset as the style map for training the style transfer sub-network;

[0014] The original surveillance image dataset is used to detect and crop faces using the RetinaFace-R50 face detection model, and the face images are scaled to the target resolution as the content graph for a downgraded training of the style transfer sub-network.

[0015] On the basis of the above technical solutions,

[0016] The multi-camera heterogeneous face fusion network uses a style transfer sub-network for unsupervised training, gradually migrating the high-definition face image used as the query image to two low-quality domains of different modalities under the original monitoring data, embedding it into a heterogeneous face feature fusion sub-network, and utilizing the modal complementarity between the face images of different modalities after style transfer to improve the quality of face feature fusion. The modalities include visible light and infrared.

[0017] The style transfer sub-network includes three convolutional layers, three deconvolutional layers, and five residual blocks;

[0018] The feature extraction subnetwork consists of a ResNet50 network based on ArcFace;

[0019] The heterogeneous face feature fusion sub-network includes two cascaded feature attention sub-modules;

[0020] The feature attention submodule consists of a fully connected layer and a weight weighting submodule. The feature attention submodule is used to convert the features output by the feature extraction subnetwork into a one-dimensional vector, and then generate normalized weights through the weight weighting submodule to apply to the features extracted by the previous feature extraction subnetwork.

[0021] Based on the above technical solution, the construction process of the heterogeneous face feature fusion sub-network includes:

[0022] Based on the feature extraction sub-network, a ResNet50 network based on ArcFace is used to extract features from the two low-quality input modalities. Specifically:

[0023] fk1 Arc =Arc(xk1),fk2 Arc =Arc(xk2)

[0024] Among them, xk1 and xk2 are two low-quality input modal images, Arc represents the feature extraction process of the feature extraction subnetwork, and fk1 Arc 、fk2 Arc Represents the feature extraction results;

[0025] Use the torch.cat() function in the torch library to concatenate the two extracted features. Specifically:

[0026] fk12 Arc =Concat(fk1 Arc ,fk2 Arc )

[0027] Among them, fk12 Arc Represents the facial features output after splicing, and Concat represents the splicing process;

[0028] The concatenated features are fed into the feature attention module, which performs dot product on fk12. Arc Filter with kernel q to obtain a set of corresponding meaningful feature vectors ek, specifically:

[0029] ek=q·fk12 Arc ;

[0030] The obtained meaning feature vector ek is sent to the weighted submodule to generate linear weights. Specifically, the meaning feature vector ek is passed to the normalized exponential function SoftMax for weighting to generate a normalized weight vector ak. The weight vector ak is equal to fk12 Arc Multiplication generates a fixed-length representation feature vector r, which has the same dimension as the original feature and is expressed as:

[0031] ak=SoftMax(ek)

[0032] r=ak·fk12 Arc ;

[0033] The two feature attention submodules are cascaded, and the first kernel is adapted to the second kernel through the conversion function.

[0034] Based on the above technical solution, the multi-camera heterogeneous face fusion network includes six style transfer sub-networks with non-shared parameters responsible for generating low-quality face images, seven pre-trained feature extraction sub-networks for assisting in generating face similarity loss, and three heterogeneous face feature fusion sub-networks with shared parameters responsible for generating fusion features.

[0035] Based on the above technical solution, the multi-camera heterogeneous face fusion network based on progressive style transfer is constructed. The specific construction steps include:

[0036] Obtain two style transfer sub-networks, use the low-quality visible light face images in the original surveillance image dataset as the style map of one style transfer sub-network, use the low-quality infrared face images in the original surveillance image dataset as the style map of the other style transfer sub-network, use the high-quality face images in the original surveillance image dataset as the content map of the two style transfer sub-networks and perform downgrade training, calculate the corresponding style loss and content loss and update the network parameters, and generate low-quality face images in the visible light modality and infrared modality at the same time. Specifically:

[0037] R cs =S1(HR) IsR , L cs =S2(HR) IsL

[0038] Among them, HR represents the high-quality face image in the input original surveillance image dataset, IsR represents the low-quality infrared style map, IsL represents the low-quality visible light style map, S1 and S2 represent the style transfer model operation corresponding to a degradation stage, and R cs 、L cs Represents the low-quality results of different styles after a style transfer degradation;

[0039] The high-quality face images in the original surveillance image dataset and the generated low-quality face images in the visible light modality and infrared modality are respectively fed into the feature extraction sub-network to extract the corresponding facial features. Specifically:

[0040] fHR Arc =Arc(HR)

[0041] f csArc =Arc(R cs )

[0042] fL cs Arc =Arc(L cs )

[0043] Among them, HR represents the high-quality face images in the original surveillance image dataset, R cs Represents the low-quality face image generated in infrared modality, L cs represents the low-quality face image generated in the visible light modality, Arc represents the feature extraction sub-network process, and fHR Arc 、fR cs Arc 、fL cs Arc Represents the feature extraction results;

[0044] The extracted facial features of the low-quality face image in the visible light modality and the facial features of the low-quality face image in the infrared modality are fed into the heterogeneous face feature fusion sub-network to generate low-quality face fusion features. Specifically:

[0045] FRL cs =F(fR cs Arc ,fL cs Arc )

[0046] Among them, F represents the heterogeneous face feature fusion sub-network, FRL cs Represents the generated low-quality face fusion features;

[0047] The facial features fHR of high-quality face images extracted from the original surveillance image dataset Arc And the generated low-quality face fusion feature FRL cs , perform face feature loss function L similarity Calculation and update network parameters, specifically:

[0048]

[0049] Among them, x and y are the face recognition feature vectors extracted by the feature extraction sub-network, x i 、y i is the size of each value in the face recognition feature vector, i is the element number in the face recognition feature vector, and n is the total number of face recognition feature vectors;

[0050] Set up four style transfer sub-networks, namely A1, A2, A3 and A4, and set the style maps in A1 and A3 to low-quality face images of visible light, and set the style maps in A2 and A4 to low-quality face images of infrared. The generated low-quality face images of visible light and infrared modalities are further degraded for training, and the corresponding style loss and content loss are calculated to update the network parameters to generate four lower-quality face images of different modalities. Specifically:

[0051] LL css =A3(L cs ) IsL ,LR css =A4(L cs ) IsR

[0052] RL css =A1(R cs ) IsL ,RR css =A2(R cs ) IsR

[0053] Among them, R cs 、L cs are two low-quality modality face images after a style transfer degradation, namely the low-quality face image of the infrared modality and the low-quality face image of the visible light modality. A1, A2, A3, and A4 represent the style transfer network operations corresponding to the secondary degradation stage. LL css , LR css RL css , RR css Represents the results of low-quality face images of different styles after secondary style transfer degradation;

[0054] The generated four different modal lower-quality face images are fed into the feature extraction sub-network to extract four different modal lower-quality face features. Specifically:

[0055] fLL css Arc =Arc(LL css ), fLR css Arc =Arc(LR css )

[0056] f css Arc =Arc(RL css ), fRR css Arc =Arc(RR css )

[0057] Among them, fLLcss Arc 、fLR css Arc 、fRL css Arc 、fRR css Arc They are the facial feature extraction results of secondary style transfer degradation;

[0058] The extracted facial features of four different modalities with lower quality are fed into the heterogeneous facial feature fusion sub-network according to the combination of visible light and infrared to generate two low-quality facial fusion features. Specifically:

[0059] FRRLL css =F(fRR css Arc ,fLL css Arc )

[0060] FRLLR css =F(fRL css Arc ,fLR css Arc )

[0061] Among them, FRRLL css 、FRLLR css Represents the generated face fusion features;

[0062] The generated low-quality face fusion feature FRL cs FRRLL and low-quality face fusion features css 、FRLLR css Perform face feature loss function L similarity Calculation and update network parameters, specifically:

[0063]

[0064]

[0065] Among them, y, y1, and y2 are the face recognition feature vectors extracted by the feature extraction sub-network, and y i 、y1 i ,y2 i is the size of each value in the face recognition feature vector, i is the element number in the face recognition feature vector, and n is the total number of face recognition feature vectors.

[0066] On the basis of the above technical solution, the aligned image dataset is used to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models, specifically:

[0067] The aligned image dataset is used as a training set of a content graph of a degraded style transfer model, and the multi-camera heterogeneous face fusion network is trained to obtain six degraded style transfer models.

[0068] Based on the above technical solution, the degraded style transfer model obtained after training is input with a high-quality query image to obtain multiple low-quality face images of different modalities. The specific steps include:

[0069] Obtain the six degraded style transfer models obtained after training, namely W1, W2, W3, W4, W5, and W6;

[0070] Input high-quality face images in the original surveillance image dataset as query images HR for W1 and W2, respectively, and obtain a low-quality face image R in infrared mode. cs and a low-quality face image L in visible light modality cs ;

[0071] The low-quality face image R cs Input into W3 and W4 respectively to obtain a visible light low-quality face image LL css and an infrared low-quality face image LR css ;

[0072] The low-quality face image L in the visible light modality cs Input into W5 and W6 respectively to obtain a visible light low-quality face image RL css and an infrared low-quality face image RR css .

[0073] On the basis of the above technical solution, the obtained multiple low-quality face images of different modalities are passed through the multi-camera heterogeneous face fusion network to generate another face fusion feature, specifically:

[0074] The obtained visible light low-quality face image LL css , infrared low-quality face image LR css , visible light low-quality face image RL css and infrared low-quality face image RR css , after passing through the multi-camera heterogeneous face fusion network, another face fusion feature is generated.

[0075] The present invention provides a multi-camera heterogeneous face fusion recognition device, comprising:

[0076] The first construction module is used to construct an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model based on low-quality face images and high-quality face images in the original surveillance image dataset;

[0077] A second construction module is used to construct a multi-camera heterogeneous face fusion network based on progressive style transfer, and train the multi-camera heterogeneous face fusion network using the aligned image dataset to obtain multiple degraded style transfer models;

[0078] A generation module is used to input a high-quality query image based on the trained degraded style transfer model to obtain multiple low-quality face images of different modalities, and pass the low-quality face images in the original surveillance image dataset through the multi-camera heterogeneous face fusion network to generate a face fusion feature;

[0079] The execution module is used to pass the obtained multiple low-quality face images of different modalities through the multi-camera heterogeneous face fusion network to generate another face fusion feature, and perform similarity calculation on the two generated face fusion features to find a matching face.

[0080] Compared with the existing technology, the advantages of the present invention are: utilizing the modal complementarity between multi-camera faces, the subjective visual effect of the queried image is close to the query image, and the identity feature information of the face input is retained; through the heterogeneous face feature fusion sub-network, the feature information complementarity between different modal cameras is fully utilized to better fuse facial features; through the style transfer sub-network, the present invention can perform unsupervised training on real data to adapt to real scenes, so that the recognition rate is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0082] Figure 1 The present invention provides a flowchart of a multi-camera heterogeneous face fusion recognition method. DETAILED DESCRIPTION

[0083] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0084] See also Figure 1 As shown, an embodiment of the present invention provides a multi-camera heterogeneous face fusion recognition method, that is, a multi-camera heterogeneous face fusion recognition method based on progressive style transfer, which specifically includes the following steps:

[0085] S1: Based on the low-quality and high-quality face images in the original surveillance image dataset, an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model is constructed;

[0086] That is, the face images in the original surveillance image dataset are preprocessed. First, the irrelevant background in the original surveillance face images is removed, and then the cropped faces are scaled to the target resolution size. A preprocessed real low-quality face image is selected as the style map required for style transfer network training. A large-scale high-quality face image dataset is selected as the content map required for style transfer network training. Finally, the high-definition face images paired with the real low-quality face dataset are used as the test dataset.

[0087] In the present invention, based on the low-quality face images and high-quality face images in the original surveillance image dataset, an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model is constructed. The specific steps include:

[0088] S101: Extract facial images of different modalities and corresponding high-quality facial images from the original surveillance image dataset using the RetinaFace-R50 face detection model;

[0089] S102: The captured low-quality facial image is operated by implementing the BICUBIC function (bicubic interpolation) in the Image.resize() function (a function for modifying the image size) of the PIL library (a third-party library of the Python language), and the low-quality facial image is upsampled and downsampled to the target resolution to construct a low-quality facial image dataset for input into the feature extraction subnetwork; that is, the resolution of the facial image is first reduced by downsampling to 11x11 resolution, and then upsampled to 112x112 resolution for input into the feature extraction network.

[0090] S103: Using the low-quality face image dataset as a query image, and selecting an image from the low-quality face image dataset as a style map for training the style transfer sub-network;

[0091] S104: The original surveillance image dataset is subjected to face detection and cropping by the RetinaFace-R50 face detection model, and the face image is scaled to the target resolution as the content graph of the style transfer sub-network for a downgraded training.

[0092] S2: constructing a multi-camera heterogeneous face fusion network based on progressive style transfer, and using the aligned image dataset to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models;

[0093] In this paper, a multi-camera heterogeneous face fusion network uses a style transfer subnetwork for unsupervised training. It gradually transfers high-definition face images, serving as query images, to low-quality domains in two different modalities derived from the original surveillance data. A heterogeneous face feature fusion subnetwork is then embedded. This network leverages the modal complementarity between face images of different modalities after style transfer to improve the quality of face feature fusion. A face feature loss is also proposed to maintain consistency between facial features before and after style transfer, using a loss function. Modalities include visible light and infrared.

[0094] The style transfer subnetwork includes three convolutional layers, three deconvolutional layers, and five residual blocks; the feature extraction subnetwork consists of a ResNet50 network (a convolutional neural network) based on ArcFace (face recognition); the heterogeneous face feature fusion subnetwork includes two cascaded feature attention submodules; the feature attention submodule consists of a fully connected layer and a weight weighting submodule. The feature attention submodule is used to convert the features output by the feature extraction subnetwork into a one-dimensional vector, and then generate normalized weights through the weight weighting submodule to apply to the features extracted by the previous feature extraction subnetwork.

[0095] In the present invention, the construction process of the heterogeneous face feature fusion sub-network includes:

[0096] S201: Based on the feature extraction sub-network, a ResNet50 network based on ArcFace is used to extract features from the two low-quality input modalities. Specifically:

[0097] fk1 Arc =Arc(xk1),fk2 Arc =Arc(xk2)

[0098] Among them, xk1 and xk2 are two low-quality input modal images, Arc represents the feature extraction process of the feature extraction subnetwork, and fk1 Arc 、fk2 Arc Represents the feature extraction results;

[0099] S202: Use the torch.cat() function in the torch library (used to concatenate two tensors according to a specified dimension) to concatenate the two extracted features. Specifically:

[0100] fk12 Arc =Concat(fk1Arc ,fk2 Arc )

[0101] Among them, fk12 Arc Represents the facial features output after splicing, and Concat represents the splicing process;

[0102] S203: The concatenated features are fed into the feature attention module, and the feature attention module performs dot product on fk12. Arc Filter with kernel q to obtain a set of corresponding meaningful feature vectors ek, specifically:

[0103] ek=q·fk12 Arc ;

[0104] S204: Send the obtained meaning feature vector ek to the weight weighting submodule to generate linear weights. Specifically, the meaning feature vector ek is passed to the normalized exponential function SoftMax for weighting to generate a normalized weight vector ak. The weight vector ak is added to fk12 Arc Multiplication generates a fixed-length representation feature vector r, which has the same dimension as the original feature. This process can transform the input face image into a more representative and distinguishable feature vector. Specifically:

[0105] ak=SoftMax(ek)

[0106] r=ak·fk12 Arc ;

[0107] S205: Cascade the two feature attention sub-modules and let the first core adapt to the second core through the conversion function.

[0108] In the present invention, the multi-camera heterogeneous face fusion network includes six style transfer sub-networks with non-shared parameters responsible for generating low-quality face images, seven pre-trained feature extraction sub-networks for assisting in generating face similarity loss, and three heterogeneous face feature fusion sub-networks with shared parameters responsible for generating fusion features.

[0109] In the present invention, a multi-camera heterogeneous face fusion network based on progressive style transfer is constructed. The specific construction steps include:

[0110] S211: Obtain two style transfer sub-networks, use the low-quality visible light face images in the original surveillance image dataset as the style graph of one style transfer sub-network, use the low-quality infrared face images in the original surveillance image dataset as the style graph of the other style transfer sub-network, use the high-quality face images in the original surveillance image dataset as the content graphs of the two style transfer sub-networks and perform downgrade training, calculate the corresponding style loss and content loss, update the network parameters, and generate low-quality face images in the visible light modality and infrared modality at the same time. Specifically:

[0111] R cs =S1(HR) IsR , L cs =S2(HR) IsL

[0112] Among them, HR represents the high-quality face image in the input original surveillance image dataset, IsR represents the low-quality infrared style map, and IsL represents the low-quality visible light style map. In the formula, S1 and S2 represent the style transfer model operation corresponding to a degradation stage, and R cs 、L cs Represents the low-quality results of different styles after a style transfer degradation;

[0113] S212: The high-quality face images in the original surveillance image dataset and the generated low-quality face images in the visible light modality and infrared modality are respectively fed into the feature extraction sub-network to extract the corresponding facial features. Specifically:

[0114] fHR Arc =Arc(HR)

[0115] f cs Arc =Arc(R cs )

[0116] fL cs Arc =Arc(L cs )

[0117] Among them, HR represents the high-quality face images in the original surveillance image dataset, R cs Represents the low-quality face image generated in infrared modality, L cs represents the low-quality face image generated in the visible light modality, Arc represents the feature extraction sub-network process, and fHR Arc 、fR cs Arc 、fL cs Arc Represents the feature extraction results;

[0118] S213: The extracted facial features of the low-quality face image in the visible light modality and the facial features of the low-quality face image in the infrared modality are fed into a heterogeneous face feature fusion sub-network to generate low-quality face fusion features. Specifically:

[0119] FRL cs =F(fR cs Arc ,fL cs Arc )

[0120] Among them, F represents the heterogeneous face feature fusion sub-network, FRL cs Represents the generated low-quality face fusion features;

[0121] S214: extracting facial features fHR of high-quality face images from the original surveillance image dataset Arc And the generated low-quality face fusion feature FRL cs , perform face feature loss function L similarity Calculation and update network parameters, specifically:

[0122]

[0123] Among them, x and y are the face recognition feature vectors extracted by the feature extraction sub-network, x i 、y i is the size of each value in the face recognition feature vector, i is the element number in the face recognition feature vector, and n is the total number of face recognition feature vectors;

[0124] S215: Set up four style transfer sub-networks, namely A1, A2, A3 and A4, and set the style maps in A1 and A3 to low-quality face images of visible light, and set the style maps in A2 and A4 to low-quality face images of infrared. The generated low-quality face images of visible light and infrared modalities are further degraded for training, and the corresponding style loss and content loss are calculated to update the network parameters, thereby generating four lower-quality face images of different modalities. Specifically:

[0125] LL css =A3(L cs ) IsL ,LR css =A4(L cs ) IsR

[0126] RL css =A1(R cs ) IsL ,RR css =A2(R cs ) IsR

[0127] Among them, R cs 、L cs are two low-quality modality face images after a style transfer degradation, namely the low-quality face image of the infrared modality and the low-quality face image of the visible light modality. A1, A2, A3, and A4 represent the style transfer network operations corresponding to the secondary degradation stage. LL css , LR css RL css , RR css Represents the results of low-quality face images of different styles after secondary style transfer degradation;

[0128] S216: The generated four different modal lower-quality facial images are fed into the feature extraction sub-network to extract the four different modal lower-quality facial features. Specifically:

[0129] fLL css Arc =Arc(LL css ), fLR css Arc =Arc(LR css )

[0130] f css Arc =Arc(RL css ), fRR css Arc =Arc(RR css )

[0131] Among them, fLL css Arc 、fLR css Arc 、fRL css Arc 、fRR css Arc They are the facial feature extraction results of secondary style transfer degradation;

[0132] S217: The extracted facial features of four different modalities with lower quality are sent to the heterogeneous facial feature fusion sub-network according to the combination of visible light and infrared to generate two low-quality facial fusion features. Specifically:

[0133] FRRLL css =F(fRR css Arc ,fLL css Arc )

[0134] FRLLR css =F(fRL cssArc ,fLR css Arc )

[0135] Among them, FRRLL css 、FRLLR css Represents the generated face fusion features;

[0136] S218: Fusing the generated low-quality face features FRL cs FRRLL and low-quality face fusion features css 、FRLLR css Perform face feature loss function L similarity Calculation and update network parameters, specifically:

[0137]

[0138]

[0139] Among them, y, y1, and y2 are the face recognition feature vectors extracted by the feature extraction sub-network, and y i 、y1 i ,y2 i is the size of each value in the face recognition feature vector, i is the element number in the face recognition feature vector, and n is the total number of face recognition feature vectors.

[0140] In the present invention, the aligned image dataset is used to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models, specifically:

[0141] The aligned image dataset is used as a training set of a content graph of a degraded style transfer model, and the multi-camera heterogeneous face fusion network is trained to obtain six degraded style transfer models.

[0142] S3: Based on the degraded style transfer model obtained after training, a high-quality query image is input to obtain multiple low-quality face images of different modalities. The low-quality face images in the original surveillance image dataset are passed through the multi-camera heterogeneous face fusion network to generate a face fusion feature;

[0143] In the present invention, based on the trained degraded style transfer model, a high-quality query image is input to obtain multiple low-quality face images of different modalities. The specific steps include:

[0144] S301: Obtain six degraded style transfer models obtained after training, namely W1, W2, W3, W4, W5, and W6;

[0145] S302: Input high-quality face images in the original surveillance image dataset as query images HR for W1 and W2 respectively, and obtain a low-quality face image R in infrared mode respectively. cs and a low-quality face image L in visible light modality cs ;

[0146] S303: Low-quality face image R cs Input into W3 and W4 respectively to obtain a visible light low-quality face image LL css and an infrared low-quality face image LR css ;

[0147] S304: Low-quality face image L in visible light mode cs Input into W5 and W6 respectively to obtain a visible light low-quality face image RL css and an infrared low-quality face image RR css .

[0148] S4: The obtained multiple low-quality face images of different modalities are passed through the multi-camera heterogeneous face fusion network to generate another face fusion feature, and the similarity of the two generated face fusion features is calculated to find a matching face.

[0149] In the present invention, multiple low-quality face images of different modalities are passed through the multi-camera heterogeneous face fusion network to generate another face fusion feature, specifically:

[0150] The obtained visible light low-quality face image LL css , infrared low-quality face image LR css , visible light low-quality face image RL css and infrared low-quality face image RR css , after passing through the multi-camera heterogeneous face fusion network, another face fusion feature is generated.

[0151] The multi-camera heterogeneous face fusion recognition method of the present invention constructs a low / high-quality resolution image pair dataset required for training a neural network; constructs a multi-camera heterogeneous face fusion network based on progressive style transfer; uses the dataset to train and test the multi-camera heterogeneous face fusion network based on progressive style transfer to obtain a trained neural network model; uses the trained network model to test a query image to obtain a face image with a quality close to that of the query image; and calculates the similarity between the face image obtained by the test and a low-quality face image in an original real monitoring dataset, and queries to obtain a recognition rate.

[0152] The present invention utilizes the complementary information between low-quality faces captured by different cameras and uses a style transfer subnetwork to migrate the query image to a low quality close to that of the query image. By degrading the query image to the same low-quality domain as query images of different modalities (visible light and infrared), and then performing feature correlation fusion, it successfully solves the difficult problem of how to improve the accuracy of heterogeneous face fusion recognition when the image degradation process is unknown.

[0153] The present invention provides a multi-camera heterogeneous face fusion recognition device, which includes a first construction module, a second construction module, a generation module and an execution module.

[0154] The first construction module is used to construct an aligned image dataset of low-quality facial style maps and high-quality facial content maps required for training the style transfer model based on the low-quality facial images and high-quality facial images in the original surveillance image dataset; the second construction module is used to construct a multi-camera heterogeneous face fusion network based on progressive style transfer, and use the aligned image dataset to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models; the generation module is used to input the high-quality query image based on the degraded style transfer model obtained after training to obtain multiple low-quality facial images of different modalities, and pass the low-quality facial images in the original surveillance image dataset through the multi-camera heterogeneous face fusion network to generate a facial fusion feature; the execution module is used to pass the obtained multiple low-quality facial images of different modalities through the multi-camera heterogeneous face fusion network to generate another facial fusion feature, and perform similarity calculation on the two generated facial fusion features to find matching faces.

[0155] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

[0156] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A multi-camera heterogeneous face fusion recognition method, characterized in that: The specific steps include: Based on the low-quality and high-quality face images in the original surveillance image dataset, an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model is constructed; Constructing a multi-camera heterogeneous face fusion network based on progressive style transfer, and using the aligned image dataset to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models; Based on the degraded style transfer model obtained after training, a high-quality query image is input to obtain multiple low-quality face images of different modalities, and the low-quality face images in the original surveillance image dataset are passed through the multi-camera heterogeneous face fusion network to generate a face fusion feature; Passing the obtained multiple low-quality face images of different modalities through the multi-camera heterogeneous face fusion network to generate another face fusion feature, and performing similarity calculation on the two generated face fusion features to find a matching face; The aligned image dataset is used to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models, specifically: Using the aligned image dataset as a training set for a content graph of a degraded style transfer model, training the multi-camera heterogeneous face fusion network to obtain six degraded style transfer models; The degraded style transfer model obtained after training is input with a high-quality query image to obtain multiple low-quality face images of different modalities. The specific steps include: The six degraded style transfer models obtained after training are: 、 、 、 、 、 ; right and Input high-quality face images in the original surveillance image dataset as query images , respectively get a low-quality face image in infrared mode and a low-quality face image in the visible light modality ; Low-quality face images Enter respectively and In the example above, we get a low-quality visible light face image. and an infrared low-quality face image ; Low-quality face images in visible light modality Enter respectively and In the example above, we get a low-quality visible light face image. and an infrared low-quality face image .

2. The multi-camera heterogeneous face fusion recognition method according to claim 1, characterized in that: The method of constructing an aligned image dataset of low-quality face style maps and high-quality face content maps required for training a style transfer model based on low-quality face images and high-quality face images in the original surveillance image dataset comprises the following specific steps: The facial images of different modalities and the corresponding high-quality facial images in the original surveillance image dataset are intercepted using the RetinaFace-R50 face detection model; The captured low-quality face images are manipulated by implementing the BICUBIC function in the Image.resize() function of the PIL library, and the low-quality face images are upsampled and downsampled to the target resolution to construct a low-quality face image dataset for feeding into the feature extraction sub-network. Use the low-quality face image dataset as the query image and select an image from the low-quality face image dataset as the style map for training the style transfer sub-network; The original surveillance image dataset is used to detect and crop faces using the RetinaFace-R50 face detection model, and the face images are scaled to the target resolution as the content graph for a downgraded training of the style transfer sub-network.

3. The multi-camera heterogeneous face fusion recognition method according to claim 2, characterized in that: The multi-camera heterogeneous face fusion network uses a style transfer sub-network for unsupervised training, gradually migrating the high-definition face image used as the query image to two low-quality domains of different modalities under the original monitoring data, embedding it into a heterogeneous face feature fusion sub-network, and utilizing the modal complementarity between the face images of different modalities after style transfer to improve the quality of face feature fusion. The modalities include visible light and infrared. The style transfer sub-network includes three convolutional layers, three deconvolutional layers, and five residual blocks; The feature extraction subnetwork consists of a ResNet50 network based on ArcFace; The heterogeneous face feature fusion sub-network includes two cascaded feature attention sub-modules; The feature attention submodule consists of a fully connected layer and a weight weighting submodule. The feature attention submodule is used to convert the features output by the feature extraction subnetwork into a one-dimensional vector, and then generate normalized weights through the weight weighting submodule to apply to the features extracted by the previous feature extraction subnetwork.

4. The multi-camera heterogeneous face fusion recognition method according to claim 3, characterized in that: The construction process of the heterogeneous face feature fusion sub-network includes: Based on the feature extraction sub-network, a ResNet50 network based on ArcFace is used to extract features from the two low-quality input modalities. Specifically: in, 、 are two low-quality modality images input, represents the feature extraction process of the feature extraction subnetwork, 、 Represents the feature extraction results; Use the torch.cat() function in the torch library to concatenate the two extracted features. Specifically: in, Represents the facial features output after splicing, Indicates the splicing process; The concatenated features are fed into the feature attention module, which performs dot product on the and nuclear Filter and obtain a set of corresponding meaning feature vectors , specifically: ； The obtained meaningful feature vector Send it to the weighted submodule to generate linear weights. Specifically, the meaning feature vector Passed to the normalized exponential function Perform weighting to generate a normalized weight vector , weight vector and Multiply to generate a fixed-length representation feature vector , represents the feature vector The same as the original feature dimension, expressed as: ； The two feature attention submodules are cascaded, and the first kernel is adapted to the second kernel through the conversion function.

5. The multi-camera heterogeneous face fusion recognition method according to claim 4, characterized in that: The multi-camera heterogeneous face fusion network includes six style transfer sub-networks with non-shared parameters responsible for generating low-quality face images, seven pre-trained feature extraction sub-networks for assisting in generating face similarity loss, and three heterogeneous face feature fusion sub-networks with shared parameters responsible for generating fusion features.

6. The multi-camera heterogeneous face fusion recognition method according to claim 5, characterized in that: The multi-camera heterogeneous face fusion network based on progressive style transfer is constructed in the following specific steps: Obtain two style transfer sub-networks, use the low-quality visible light face images in the original surveillance image dataset as the style map of one style transfer sub-network, use the low-quality infrared face images in the original surveillance image dataset as the style map of the other style transfer sub-network, use the high-quality face images in the original surveillance image dataset as the content map of the two style transfer sub-networks and perform downgrade training, calculate the corresponding style loss and content loss and update the network parameters, and generate low-quality face images in the visible light modality and infrared modality at the same time. Specifically: in, represents the high-quality face image in the input original surveillance image dataset, Represents a low-quality infrared style image, represents a low-quality visible light style map, 、 Represents the style transfer model operation corresponding to a degradation stage, 、 Represents the low-quality results of different styles after a style transfer degradation; The high-quality face images in the original surveillance image dataset and the generated low-quality face images in the visible light modality and infrared modality are respectively fed into the feature extraction sub-network to extract the corresponding facial features. Specifically: in, represents the high-quality face images in the original surveillance image dataset, represents the generated low-quality face image in infrared modality, represents the generated low-quality face image in the visible light modality, Represents the feature extraction sub-network process, 、 、 Represents the feature extraction results; The extracted facial features of the low-quality face image in the visible light modality and the facial features of the low-quality face image in the infrared modality are fed into the heterogeneous face feature fusion sub-network to generate low-quality face fusion features. Specifically: in, represents the heterogeneous face feature fusion sub-network, Represents the generated low-quality face fusion features; The facial features of high-quality face images extracted from the original surveillance image dataset and generated low-quality face fusion features , perform face feature loss function Calculation and update of network parameters, specifically: in, 、 The face recognition feature vector extracted by the feature extraction sub-network, 、 is the size of each value in the face recognition feature vector, is the element number in the face recognition feature vector, is the total number of face recognition feature vectors; Set up four style transfer sub-networks, namely 、 、 and , and and The style map in is set to a low-quality face image of visible light. and The style map in the image is set to a low-quality infrared face image. The generated low-quality face images in the visible light and infrared modes are further degraded for training. The corresponding style loss and content loss are calculated to update the network parameters, and four lower-quality face images in different modes are generated. Specifically: in, 、 These are two low-quality face images of different modalities after a style transfer degradation, namely a low-quality face image of the infrared modality and a low-quality face image of the visible light modality. 、 、 、 Represents the style conversion network operation corresponding to the secondary degradation stage, 、 、 、 Represents the results of low-quality face images of different styles after secondary style transfer degradation; The generated four different modal lower-quality face images are fed into the feature extraction sub-network to extract four different modal lower-quality face features. Specifically: in, 、 、 、 They are the facial feature extraction results of secondary style transfer degradation; The extracted facial features of four different modalities with lower quality are fed into the heterogeneous facial feature fusion sub-network according to the combination of visible light and infrared to generate two low-quality facial fusion features. Specifically: in, 、 Represents the generated face fusion features; The generated low-quality face fusion features Separately and low-quality face fusion features 、 Perform face feature loss function Calculation and update of network parameters, specifically: in, 、 、 The face recognition feature vector extracted by the feature extraction sub-network, 、 、 is the size of each value in the face recognition feature vector, is the element number in the face recognition feature vector, is the total number of face recognition feature vectors.

7. The multi-camera heterogeneous face fusion recognition method according to claim 6, characterized in that: The obtained multiple low-quality face images of different modalities are passed through the multi-camera heterogeneous face fusion network to generate another face fusion feature, specifically: The obtained visible light low-quality face image , infrared low-quality face images , low-quality visible light face images and infrared low-quality face images , after passing through the multi-camera heterogeneous face fusion network, another face fusion feature is generated.

8. A multi-camera heterogeneous face fusion recognition device, characterized in that: include: The first construction module is used to construct an aligned image dataset of low-quality face style maps and high-quality face content maps required for training the style transfer model based on low-quality face images and high-quality face images in the original surveillance image dataset; A second construction module is used to construct a multi-camera heterogeneous face fusion network based on progressive style transfer, and train the multi-camera heterogeneous face fusion network using the aligned image dataset to obtain multiple degraded style transfer models; A generation module is used to input a high-quality query image based on the trained degraded style transfer model to obtain multiple low-quality face images of different modalities, and pass the low-quality face images in the original surveillance image dataset through the multi-camera heterogeneous face fusion network to generate a face fusion feature; an execution module, configured to pass the obtained multiple low-quality facial images of different modalities through the multi-camera heterogeneous face fusion network to generate another facial fusion feature, and perform similarity calculation on the two generated facial fusion features to find a matching face; The aligned image dataset is used to train the multi-camera heterogeneous face fusion network to obtain multiple degraded style transfer models, specifically: Using the aligned image dataset as a training set for a content graph of a degraded style transfer model, training the multi-camera heterogeneous face fusion network to obtain six degraded style transfer models; The degraded style transfer model obtained after training is input with a high-quality query image to obtain multiple low-quality face images of different modalities. The specific steps include: The six degraded style transfer models obtained after training are: 、 、 、 、 、 ; right and Input high-quality face images in the original surveillance image dataset as query images , respectively get a low-quality face image in infrared mode and a low-quality face image in the visible light modality ; Low-quality face images Enter respectively and In the example above, we get a low-quality visible light face image. and an infrared low-quality face image ; Low-quality face images in visible light modality Enter respectively and In the example above, we get a low-quality visible light face image. and an infrared low-quality face image .

Images