A multi-frame depression angle face recognition method and device based on fusion reordering

Abstract

The application relates to a multi-frame low-angle face recognition method and device based on fusion reordering, and relates to the technical field of digital images. The method comprises the following steps: based on low-angle face feature information of multi-frame low-angle face images of the same person, the similarity of high-definition front face feature information of different persons in high-definition front face images in a preset gallery set is calculated, and an initial similarity list corresponding to each low-angle face image is obtained; based on a preset weight prediction network, a fusion weight is predicted; based on the fusion weight, the initial similarity lists corresponding to different low-angle face images are weighted and fused, and a fused similarity list is obtained; and based on the fused similarity list, face recognition is performed in combination with the high-definition front face images. According to the application, the similarity lists are weighted and fused based on the multi-frame low-angle face images and the weight prediction network, so that a more accurate similarity list is obtained, and the multi-frame low-angle face image recognition accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital images, in particular to a multi-frame oblique angle face recognition method and device based on fusion reordering. BACKGROUND

[0002] With the popularization of monitoring camera equipment and the rapid development of face recognition technology, face recognition is increasingly widely used in the security field. However, due to the high position of the camera, oblique angle face recognition has become a common problem in monitoring scenarios. In general, the oblique angle face in the distance has a complete facial structure, but lacks facial texture information, and the oblique angle face in the near distance has high clarity, but lacks a complete facial contour. Due to the incompatibility of structure integrity and resolution, the performance of single-frame oblique angle face recognition is not ideal. Therefore, it is necessary to utilize the complementarity between multi-frame oblique angle faces to improve the recognition performance.

[0003] Most of the existing multi-frame face recognition methods use feature-level fusion strategies to utilize the complementarity between different frames. These multi-frame face recognition methods all need to detect and align the face. However, for oblique angle faces with large changes in angle and resolution, the detected key points are incomplete and inaccurate. Therefore, it is difficult to align the oblique angle face, which leads to the fact that the existing multi-frame face recognition methods are not suitable for oblique angle face recognition.

[0004] Therefore, in order to meet the current face recognition needs, the present application provides a multi-frame oblique angle face recognition method and device based on fusion reordering. SUMMARY

[0005] The present application provides a multi-frame oblique angle face recognition method and device based on fusion reordering, which weights and fuses the similarity list based on multi-frame oblique angle face images and a weight prediction network, so as to obtain a more accurate similarity list, thereby improving the recognition accuracy of multi-frame oblique angle face images.

[0006] To achieve the above-mentioned purpose, the present application provides the following solutions.

[0007] In a first aspect, the present application provides a multi-frame oblique angle face recognition method based on fusion reordering, which comprises the following steps:

[0008] Based on the oblique angle face feature information of the multi-frame oblique angle face images of the same person, the similarity between the high-definition frontal face feature information of different identity persons in the preset gallery set and the high-definition frontal face feature information of the multi-frame oblique angle face images is calculated, and an initial similarity list corresponding to each of the oblique angle face images is obtained;

[0009] Based on the preset weight prediction network, a fusion weight is predicted and obtained;

[0010] Based on the fusion weight, the initial similarity lists corresponding to different oblique face image pairs are weighted and fused to obtain a fused similarity list;

[0011] Based on the fused similarity list, face recognition is performed in combination with the high-definition frontal face image.

[0012] Further, the method further comprises the following steps:

[0013] Based on the set fusion weight optimization target, a corresponding fusion re-optimization interval is obtained;

[0014] Based on the preset loss function and the fusion re-optimization interval, the loss value of the fusion weight is calculated;

[0015] The loss value of the fusion weight is back propagated to the weight prediction network for fusion weight optimization by the weight prediction network.

[0016] Specifically, the loss function is Loss=max(0,(w i -Jump n (x))×(w i -Jump n (y)));wherein,

[0017] w i is the fusion weight, Jump n (x) and Jump n (y) are two interval end values corresponding to the fusion weight optimization interval, and Loss is the loss value of the fusion weight.

[0018] Further, based on the oblique face feature information of the multiple oblique face images of the same person, the similarity between the high-definition frontal face feature information of different persons in the high-definition frontal face images in the preset gallery set is calculated to obtain the initial similarity list corresponding to each oblique face image, including the following steps:

[0019] The multiple far and near oblique face images {p1, p2,..., p M} of the same person are extracted to obtain the oblique face feature information {f p1 ,f p2 ,...,f pM};

[0020] The high-definition frontal face images {g1, g2,..., g N} in the gallery set are extracted to obtain the high-definition frontal face feature information {f g1 ,f g2 ,...,f gN};

[0021] Based on the aforementioned top-angle face feature information and the high-resolution frontal face feature information, an initial similarity list {S1,S2,...,S} corresponding to each top-angle face image is obtained using cosine similarity. M};in,

[0022] M is the number of multiple frames of facial images of the same person from near and far angles;

[0023] N is the number of people of different identities gathered in the gallery;

[0024] S i (i = 1, 2, ..., M) represents the initial similarity list obtained from the i-th top-angle face, S i The j-th element S i,j (j = 1, 2, ..., N) represents f pi with f gj The similarity between them.

[0025] Furthermore, based on a pre-defined weight prediction network, the process of predicting and obtaining the fusion weights includes the following steps:

[0026] The downward-angle facial feature information is input into a dynamic convolutional kernel generation network to obtain a first output parameter corresponding to the downward-angle facial feature information;

[0027] Based on the first output parameters, the convolution kernel is obtained;

[0028] The two initial similarity lists that need to be fused are concatenated and used as the first input information of the weight prediction network.

[0029] Based on the first input information and the convolution kernel, shallow features adjusted by the downward angle of the face are obtained;

[0030] The fusion weights are predicted by iterating based on the shallow features.

[0031] Furthermore, based on the fusion weights and the similarity list calculation formula, the initial similarity lists corresponding to different top-angle face images are weighted and fused to obtain a fused similarity list.

[0032] The formula for calculating the similarity list is:

[0033] in,

[0034] w i-1 The fusion weights have values ​​ranging from 0 to 1.

[0035] This is the list of similarities after fusion.

[0036] Secondly, this application provides a multi-frame top-angle face recognition device based on fusion reordering, the device comprising:

[0037] The similarity list acquisition module is used to calculate the similarity between the high-definition frontal face feature information of different people in the high-definition frontal face images in the preset gallery set and the high-definition frontal face feature information of the same person based on the high-definition frontal face feature information of multiple frames of the high-definition face images of the same person, so as to obtain the initial similarity list corresponding to each high-definition face image.

[0038] The fusion weight acquisition module is used to predict and obtain fusion weights based on a preset weight prediction network.

[0039] The weighted fusion module is used to perform weighted fusion on the initial similarity lists corresponding to different top-angle face images based on the fusion weights, so as to obtain a fused similarity list.

[0040] A face recognition module is used to perform face recognition based on the fused similarity list and the high-definition frontal face image.

[0041] Furthermore, the device also includes a weighted ranking loss module;

[0042] The weighted ranking loss module is used to obtain the corresponding fusion re-optimization interval based on the set fusion weight optimization objective;

[0043] The weighted ranking loss module is also used to calculate the loss value of the fusion weight based on a preset loss function and the fusion re-optimization interval;

[0044] The weighted ranking loss module is also used to backpropagate the loss value of the fusion weights to the weight prediction network so that the weight prediction network can optimize the fusion weights.

[0045] Furthermore, the loss function is:

[0046] Loss = max(0,(w i -Jump n (x))×(w i -Jump n (y)));wherein,

[0047] w i For the fusion weights, Jump n (x) and Jump n (y) represents the two interval endpoints corresponding to the fusion weight optimization interval, and Loss represents the loss value of the fusion weight.

[0048] Further, the fusion weight acquisition module is further configured to input the depression angle face feature information into a dynamic convolution kernel generation network to obtain a first output parameter corresponding to the depression angle face feature information.

[0049] The fusion weight acquisition module is further configured to obtain a convolution kernel based on the first output parameter.

[0050] The fusion weight acquisition module is further configured to splice two initial similarity lists that need to be fused as first input information of a weight prediction network.

[0051] The fusion weight acquisition module is further configured to obtain a shallow feature adjusted by a depression angle face based on the first input information and the convolution kernel.

[0052] The fusion weight acquisition module is further configured to predict the fusion weight based on iteration of the shallow feature.

[0053] The technical scheme provided in the application has the following beneficial effects:

[0054] The application weights and fuses the similarity lists based on the multi-frame depression angle face images and the weight prediction network, so that a more accurate similarity list is obtained, thereby improving the multi-frame depression angle face image recognition accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0055] In order to more clearly illustrate the technical scheme in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0056] Figure 1 The step flow chart of the multi-frame depression angle face recognition method based on fusion reordering provided in the embodiments of the application;

[0057] Figure 2 The overall flow chart of the multi-frame depression angle face recognition method based on fusion reordering provided in the embodiments of the application;

[0058] Figure 3 The solution schematic diagram of the best weight interval in the multi-frame depression angle face recognition method based on fusion reordering provided in the embodiments of the application;

[0059] Figure 4 The structural block diagram of the multi-frame depression angle face recognition device based on fusion reordering provided in the embodiments of the application. DETAILED DESCRIPTION

[0060] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0061] The embodiments of the present application are further described in detail below with reference to the drawings.

[0062] The embodiments of the present application provide a multi-frame oblique face recognition method and device based on fusion reordering, which weights and fuses a similarity list based on a multi-frame oblique face image and a weight prediction network, so as to obtain a more accurate similarity list, thereby improving the multi-frame oblique face image recognition accuracy.

[0063] In order to achieve the above technical effects, the general idea of the present application is as follows:

[0064] A multi-frame oblique face recognition method based on fusion reordering, the method comprising the following steps:

[0065] S1, based on the oblique face feature information of the multi-frame oblique face images of the same person, calculating the similarity between the high-definition frontal face feature information of different identity persons in the high-definition frontal face images in the preset gallery set, obtaining the initial similarity list corresponding to each oblique face image;

[0066] S2, based on the preset weight prediction network, predicting and obtaining a fusion weight;

[0067] S3, based on the fusion weight, weighting and fusing the initial similarity list corresponding to different oblique face images, obtaining a fused similarity list;

[0068] S4, based on the fused similarity list, combining the high-definition frontal face image, performing face recognition.

[0069] The embodiments of the present application are further described in detail below with reference to the drawings.

[0070] Referring to Figures 1 to 3 The embodiments of the present application provide a multi-frame oblique face recognition method based on fusion reordering, the method comprising the following steps:

[0071] S1, based on the oblique face feature information of the multi-frame oblique face images of the same person, calculating the similarity between the high-definition frontal face feature information of different identity persons in the high-definition frontal face images in the preset gallery set, obtaining the initial similarity list corresponding to each oblique face image;

[0072] S2, predicting a fusion weight based on a preset weight prediction network;

[0073] S3, weighting and fusing the initial similarity lists corresponding to different low-angle face images based on the fusion weight to obtain a fused similarity list;

[0074] S4, performing face recognition based on the fused similarity list and a high-definition frontal face image.

[0075] In the embodiments of the present application, the similarity list is weighted and fused based on the multi-frame low-angle face image and the weight prediction network, so as to obtain a more accurate similarity list, thereby improving the recognition accuracy of the multi-frame low-angle face image.

[0076] Further, the method further comprises the following steps:

[0077] The face images in the low-angle face data set are cropped to obtain a plurality of low-angle face images of a fixed size.

[0078] Further, the method further comprises the following steps:

[0079] Based on the set fusion weight optimization target, a corresponding fusion weight optimization interval is obtained.

[0080] Based on the preset loss function and the fusion weight optimization interval, a loss value of the fusion weight is calculated.

[0081] The loss value of the fusion weight is back-propagated to the weight prediction network for fusion weight optimization by the weight prediction network.

[0082] Specifically, the loss function is Loss=max(0,(w i -Jump n (x))×(w i -Jump n (y)));wherein,

[0083] w i is the fusion weight, Jump n (x) and Jump n (y) are two interval end values corresponding to the fusion weight optimization interval, and Loss is the loss value of the fusion weight.

[0084] Further, based on the low-angle face feature information of the multi-frame low-angle face image of the same person, the similarity between the high-definition frontal face feature information of different persons in the preset gallery set and the high-definition frontal face feature information is calculated to obtain an initial similarity list corresponding to each low-angle face image, comprising the following steps:

[0085] Multiple frames of facial images of the same person from near and far angles (p1, p2, ..., p) are used to represent the face of the same person at different distances and from different angles. M Extract the top-down facial feature information {f} p1 ,f p2 ,...,f pM};

[0086] The high-resolution frontal images {g1, g2, ..., g} in the gallery collection are selected from the high-resolution frontal images. N Extract high-resolution frontal face feature information respectively {f g1 ,f g2 ,...,f gN};

[0087] Based on the aforementioned top-angle face feature information and the high-resolution frontal face feature information, an initial similarity list {S1,S2,...,S} corresponding to each top-angle face image is obtained using cosine similarity. M};in,

[0088] M is the number of multiple frames of facial images of the same person from near and far angles;

[0089] N is the number of people of different identities gathered in the gallery;

[0090] S i (i = 1, 2, ..., M) represents the initial similarity list obtained from the i-th top-angle face, S i The j-th element S i,j (j = 1, 2, ..., N) represents f pi with f gj The similarity between them.

[0091] Furthermore, based on a pre-defined weight prediction network, the process of predicting and obtaining the fusion weights includes the following steps:

[0092] The downward-angle facial feature information is input into a dynamic convolutional kernel generation network to obtain a first output parameter corresponding to the downward-angle facial feature information;

[0093] Based on the first output parameters, the convolution kernel is obtained;

[0094] The two initial similarity lists that need to be fused are concatenated and used as the first input information of the weight prediction network.

[0095] Based on the first input information and the convolution kernel, shallow features adjusted by the downward angle of the face are obtained;

[0096] The fusion weights are predicted by iterating based on the shallow features.

[0097] Further, based on the fusion weight and a similarity list calculation formula, the initial similarity lists corresponding to different the depression angle face images are weighted and fused to obtain a fused similarity list.

[0098] The similarity list calculation formula is:

[0099] wherein,

[0100] w i-1 is the fusion weight, and the value range is between 0 and 1.

[0101] Based on the above technical solution, a practical implementation is given, and the specific case is as follows:

[0102] Step 1, data set preprocessing, including detecting and cutting the face images in the depression angle face data set and obtaining fixed size depression angle face images.

[0103] Step 2, extracting the features of the depression angle face, and obtaining the initial similarity list according to the similarity between the features.

[0104] Step 3, constructing a weight prediction network, which mainly consists of a dynamic convolution kernel generation module and a fusion weight generation module. The dynamic convolution kernel generation module is to make the predicted weight adjustable to the depression angle face of different angles, and the fusion weight module is to obtain the fusion weight between the input similarity lists.

[0105] Step 4, according to the predicted fusion weight, the two input similarity lists are weighted and fused to obtain the fused similarity list.

[0106] Step 5, in order to train the weight prediction network model, a weighted ranking loss is designed to obtain the best fusion weight interval.

[0107] Specifically, the above step 1 includes the following steps:

[0108] Step 1.1: use a face detection network to detect the face region and cut the face image.

[0109] Step 1.2: upsample the cut depression angle face image to a specified size;

[0110] In the specific implementation process, the retinaface face detection algorithm is used to detect the position of the face region in the image. First, the left upper corner coordinate (x left-top ,y left-top ) and the right lower corner coordinate (x right-bottom ,y right-bottomBased on the coordinates of these two points, a rectangular area of ​​the face is cropped out. Then, the cropped face image is adjusted to a size of 112×112.

[0111] Specifically, step 2 above includes the following steps:

[0112] Step 2.1: Extract multiple frames of the same person's face from near and far angles, including the top and bottom views {p1, p2, ..., p...}. M Extract features {f} respectively p1 ,f p2 ,...,f pM}, where M is the number of faces of the same person at different downward angles.

[0113] Step 2.2: Extract the high-resolution frontal face images {g1, g2, ..., g from the gallery collection. N Extract features {f} respectively g1 ,f g2 ,...,f gN}, where N is the number of people with different identities in the gallery set.

[0114] Step 2.3: Obtain the initial similarity list {S1, S2, ..., S} using cosine similarity. M};

[0115] Among them, S i (i = 1, 2, ..., M) represents the similarity list obtained from the i-th top-angle face, S i The j-th element S i,j (j = 1, 2, ..., N) represents f pi with f gj The similarity between them;

[0116] In the specific implementation process, the face recognition feature extraction network used is an existing face recognition feature extraction network. In this embodiment of the application, the ArcFace network is used, the skeleton network is IResNet-50, and the extracted feature dimension is 512.

[0117] The dataset used is the Tilt-angle Face Dataset (TFD). There are 5 images of the same person's face viewed from above, with angles ranging from 15°, 30°, 45°, 60°, and 75°. The gallery set contains 1399 images. Cosine similarity is used for similarity, and the calculated similarity list is a vector of length 1399.

[0118] Specifically, step 3 above includes the following steps:

[0119] Step 3.1: The feature f obtained in step 2.1 piThe input is input into the dynamic convolution kernel generation network to obtain a face p adjusted by the depression angle i A related tensor out i (i = 1, 2,... M).

[0120] Step 3.2: The out i obtained in step 3.1 is adjusted to the size and shape required by the convolution kernel, thereby obtaining the dynamic convolution kernel kernel i (i = 1, 2,... M).

[0121] Step 3.3: The two similarity lists and S i that need to be fused are spliced together as the input in i of the weight prediction network. (i = 1, 2,... M).

[0122] Step 3.4: The in i obtained in step 3.3 is sent to the first block C1 of the weight prediction network together with the convolution kernel kernel i obtained in step 3.2,

[0123] wherein in i is the input of C1, kernel i is the convolution kernel of C1, thereby obtaining the shallow feature F1 adjusted by the face of the depression angle.

[0124] Step 3.5: The F1 obtained in step 3.4 is sent to the next block C1 to obtain F2, and the iteration is continued in this way, and the output F5 of the last block C5 is taken as the predicted weight w i-1 .

[0125] In the specific implementation process, the dynamic convolution kernel generation network is the existing Long Short Term Memory (LSTM) structure, and in the present example, an LSTM network with an input size of 512, a hidden layer size and an output size of 576, and 2 layers is adopted. The size of the adjusted convolution kernel kernel i is 64*1*3*3, and the size of the spliced input in i is 2*1399.

[0126] The skeleton network adopted by the weight prediction network is the existing resnet-18, which is composed of 5 blocks, denoted as C1, C2,..., C5.

[0127] The difference is that the convolution kernel in the first block of resnet-18 is replaced by dynamic convolution, and a fully connected layer is connected after the last block, so as to convert the extracted features into fusion weights.

[0128] Specifically, the step 4 comprises the following steps:

[0129] The formula for calculating the similarity list after fusion is:

[0130]

[0131] wherein w i-1 is the fusion weight obtained in step 3.5, and the value range is between 0 and 1;

[0132] In the specific implementation process, the present example performs two-by-two fusion for four iterations. The first fusion is to fuse two initial similarity lists, and each subsequent fusion is to fuse the similarity list obtained in the last fusion with the next initial similarity list. The specific process is shown in the Figure 2 of the accompanying drawings.

[0133] Specifically, the step 5 comprises the following steps:

[0134] Step 5.1: Determine the optimization target. In order to make the recognition accuracy high, the similarity value of the correct face should be ranked as far as possible in the similarity list after fusion, that is:

[0135]

[0136] wherein n is the subscript of the correct face.

[0137] Step 5.2: Change the optimization target equivalently as:

[0138] wherein Indicator n (j,w i ) represents an indicator function, which means whether the ranking of g n in the similarity list after fusion is in front of g j . The mathematical definition is as follows:

[0139]

[0140] Step 5.3: Determine the new optimization target. In order to make the ranking of g n more in front, Indicator n (j,w i ) should be as close to 0 as possible.

[0141] Step 5.4: Determine Indicatorn (j,w i When w is 0 i The conditions are met, according to step 3.5. As can be seen from the definition, when Indicator n (j,w i When ) is 0, w i The following conditions must be met:

[0142]

[0143] Among them, Jump n (j) is an indicator n (j,w i The unique jump discontinuity of ) is defined as follows:

[0144]

[0145] Step 5.5: Determine the optimal interval. For each j (j = 1, 2, ... N), a corresponding jump discontinuity point can be obtained. n (j), thus obtaining N jump discontinuities {Jump n (1), Jump n (2),...Jump n (N)}, these N jump discontinuities divide the number line into N+1 intervals, then within each interval All of them are fixed values, so by traversing these N+1 intervals, we can obtain the value that makes... The smallest interval (Jump) n (x), Jump n (y)), the specific process is shown in the attached diagram of the instruction manual. Figure 3 As shown.

[0146] Step 5.6: Define the loss function as follows:

[0147] Loss = max(0,(w i -Jump n (x))×(w i -Jump n (y)));

[0148] In the specific implementation process, since the fusion weight should be a real value between 0 and 1, this example considers those intervals between 0 and 1.

[0149] Subsequently, the loss value of the fusion weight is back propagated to the weight prediction network, and the weight prediction network can learn the loss value corresponding to the fusion weight predicted by the weight prediction network, so that the weight prediction network can be referenced during subsequent prediction of the fusion weight, and the prediction operation is optimized to minimize the loss value for the weight prediction network to optimize the fusion weight.

[0150] It should be noted that the step numbers of the steps in the embodiments of the present application do not limit the order of the operations in the technical solutions of the present application.

[0151] Referring to Figure 4 Based on the same inventive concept as the method embodiment, the embodiments of the present application provide a multi-frame oblique angle face recognition device based on fusion reordering, which comprises:

[0152] A similarity list acquisition module is configured to calculate the similarity between the high-definition frontal face feature information of different identities in the high-definition frontal face images in the preset gallery set and the oblique angle face feature information of the multi-frame oblique angle face images of the same person, and obtain an initial similarity list corresponding to each oblique angle face image.

[0153] A fusion weight acquisition module is configured to predict a fusion weight based on a preset weight prediction network.

[0154] A weighted fusion module is configured to perform weighted fusion on the initial similarity lists corresponding to different oblique angle face images based on the fusion weight, and obtain a fused similarity list.

[0155] A face recognition module is configured to perform face recognition based on the fused similarity list and the high-definition frontal face images.

[0156] In the embodiments of the present application, the similarity list is weighted and fused based on the multi-frame oblique angle face images and the weight prediction network, so that a more accurate similarity list is obtained to improve the recognition accuracy of the multi-frame oblique angle face images.

[0157] Further, the device further comprises:

[0158] An image cropping module is configured to crop the face images in the oblique angle face data set to obtain a plurality of oblique angle face images of a fixed size.

[0159] Further, the device further comprises a weight weighted ranking loss module.

[0160] The weighted ranking loss module is configured to obtain a corresponding fusion re-optimization interval based on a set fusion weight optimization target.

[0161] The weighted ranking loss module is also used to calculate the loss value of the fusion weight based on a preset loss function and the fusion re-optimization interval;

[0162] The weighted ranking loss module is also used to backpropagate the loss value of the fusion weights to the weight prediction network so that the weight prediction network can optimize the fusion weights.

[0163] Furthermore, the loss function is:

[0164] Loss = max(0,(w i -Jump n (x))×(w i -Jump n (y)));wherein,

[0165] w i For the fusion weights, Jump n (x) and Jump n (y) represents the two interval endpoints corresponding to the fusion weight optimization interval, and Loss represents the loss value of the fusion weight.

[0166] Furthermore, the similarity list acquisition module is also used to obtain multiple frames of near and far angle face images of the same person {p1,p2,...,p M Extract the top-down facial feature information {f} p1 ,f p2 ,...,f pM};

[0167] The similarity list acquisition module is also used to select high-definition frontal face images {g1,g2,...,g...} from the high-definition frontal face images in the gallery collection. N Extract high-resolution frontal face feature information respectively {f g1 ,f g2 ,...,f gN};

[0168] The similarity list acquisition module is further configured to obtain an initial similarity list {S1, S2, ..., S...} corresponding to each of the top-angle face images based on the top-angle face feature information and the high-definition frontal face feature information, using cosine similarity. M};in,

[0169] M is the number of multiple frames of facial images of the same person from near and far angles;

[0170] N is the number of people of different identities gathered in the gallery;

[0171] S i (i = 1, 2, ..., M) represents the initial similarity list obtained from the i-th top-angle face, Si The jth element S of the middle i,j (j = 1, 2,... N) represents f pi Similarity between f gj .

[0172] Further, the fusion weight acquisition module is further used to input the depression angle face feature information into a dynamic convolution kernel generation network to obtain a first output parameter corresponding to the depression angle face feature information;

[0173] The fusion weight acquisition module is further used to obtain a convolution kernel based on the first output parameter;

[0174] The fusion weight acquisition module is further used to splice two initial similarity lists that need to be fused as first input information of a weight prediction network;

[0175] The fusion weight acquisition module is further used to obtain a shallow feature adjusted by a depression angle face based on the first input information and the convolution kernel;

[0176] The fusion weight acquisition module is further used to predict the fusion weight based on iteration of the shallow feature.

[0177] Further, the weighted fusion module is used to perform weighted fusion on initial similarity lists corresponding to different depression angle face images based on the fusion weight and a similarity list calculation formula to obtain a similarity list after fusion;

[0178] The similarity list calculation formula is:

[0179] Wherein,

[0180] w i-1 The fusion weight is in a range of 0 to 1.

[0181] Based on the above technical solution, an actual implementation case is given, and the specific case is as follows:

[0182] Step 1, data set preprocessing, including detecting and cutting the face images in the depression angle face data set and obtaining depression angle face images with a fixed size.

[0183] Step 2, extracting the features of the depression angle face, and obtaining an initial similarity list according to the similarity between the features.

[0184] Step 3, constructing a weight prediction network, which mainly consists of a dynamic convolution kernel generation module and a fusion weight generation module. The dynamic convolution kernel generation module is used to adjust the predicted weight according to the depression angle face of different angles, and the fusion weight module is used to obtain the fusion weight between the input similarity lists.

[0185] Step 4, according to the predicted fusion weight, the input two similarity lists are weighted and fused to obtain the fused similarity list.

[0186] Step 5, in order to train the weight prediction network model, a weighted ranking loss is designed to obtain the best fusion weight interval.

[0187] Specifically, the above step 1 includes the following steps:

[0188] Step 1.1: using a face detection network to detect the face region, and cutting the face image.

[0189] Step 1.2: up-sampling the cut oblique face image to a specified size;

[0190] In the specific implementation process, the retinaface face detection algorithm is used to detect the position of the face region in the image, first the left upper corner coordinate (x left-top ,y left-top ) and the right lower corner coordinate (x right-bottom ,y right-bottom ) of the face region are detected, and the rectangular region of the face is cut out according to the coordinates of the two points. Then the cut face picture is adjusted to 112x112 size.

[0191] Specifically, the above step 2 includes the following steps:

[0192] Step 2.1: extracting features {f p1 ,f p2 ,...,f pM} from multiple frames of near and far oblique face images {p1, p2,..., p M} of the same person, wherein M is the number of different oblique faces of the same person.

[0193] Step 2.2: extracting features {f g1 ,f g2 ,...,f gN} from high-definition front face images {g1, g2,..., g N} in the gallery set, wherein N is the number of different identities in the gallery set.

[0194] Step 2.3: using cosine similarity to obtain the initial similarity list {S1, S2,..., S M};

[0195] Wherein, S i (i=1, 2,...M) represents the similarity list obtained by the ith oblique face, and the jth element S i in S i,j(j = 1, 2,...N) represents f pi Similarity between f gj ;

[0196] In the implementation process, the face recognition feature extraction network used is an existing face recognition feature extraction network, and in the embodiment of the application, an ArcFace network is used, the skeleton network is an IResNet-50, and the extracted feature dimension is 512.

[0197] The data set used is a Tilt-angle Face Dataset (TFD) data set, the number of tilt-angle faces of the same person is 5, the angles are 15°, 30°, 45°, 60° and 75° from small to large, the number of gallery sets is 1399, the similarity is a cosine similarity, and the calculated similarity list is a vector with a length of 1399.

[0198] Specifically, the above step 3 includes the following steps:

[0199] Step 3.1: input the feature f pi obtained in step 2.1 into a dynamic convolution kernel generation network to obtain a tensor out i related to the tilt-angle face p i (i = 1, 2,...M).

[0200] Step 3.2: adjust the out i obtained in step 3.1 to the size and shape required by the convolution kernel, thereby obtaining a dynamic convolution kernel kernel i (i = 1, 2,...M).

[0201] Step 3.3: splice the two similarity lists S and S i together as the input in i (i = 1, 2,...M) of the weight prediction network.

[0202] Step 3.4: input the in i obtained in step 3.3 and the convolution kernel kernel i obtained in step 3.2 into the first block C1 of the weight prediction network,

[0203] wherein in i is the input of C1, and kernel i is the convolution kernel of C1, thereby obtaining a shallow feature F1 adjusted by the tilt-angle face.

[0204] Step 3.5: F1 obtained in step 3.4 is sent to the next block C1 to obtain F2, and this iteration is continued until the output F5 of the last block C5 is taken as the predicted weight w i-1 ;

[0205] In the implementation process, the dynamic convolution kernel generation network is a Long Short Term Memory (LSTM) structure, and in this example, an LSTM network with an input size of 512, a hidden layer size and an output size of 576, and 2 layers is used. The adjusted convolution kernel kernel i has a size of 64*1*3*3, and the input in i has a size of 2*1399 after splicing.

[0206] The skeleton network used by the weight prediction network is a reference to the existing resnet-18, and the resnet-18 is composed of 5 blocks, which are denoted as C1, C2,..., C5.

[0207] The difference is that the convolution kernel in the first block of resnet-18 is replaced by a dynamic convolution, and a fully connected layer is connected after the last block, so as to convert the extracted features into fusion weights.

[0208] Specifically, the above step 4 includes the following steps:

[0209] The similarity list after fusion is calculated according to the following formula:

[0210]

[0211] Where w i-1 is the fusion weight obtained in step 3.5, and the value range is between 0 and 1.

[0212] In the implementation process, the two-by-two fusion is iterated four times. The first fusion is to fuse two initial similarity lists, and each subsequent fusion is to fuse the similarity list obtained after the last fusion with the next initial similarity list.

[0213] Specifically, the above step 5 includes the following steps:

[0214] Step 5.1: Determine the optimization goal. In order to make the recognition accuracy high, the similarity value of the correct face should be ranked as far as possible in the similarity list after fusion, that is:

[0215]

[0216] Where n is the subscript of the correct face.

[0217] Step 5.2: Transform the optimization objective into the following equivalent form:

[0218] Among them, Indicator n (j,w i ) represents an indicator function, meaning if g n Is the ranking in the merged similarity list higher than g? j Firstly, its mathematical definition is as follows:

[0219]

[0220] Step 5.3: Determine a new optimization objective, in order to make g n If the ranking is higher, then the Indicator should be improved. n (j,w i ) should be as close to 0 as possible.

[0221] Step 5.4: Determine the Indicator n (j,w i When w is 0 i The conditions are met, according to step 3.5. As can be seen from the definition, when Indicator n (j,w i When ) is 0, w i The following conditions must be met:

[0222]

[0223] Among them, Jump n (j) is an indicator n (j,w i The unique jump discontinuity of ) is defined as follows:

[0224]

[0225] Step 5.5: Determine the optimal interval. For each j (j = 1, 2, ... N), a corresponding jump discontinuity point can be obtained. n (j), thus obtaining N jump discontinuities {Jump n (1), Jump n (2),...Jump n (N)}, these N jump discontinuities divide the number line into N+1 intervals, then within each interval All of them are fixed values, so by traversing these N+1 intervals, we can obtain the value that makes... The smallest interval (Jump) n (x), Jump n(y))).

[0226] Step 5.6: Define the loss function as:

[0227] Loss = max(0, (w i -Jump n (x)) x (w i -Jump n (y))) ;

[0228] In the implementation process, since the fusion weight should be a real value between 0 and 1, this example considers those intervals between 0 and 1.

[0229] Subsequently, the loss value of the fusion weight is back propagated to the weight prediction network, which can master the loss value corresponding to the fusion weight obtained by prediction, so as to be able to be referenced when predicting the fusion weight subsequently, optimize the prediction operation, and make the loss value as small as possible, so that the weight prediction network performs fusion weight optimization.

[0230] It should be noted that the corresponding technical problems, technical means and technical effects of the multi-frame depression angle face recognition device based on fusion reordering provided by the embodiments of the present application are similar to the principles of the multi-frame depression angle face recognition method based on fusion reordering.

[0231] It should be noted that in the present application, relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the sentence "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0232] The above is only a specific embodiment of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications of these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied herein.

Claims

1. A multi-frame depression angle face recognition method based on fusion reordering, characterized in that, The method comprises the following steps: Based on the oblique face feature information of the multiple oblique face images of the same person, the similarity between the high-definition frontal face feature information of different identities in the preset gallery set of high-definition frontal face images is calculated, and an initial similarity list corresponding to each oblique face image is obtained; Based on the preset weight prediction network, a fusion weight is predicted; Based on the fusion weight, the initial similarity lists corresponding to different oblique face images are weighted and fused to obtain a fused similarity list; Based on the fused similarity list, face recognition is performed in combination with the high-definition frontal face images; In the prediction of the fusion weight based on the preset weight prediction network, the following steps are included: The oblique face feature information is input into a dynamic convolution kernel generation network to obtain first output parameters corresponding to the oblique face feature information; Based on the first output parameters, a convolution kernel is obtained; The two initial similarity lists that need to be fused are spliced as first input information of the weight prediction network; Based on the first input information and the convolution kernel, a shallow feature adjusted by the oblique face is obtained; Based on the shallow feature, the fusion weight is predicted by iteration; Based on the fusion weight and a similarity list calculation formula, the initial similarity lists corresponding to different oblique face images are weighted and fused to obtain a fused similarity list; The similarity list calculation formula is: ; wherein, fusion weight, with a value ranging between 0 and 1 ; is the post fusion similarity list.

2. The multi-frame oblique face recognition method based on fusion reordering according to claim 1, wherein, The method further comprises the following steps: Based on the set fusion weight optimization target, a corresponding fusion weight optimization interval is obtained; Based on the preset loss function and the fusion weight optimization interval, a loss value of the fusion weight is calculated; The loss value of the fusion weight is back propagated to the weight prediction network for fusion weight optimization by the weight prediction network.

3. The multi-frame oblique face recognition method based on fusion reordering according to claim 2, wherein: The loss function is ; wherein, for the fusion weight, and for the fusion weight optimization interval corresponding to the two interval end values, for the loss value of the fusion weight.

4. The multi-frame low-angle face recognition method based on fusion reordering according to claim 1, wherein, Based on the oblique face feature information of the multiple oblique face images of the same person, the similarity between the high-definition frontal face feature information of different identities in the preset gallery set of high-definition frontal face images is calculated, and an initial similarity list corresponding to each oblique face image is obtained, including the following steps: Multiple frames of close and far oblique angle face images of the same person Respectively extract oblique angle face feature information ; high definition frontal face image from the gallery of high definition frontal face images extracting high definition frontal face feature information ; Based on the face feature information at the oblique angle and the high-definition face feature information, an initial similarity list corresponding to each of the face images at the oblique angle is obtained by using cosine similarity ; wherein, M is the number of multiple oblique face images of the same person; N is the number of different identities in the gallery set; represents the initial similarity list obtained from the ith oblique face, the jth element in the ith similarity list represents the similarity between and 5. A multi-frame oblique face recognition device based on fusion reordering according to any one of claims 1-4, characterized in that, The device comprises: A similarity list acquisition module is configured to calculate the similarity between the high-definition frontal face feature information of different identities in the preset gallery set of high-definition frontal face images based on the oblique face feature information of the multiple oblique face images of the same person, and obtain an initial similarity list corresponding to each oblique face image; A fusion weight acquisition module is configured to predict a fusion weight based on a preset weight prediction network; A weighted fusion module is configured to perform weighted fusion on the initial similarity lists corresponding to different oblique face images based on the fusion weight to obtain a fused similarity list; A face recognition module is configured to perform face recognition based on the fused similarity list in combination with the high-definition frontal face images.

6. The multi-frame low-angle face recognition device based on fusion reordering according to claim 5, wherein, The device further comprises a weighted ranking loss module; The weighted ranking loss module is configured to obtain a corresponding fusion re-optimization interval based on a set fusion weight optimization target; The weighted ranking loss module is further configured to calculate a loss value of the fusion weight based on a preset loss function and the fusion re-optimization interval; The weighted ranking loss module is further configured to back-propagate the loss value of the fusion weight to the weight prediction network, so that the weight prediction network performs fusion weight optimization.

7. The multi-frame depression angle face recognition device based on fusion reordering according to claim 6, wherein: The loss function is ; wherein, for the fusion weight, and for the fusion weight optimization interval corresponding to the two interval end values, for the loss value of the fusion weight.

8. The multi-frame depression angle face recognition device based on fusion reordering according to claim 5, wherein: The fusion weight acquisition module is further configured to input the depression angle face feature information into a dynamic convolution kernel generation network to obtain first output parameters corresponding to the depression angle face feature information; The fusion weight acquisition module is further configured to obtain a convolution kernel based on the first output parameters; The fusion weight acquisition module is further configured to splice two initial similarity lists that need to be fused as first input information of a weight prediction network; The fusion weight acquisition module is further configured to obtain a depression angle face adjusted shallow feature based on the first input information and the convolution kernel; The fusion weight acquisition module is further configured to predict the fusion weight based on iteration of the shallow feature.

Images