Training methods for human face matching models, human face matching methods and devices
By training a human face matching model and combining feature extraction and fusion, a head network layer is designed to achieve end-to-end detection and matching of human bodies and faces. This solves the problems of long process time and missed detection in existing technologies and improves matching accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING LONGZHI DIGITAL TECH CO LTD
- Filing Date
- 2022-12-20
- Publication Date
- 2026-05-26
Smart Images
Figure CN115984926B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence, and in particular to a training method for a human face matching model, a human face matching method, and an apparatus. Background Technology
[0002] In pedestrian recognition scenarios, there is often a need to pair faces and bodies belonging to the same person within a scene. Existing face matching methods typically involve two steps: First, a two-class detection algorithm is used to detect faces and bodies in the input image, outputting all body and face bounding boxes. Second, a matching algorithm is used to match all body and face bounding boxes, outputting the matched face-body pairs. This method requires two independent steps, resulting in a lengthy overall process. Furthermore, the two-class detection algorithm used in the first step does not consider the prior positional relationship between the body and face of the same person, which can easily lead to missed detections in this step and directly affect the accuracy of face matching in the second step.
[0003] It is evident that existing human face matching methods still suffer from time-consuming overall detection and matching processes, and are prone to missed detections in the human body and face detection steps, thus affecting the accuracy of subsequent human body and face matching. Summary of the Invention
[0004] In view of this, the present disclosure provides a training method for a human face matching model, a human face matching method and apparatus, to solve the problem that the existing human face matching methods still have a long overall process of detection and matching, and are prone to missed detection in the human body and face detection steps, thereby affecting the accuracy of subsequent human body and face matching.
[0005] A first aspect of this disclosure provides a method for training a human face matching model, comprising:
[0006] Obtain the training image set, which includes multiple training images carrying the first annotation information;
[0007] Feature extraction and fusion are performed on the training image set to obtain a fused feature map;
[0008] The fused feature map is input into the head network layer of the basic matching model for training until the preset iteration termination condition is reached to obtain the initial human face matching model. The head network layer includes a human binary classification branch and a multi-branch cell. The multi-branch cell includes a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch.
[0009] Obtain the test image set, which includes multiple test images carrying second annotation information;
[0010] Input the test image set into the initial human face matching model and output the prediction results;
[0011] Based on the prediction results and the second annotation information, the human body binary classification threshold of the human body binary classification branch and the human body binary classification threshold of the face binary classification branch are determined to obtain the final human body and face matching model.
[0012] A second aspect of this disclosure provides a human face matching method, including:
[0013] Obtain the image to be inspected;
[0014] The image to be inspected is input into the final human face matching model, which outputs the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the binary classification information of the face belonging to the same person in the image. The final human face matching model is trained using the training method of the human face matching model in the first aspect.
[0015] A third aspect of this disclosure provides a training apparatus for a human face matching model, comprising:
[0016] The first acquisition module is configured to acquire a training image set, which includes multiple training images carrying first annotation information.
[0017] The fusion module is configured to extract and fuse features from the training image set to obtain a fused feature map.
[0018] The training module is configured to input the fused feature map into the head network layer of the basic matching model for training until the preset iteration termination condition is reached to obtain the initial human face matching model. The head network layer includes a human binary classification branch and a multi-branch cell. The multi-branch cell includes a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch.
[0019] The second acquisition module is configured to acquire a test image set, which includes multiple test images carrying second annotation information.
[0020] The prediction module is configured to take a set of test images as input to the initial human face matching model and output the prediction results.
[0021] The determination module is configured to determine the human binary classification threshold of the human binary classification branch and the human binary classification threshold of the face binary classification branch based on the prediction results and the second annotation information, so as to obtain the final human-face matching model.
[0022] A fourth aspect of this disclosure provides a human face matching device, comprising:
[0023] The image acquisition module is configured to acquire images to be inspected.
[0024] The matching module is configured to input the image to be inspected into the final human face matching model and output the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the face binary classification information belonging to the same person in the image to be inspected. The final human face matching model is trained using the training method of the human face matching model in the first aspect.
[0025] A fifth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0026] A sixth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0027] Compared with the prior art, the advantages of this embodiment include at least the following: acquiring a training image set, which includes multiple training images carrying first annotation information; performing feature extraction and fusion on the training image set to obtain a fused feature map; inputting the fused feature map into the head network layer of the basic matching model for training until a preset iteration termination condition is reached to obtain an initial human face matching model, wherein the head network layer includes a human binary classification branch and multi-branch cells, and the multi-branch cells include a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch; and acquiring a test image set, which includes multiple test images carrying second annotation information. The test image set is input into the initial human face matching model, and the prediction results are output. Based on the prediction results and the second annotation information, the human binary classification threshold and the human binary classification threshold of the face binary classification branch are determined to obtain the final human face matching model. This model integrates the traditional human face detection algorithm and matching algorithm into one algorithm model, which can realize the simultaneous detection and matching of human body and face. That is, the detection and matching of human body and face can be completed in one step, thus realizing end-to-end human face matching. In addition, the model also organically combines the prior relationship of human body and face position, further improving the accuracy of human face matching. Attached Figure Description
[0028] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0029] Figure 1 This is a flowchart illustrating a training method for a human face matching model provided in an embodiment of this disclosure;
[0030] Figure 2 This is a schematic diagram of the overall structure of a basic matching model in the training method of the human face matching model provided in this embodiment of the disclosure;
[0031] Figure 3 This is a schematic diagram of a feature map fusion process in the training method of the human face matching model provided in this embodiment of the disclosure;
[0032] Figure 4 This is a schematic diagram of a feature map partitioning method in the training method of the human face matching model provided in this embodiment of the disclosure;
[0033] Figure 5 This is a flowchart illustrating a human face matching method provided in an embodiment of this disclosure;
[0034] Figure 6 This is a schematic diagram of the structure of a training device for a human face matching model provided in an embodiment of this disclosure;
[0035] Figure 7 This is a schematic diagram of the structure of a human face matching device provided in an embodiment of this disclosure;
[0036] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0037] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.
[0038] The training method, human face matching method, and apparatus for the human face matching model according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.
[0039] Figure 1 This is a flowchart illustrating a training method for a human face matching model provided in an embodiment of this disclosure. Figure 1 As shown, the training method for this human face matching model includes:
[0040] Step S101: Obtain the training image set, which includes multiple training images carrying first annotation information.
[0041] Training images can be images containing at least one pedestrian, captured by camera devices (such as surveillance cameras) set up in various application scenarios (such as traffic roads, hotel lobbies, squares, etc.).
[0042] The first annotation information refers to the body bounding box coordinates p_x1y1x2y2 and the face bounding box coordinates of each person in each training image. If the face is exposed, such as a frontal or side view, the face bounding box coordinates are labeled as f_x1y1x2y2. Otherwise, the face bounding box is not labeled, or all four coordinates of the face bounding box are marked as -1.
[0043] In practical applications, the coordinates of the human body bounding box and the face bounding box of each training image can be labeled manually or by machine.
[0044] Figure 2 This is a schematic diagram of the overall structure of a basic matching model in the training method of the human face matching model provided in this embodiment.
[0045] like Figure 2 As shown, the basic matching model mainly includes an input layer 201, a backbone network layer 202, a neck network layer 203, a head network layer 204, and an output layer 205.
[0046] The input layer 201 is used to annotate the body bounding box coordinates and face bounding box coordinates of each person in each training image in the training image set input to the basic matching model, and input the training image carrying the first annotation information into the backbone network layer 202.
[0047] The backbone network layer 202 can employ feature extraction networks such as CSPDarknet or Darknet, with CSPDarknet being the preferred choice. The backbone network layer 202 is primarily used to extract features from the training images input to the input layer 201, obtaining feature maps, and then transmitting these feature maps to the neck network layer 203.
[0048] The backbone network layer 202 includes five network layers: the first network layer, the second network layer, the third network layer, the fourth network layer, and the fifth network layer.
[0049] The neck network layer 203 can be a YOLOX-PAFPN network. The neck network layer 203 is mainly used to fuse the feature maps transmitted from the backbone network layer 202 to obtain a fused feature map, and then transmit the fused feature map to the head network layer 204.
[0050] Traditional two-class detection consists of two branches: a classification branch to determine whether the current target is a human or face bounding box, and a bounding box regression branch to regress the four coordinate values (x1, y1, x2, y2) of the bounding box. However, traditional two-class detection can only detect all human and face bounding boxes in each image, and cannot match the human and face bounding boxes of the same person. Unlike traditional two-class detection, the head network layer 204 designed in this embodiment has two generalized branches: a human binary classification branch 2041 and a multi-branch cell 2042. The multi-branch cell 2042 integrates three detection branches: a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch. This embodiment, through a clever head network layer structure design, transforms the two independently performed detection steps of human detection and face detection into a generalized single-class human detection, thereby achieving simultaneous detection and matching of faces and bodies, and simultaneously outputting the face bounding box corresponding to the same person's body, thus realizing the matching of human bodies and faces. The generalized human body here includes three outputs: human body frame, human face frame, and whether there is a human face frame.
[0051] The head network layer 204 is mainly used to detect and match human faces using the fused feature map transmitted from the neck network layer 203, and transmit the detection and matching results to the output layer 205.
[0052] Output layer 205 is mainly used to output the detection and matching results transmitted from header network layer 204.
[0053] Step S102: Extract and fuse features from the training image set to obtain a fused feature map.
[0054] During each round of model training, several training images can be randomly selected from the training image set as input. For example, if the training image set contains 100 training images, 4 images can be randomly selected from the 100 training images as input in each round of model training. The input training image set for each round can be labeled in the form of [N,C,H,W], where N represents the number of training images, C represents the number of channels, H represents the height of the training image, and W represents the width of the training image.
[0055] The following section uses a training image as an example to explain the operation process of step S102 in detail. An original training image has dimensions [1,16,640,640] ([N,C,H,W]). This original training image is input into the backbone network layer 202 via the input layer 201. The image features are extracted sequentially through the first, second, third, fourth, and fifth network layers of the backbone network layer 202, resulting in five feature maps with dimensions [1,32,320,320] ([N,C,H,W]) (output of the first network layer), [1,64,160,160] ([N,C,H,W]) (output of the second network layer), [1,128,80,80] ([N,C,H,W]) (output of the third network layer), [1,256,40,40] ([N,C,H,W]) (output of the fourth network layer), and [1,512,20,20] ([N,C,H,W]) (output of the fifth network layer).
[0056] Next, combined Figure 3 The feature maps (C3, C4, C5) output from the third, fourth, and fifth network layers of the backbone network layer 202 are respectively input into the first, second, and third network layers of the neck network layer 203. First, the C5 feature map with dimensions [1,512,20,20] ([N,C,H,W]) is upsampled to obtain the C5' feature map [1,256,40,40] ([N,C,H,W]). Then, the C4 feature map with dimensions [1,256,40,40] ([N,C,H,W]) is fused with the C5' feature map to obtain the C4+C5' fused feature map. Next, the C4+C5' fused feature map is upsampled to obtain the (C4+C5')' feature map with dimensions [1,128,80,80] ([N,C,H,W]). Finally, the C3 feature map with dimensions [1,128,80,80] ([N,C,H,W]) is fused with the (C4+C5')' feature map to obtain the P3 fused feature map. The P3 fusion feature map is downsampled to obtain the P3' fusion feature map with dimensions [1,256,40,40] ([N,C,H,W]). The P3' fusion feature map is then fused with the C4+C5' fusion feature map to obtain the P4 fusion feature map. The P4 fusion feature map is then downsampled to obtain the P4' fusion feature map with dimensions [1,512,20,20] ([N,C,H,W]). The P4' fusion feature map is then fused with the C5 feature map to obtain the P5 fusion feature map.
[0057] The receptive fields of the P3, P4, and P5 fusion feature maps obtained after the above processing are successively increased compared to the original training images, and they are respectively good at detecting small (corresponding to the 8×8 region of the original training image), medium (corresponding to the 16×16 region of the original training image), and large (corresponding to the 32×32 region of the original training image) targets.
[0058] Then, the P3 fusion feature map, P4 fusion feature map, and P5 fusion feature map are input into the head network layer 204.
[0059] in, Figure 3 In this diagram, Backbone represents the backbone network layer, Neck represents the neck network layer, Head represents the head network layer, C3 represents the feature map output of the third network layer of backbone network layer 202, C4 represents the feature map output of the fourth network layer of backbone network layer 202, C5 (or ①) represents the feature map output of the fifth network layer of backbone network layer 202, ② represents the C4+C5' fused feature map, ③ represents the P3 fused feature map (abbreviated as P3), ④ represents the P4 fused feature map (abbreviated as P4), and ⑤ represents the P5 fused feature map (abbreviated as P5).
[0060] Step S103: Input the fused feature map into the head network layer of the basic matching model for training until the preset iteration termination condition is reached to obtain the initial human face matching model. The head network layer includes a human binary classification branch and multi-branch cells. The multi-branch cells include a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch.
[0061] In some embodiments, step S103 above may specifically include:
[0062] The fused feature map is input into the human body binary classification branch to obtain the human body classification result;
[0063] The fused feature map is input into the multi-branch cell to obtain human bounding box coordinates, face bounding box coordinates, and face classification results, thus completing one round of training.
[0064] Training stops when the number of training rounds reaches a preset threshold or the accuracy of the base model reaches a preset accuracy threshold, and the initial human face matching model is obtained.
[0065] This explanation will focus on training the head network layer of the basic matching model using the P3 fused feature map as an example. The P3 fused feature map is input into the human body binary classification branch of the head network layer. The detection algorithm in this branch determines whether a human body is present in the P3 fused feature map and outputs the human body classification result, i.e., the probability value of a human body. Simultaneously, the P3 fused feature map is input into the multi-branch cells of the head network layer. The embedded algorithms of the human body bounding box regression branch, face bounding box regression branch, and face binary classification branch within these multi-branch cells are used for detection. This yields the human body bounding box coordinates output by the human body bounding box regression branch, the face bounding box coordinates output by the face bounding box regression branch, and the face classification result output by the face binary classification branch, i.e., the probability value of a face.
[0066] Similarly, the training methods for P4 and P5 fusion feature maps are the same as those for P3 fusion feature maps, and will not be repeated here.
[0067] Next, the non-maximum suppression algorithm is used to process the three sets of training results (including human body classification results, human body bounding box coordinate information, face bounding box coordinate information and face classification results) obtained by training the fused feature maps of P3, P4 and P5 through the above steps, so as to obtain the human face matching training result of the training image, and output the human face matching training result through the output layer 205.
[0068] By processing several training images in one round according to the above steps, the human face matching training result of each training image is obtained, thus completing one round of training.
[0069] Then, repeat the above training steps, and continue to randomly select a number of training images from the training image set as input for training until the preset number of rounds threshold is reached (which can be flexibly set according to the actual situation, for example, it can be set to 50 rounds, 100 rounds, etc.), or the accuracy of the basic model reaches the preset accuracy threshold (which can be flexibly set according to the actual situation, for example, it can be set to 50%, 60%, 70%, 80%, etc.), then stop training and obtain the initial human face matching model.
[0070] In some embodiments, the above steps involve fusing feature maps into multi-branch cells to obtain human bounding box coordinates, face bounding box coordinates, and face classification results, which may specifically include:
[0071] The fused feature map is divided into multiple feature map blocks;
[0072] Perform human detection on each feature map block to obtain the human detection results;
[0073] Based on the human body detection results, the first positive sample is selected from multiple feature blocks;
[0074] The first positive sample is input into the multi-branch cell to obtain the human body bounding box coordinates, the face bounding box coordinates, and the face classification result.
[0075] Object detection requires simultaneously outputting the position coordinates and category of each object in the image. To locate multiple objects, the fused feature map is divided, specifically by dividing the fused feature map into a grid, resulting in multiple feature map patches of the same size. In an exemplary embodiment, taking the P3 fused feature map as an example, as follows... Figure 4 As shown, the P3 fused feature map is divided into nine feature map blocks of equal size, labeled from left to right as 01, 02, 03, 04, 05, 06, 07, 08, and 09. Next, human detection is performed on each feature map block, that is, the presence of a human body is detected in each feature map block, yielding the human detection results. Figure 4 As shown, the feature patches containing human figures are 01, 04, 05, 06, 08, and 09. These six feature patches are designated as the first positive samples (i.e., feature patches with human bounding boxes), while the remaining feature patches 02, 03, and 07 are designated as the first negative samples (i.e., feature patches without human bounding boxes). Next, these six feature patches are input into a multi-branch cell for detection. This involves using the multi-branch cell to detect each of the six feature patches, obtaining the human bounding box coordinates, face bounding box coordinates, and face classification result for each feature patch. Then, a non-maximum suppression algorithm is used to filter the detection results of these six feature patches, yielding the final human-face matching training result, which is output through output layer 205.
[0076] Similarly, the positive and negative sample allocation method and training method for P4 fusion feature map and P5 fusion feature map are the same as those for P3 fusion feature map, and will not be repeated here.
[0077] Subsequently, the non-maximum suppression algorithm is used to process the three sets of training results (including human body classification results, human body bounding box coordinate information, face bounding box coordinate information and face classification results) obtained by the above positive and negative sample allocation method and training method for the fused feature maps of P3, P4 and P5, to obtain the human face matching training result of the training image, and output the human face matching training result through the output layer 205.
[0078] In other embodiments, the first positive sample is input into the multi-branch cell to obtain human bounding box coordinates, face bounding box coordinates, and face classification results, including:
[0079] Face detection is performed on the first positive sample to obtain the face detection results;
[0080] Based on the face detection results, a second positive sample is selected from the first positive sample;
[0081] The second positive sample is input into the multi-branch cell to obtain the human body bounding box coordinates, the face bounding box coordinates, and the face classification results.
[0082] As an example, combined with Figure 4 Face detection is performed on the first positive sample, specifically on feature patches 01, 04, 05, 06, 08, and 09, yielding face detection results (i.e., probability values of faces or bounding boxes). Feature patches in the first positive sample whose face probability values are greater than a preset threshold or whose bounding box coordinates are displayed are designated as the second positive sample. For example... Figure 4 As shown, if a face bounding box is detected in feature blocks 01, 04, 05, and 06, then feature blocks 01, 04, 05, and 06 are determined as the second positive samples, and the remaining feature blocks 08 and 09 are determined as the second negative samples.
[0083] The specific process of inputting the second positive sample into the multi-branch cell to obtain the human bounding box coordinates, face bounding box coordinates, and face classification results is basically the same as the process of inputting the first positive sample into the multi-branch cell to obtain the human bounding box coordinates, face bounding box coordinates, and face classification results, and will not be repeated here.
[0084] The scheme of this disclosure involves dividing the fused feature map into multiple feature map patches. Human detection and face detection are then performed on these patches to further divide them into positive and negative samples. Positive samples indicate the presence of a target in the current grid, while negative samples indicate the absence of a target. In subsequent training, only positive samples are used for model training, which helps improve the model's recall rate.
[0085] The first positive and negative sample allocation is performed using the human body bounding box to represent the entire generalized human body position, since there will always be a human body bounding box for each pedestrian. In addition, the face bounding box is extracted separately for the second sample allocation, thereby selecting more positive samples for learning, which can further improve the recall rate.
[0086] Step S104: Obtain the test image set, which includes multiple test images carrying the second annotation information.
[0087] Test images can be images containing at least one pedestrian, captured by camera devices (such as surveillance cameras) set up in various application scenarios (such as traffic roads, hotel lobbies, squares, etc.).
[0088] The second annotation information refers to the body bounding box coordinates p_x1y1x2y2 and the face bounding box coordinates of each person in each test image. If the face is visible, such as a frontal or side view, the face bounding box coordinates are labeled as f_x1y1x2y2. Otherwise, the face bounding box is not labeled, or all four coordinates of the face bounding box are marked as -1.
[0089] Step S105: Input the test image set into the initial human face matching model and output the prediction results.
[0090] As an example, suppose there are M test images in the test image set, where M is a positive integer greater than or equal to 1. Then, the M test images can be input in batches into the initial human face matching model trained by the above steps for inference testing, and the prediction result corresponding to each test image can be output. The prediction result includes the coordinates of the human body bounding box, the coordinates of the face bounding box, and whether there is a face bounding box (or the probability value of having a face bounding box).
[0091] Step S106: Based on the prediction results and the second annotation information, determine the human body binary classification threshold of the human body binary classification branch and the human body binary classification threshold of the face binary classification branch to obtain the final human body and face matching model.
[0092] In some embodiments, the second annotation information includes human bounding box annotation information, and the prediction result includes human bounding box prediction information. The human bounding box annotation information typically refers to the human bounding box coordinates p_x1y1x2y2. The human bounding box prediction information typically refers to the human bounding box coordinates p'_x1'y1'x2'y2' inferred from the initial human face matching model.
[0093] Based on the prediction results and the second annotation information, the human binary classification threshold for the human binary classification branch is determined. Specifically, this includes: calculating the accuracy and recall of human bounding box detection for each threshold in the preset threshold range based on the human bounding box annotation information, human bounding box prediction information, and the number of images in the test image set; calculating the first score value corresponding to each threshold based on the accuracy and recall of human bounding box detection; and determining the human binary classification threshold for the human binary classification branch based on the first score value.
[0094] The preset threshold range can usually be set to [0.1~1].
[0095] As an example, thresholds from 0.1 to 1 are iterated at intervals of 0.01. Based on the human bounding box annotations, predicted human bounding boxes, and the number of images in the test image set, the precision (P) and recall (R) of human bounding box detection at each threshold are calculated. Taking a threshold of 0.01 as an example, M test images are input in batches into the initial human face matching model trained in the above steps for inference testing, and the prediction result corresponding to each test image is output. The human bounding box annotations and predicted human bounding box information of each test image are compared. Specifically, the coordinate difference between the pre-annotated human bounding box coordinates and the human bounding box coordinates predicted by the model is compared to obtain the comparison result. Then, based on the comparison result and the number of images M in the test image set, the precision P and recall R of the test image set are calculated. Here, precision P refers to the proportion of the number of correctly predicted test images out of the total number of test image images. Recall R refers to the proportion of samples that actually contain human bounding boxes but are predicted to contain human bounding boxes out of the total number of samples that actually contain human bounding boxes. Next, the first score value F1 when the threshold is 0.01 is calculated according to the following formula F1=2*P*R / (P+R).
[0096] Similarly, the calculation method for the first score value F1 with thresholds of 0.02, 0.03, 0.04....1 is the same as the calculation method for the first score value F1 with a threshold of 0.01, and will not be repeated here.
[0097] The first score can be used to evaluate the model accuracy at the current threshold. Preferably, the threshold corresponding to the maximum value among the aforementioned series of first scores is determined as the human binary classification threshold for the human binary classification branch. For example, if the threshold corresponding to the maximum value among the aforementioned series of first scores is 0.85, then the threshold of 0.85 can be determined as the human binary classification threshold for the human binary classification branch. In practical applications, when the detection result of the human binary classification branch is greater than 0.85, a human bounding box is determined to be present.
[0098] Understandably, the interval of the above traversal threshold can be flexibly set according to the actual situation; for example, it can also be 0.02, 0.03, etc. Usually, 0.01 is used as the traversal interval.
[0099] In some embodiments, the second annotation information further includes face bounding box annotation information, and the prediction result includes face bounding box prediction information. Face bounding box annotation information typically refers to face bounding box coordinates f_x1y1x2y2. Face bounding box prediction information typically refers to face bounding box coordinates f'_x1'y1'x2'y2' inferred from the initial human face matching model.
[0100] Based on the prediction results and the second annotation information, the face binary classification threshold of the face binary classification branch is determined. Specifically, this may include: calculating the accuracy and recall of face bounding box detection corresponding to each threshold in the preset threshold range based on the face bounding box annotation information, face bounding box prediction information, and the number of images in the test image set; calculating the second score value corresponding to each threshold based on the accuracy and recall of face bounding box detection; and determining the face binary classification threshold of the face binary classification branch based on the second score value.
[0101] The specific process of determining the face binary classification threshold of the face binary classification branch based on the face bounding box annotation information and the face bounding box prediction information in this embodiment is similar to the specific process of determining the human binary classification threshold of the human binary classification branch based on the prediction results and the second annotation information described above.
[0102] The threshold corresponding to the maximum value among the aforementioned series of second score values F2 is determined as the face binary classification threshold for the face binary classification branch. For example, if the threshold corresponding to the maximum value among the aforementioned series of second score values is 0.70, then the threshold of 0.70 can be determined as the face binary classification threshold for the face binary classification branch. In practical applications, when the detection result of the face binary classification branch is greater than 0.70, a face bounding box is determined.
[0103] In other embodiments, the second annotation information also includes face bounding box annotation information, and the prediction result includes face bounding box prediction information.
[0104] In this embodiment, the output value of the face binary classification branch in the multi-cell branch can be regarded as the conditional probability of the existence of a face bounding box given the existence of a human bounding box. Therefore, the final face probability is the product of the human bounding box and the face classification branch. Based on the prediction result and the second annotation information, the human bounding box binary classification threshold of the face binary classification branch is determined. Specifically, this includes: calculating the accuracy and recall of face bounding box detection for each threshold in the preset threshold range based on the face bounding box annotation information, face bounding box prediction information, and the number of images in the test image set; calculating the second score value corresponding to each threshold based on the accuracy and recall of face bounding box detection; determining the candidate face threshold based on the second score value; and determining the face binary classification threshold of the face binary classification branch based on the human bounding box binary classification threshold and the candidate face threshold.
[0105] Specifically, the calculation method for the second score value F2 corresponding to each threshold can be referred to the previous description, and will not be repeated here. Next, the threshold corresponding to the maximum value among the series of second score values F2 is determined as the candidate face threshold. For example, if the threshold corresponding to the maximum value among the series of second score values calculated in the above steps is 0.70, then the threshold of 0.70 can be determined as the candidate face threshold. If the human body binary classification threshold determined in the above steps is 0.85, then the product of the human body binary classification threshold 0.85 and the candidate face threshold 0.70 is further calculated, that is, 0.85 × 0.70 = 0.595. 0.595 is determined as the face binary classification threshold for the face binary classification branch.
[0106] In some embodiments, P and R in the above formulas for calculating the first and second scores can be further weighted by appropriate coefficients based on actual circumstances. For example, the formula for calculating the first score can be: F1 = w * P * R / (w * P + R), where w is the weighting coefficient. w can be flexibly set according to actual circumstances, for example, it can be set to 2, 3, 4, etc., without limitation.
[0107] The technical solution provided in this disclosure integrates human face matching into the final human face matching model. It innovatively designs the head network layer into a human binary classification branch and a multi-branch cell. This multi-branch cell includes three branches: a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch. This cleverly transforms the traditional two independently performed detection steps of human detection and face detection into a generalized single-class human detection. This achieves end-to-end direct output of human bounding boxes, face bounding boxes, and the matching relationship between human bounding boxes and face bounding boxes, significantly shortening the detection and matching time of the human face matching task and improving its efficiency. Furthermore, during the model training phase, the division of the fused feature map and the selection of positive samples for subsequent model training improve the model's accuracy. Additionally, determining the human threshold for the human binary classification branch and the face threshold for the face binary classification branch through the above methods further enhances the model's detection and matching accuracy.
[0108] The technical solution provided in this disclosure involves: acquiring a training image set, which includes multiple training images carrying first annotation information; performing feature extraction and fusion on the training image set to obtain a fused feature map; inputting the fused feature map into the head network layer of the basic matching model for training until a preset iteration termination condition is reached to obtain an initial human face matching model, wherein the head network layer includes a human binary classification branch and multi-branch cells, and the multi-branch cells include a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch; acquiring a test image set, which includes multiple test images carrying second annotation information; and inputting the test image... The initial human-face matching model is input into a set of data, and the prediction results are output. Based on the prediction results and the second annotation information, the human binary classification threshold for the human binary classification branch and the human binary classification threshold for the face binary classification branch are determined to obtain the final human-face matching model. This model integrates the traditional human-face two-class detection algorithm and matching algorithm into one algorithm model, which can realize the simultaneous detection and matching of human bodies and faces. That is, the detection and matching of human bodies and faces can be completed in one step, thus realizing end-to-end human-face matching. Furthermore, the model also organically combines the prior positional relationship between human bodies and faces, further improving the accuracy of human-face matching.
[0109] Figure 5 This is a flowchart illustrating a human face matching method provided in an embodiment of this disclosure. Figure 5 As shown, the human face matching method includes:
[0110] Step S501: Obtain the image to be inspected.
[0111] The images to be inspected can be pictures / images captured by camera devices (such as surveillance cameras) set up in various application scenarios (such as traffic roads, hotel lobbies, squares, etc.).
[0112] Step S502: Input the image to be inspected into the final human face matching model, and output the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the face binary classification information belonging to the same person in the image to be inspected. The final human face matching model is trained by the training method of the human face matching model described above.
[0113] The collected image to be tested is input into the final human face matching model trained by the training method of the human face matching model mentioned above, and the output is the coordinates of the face bounding box, the coordinates of the human body bounding box, and the classification result of whether a face bounding box exists (face bounding box exists / face bounding box does not exist) of the same person in the image to be tested.
[0114] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.
[0115] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.
[0116] Figure 6 This is a schematic diagram of the structure of a training device for a human face matching model provided in an embodiment of this disclosure. Figure 6 As shown, the training device for this human face matching model includes:
[0117] The first acquisition module 601 is configured to acquire a training image set, which includes multiple training images carrying first annotation information.
[0118] The fusion module 602 is configured to extract and fuse features from the training image set to obtain a fused feature map.
[0119] Training module 603 is configured to input the fused feature map into the head network layer of the basic matching model for training until the preset iteration termination condition is reached to obtain the initial human face matching model. The head network layer includes a human binary classification branch and a multi-branch cell. The multi-branch cell includes a human bounding box regression branch, a face bounding box regression branch and a face binary classification branch.
[0120] The second acquisition module 604 is configured to acquire a test image set, which includes multiple test images carrying second annotation information.
[0121] Prediction module 605 is configured to input a set of test images into an initial human face matching model and output prediction results;
[0122] The determination module 606 is configured to determine the human binary classification threshold of the human binary classification branch and the human binary classification threshold of the face binary classification branch based on the prediction results and the second annotation information, so as to obtain the final human-face matching model.
[0123] In some embodiments, the training module 603 includes:
[0124] The first training unit is configured to input the fused feature map into the human body binary classification branch to obtain the human body classification result;
[0125] The second training unit is configured to input the fused feature map into the multi-branch cell to obtain human bounding box coordinate information, face bounding box coordinate information and face classification results, thus completing one round of training.
[0126] The third training unit is configured to stop training and obtain an initial human face matching model when the number of training rounds reaches a preset threshold or the accuracy of the base model reaches a preset accuracy threshold.
[0127] In some embodiments, the second training unit described above can be specifically configured as follows:
[0128] The fused feature map is divided into multiple feature map blocks;
[0129] Perform human detection on each feature map block to obtain the human detection results;
[0130] Based on the human body detection results, the first positive sample is selected from multiple feature blocks;
[0131] The first positive sample is input into the multi-branch cell to obtain the human body bounding box coordinates, the face bounding box coordinates, and the face classification result.
[0132] In some embodiments, the first positive sample is input into a multi-branch cell to obtain human bounding box coordinate information, face bounding box coordinate information, and face classification results, specifically including:
[0133] Face detection is performed on the first positive sample to obtain the face detection results;
[0134] Based on the face detection results, a second positive sample is selected from the first positive sample;
[0135] The second positive sample is input into the multi-branch cell to obtain the human body bounding box coordinates, the face bounding box coordinates, and the face classification results.
[0136] In some embodiments, the second annotation information includes human bounding box annotation information, and the prediction result includes human bounding box prediction information. The determination module 606 includes a first threshold determination unit. The first threshold determination unit is configured to: determine the human binary classification threshold for the human binary classification branch based on the prediction result and the second annotation information. Specifically, based on the human bounding box annotation information, the human bounding box prediction information, and the number of images in the test image set, calculate the accuracy and recall of human bounding box detection corresponding to each threshold in a preset threshold range; calculate a first score value corresponding to each threshold based on the accuracy and recall of human bounding box detection; and determine the human binary classification threshold for the human binary classification branch based on the first score value.
[0137] In some embodiments, the second annotation information further includes face bounding box annotation information, and the prediction result includes face bounding box prediction information. The determination module 606 further includes a second threshold determination unit. The second threshold determination unit is configured to: determine the face binary classification threshold for the face binary classification branch based on the prediction result and the second annotation information. Specifically, based on the face bounding box annotation information, the face bounding box prediction information, and the number of images in the test image set, calculate the accuracy and recall of face bounding box detection corresponding to each threshold in the preset threshold range; calculate the second score value corresponding to each threshold based on the accuracy and recall of face bounding box detection; and determine the face binary classification threshold for the face binary classification branch based on the second score value.
[0138] In other embodiments, the face binary classification threshold of the face binary classification branch is determined based on the prediction result and the second annotation information, including:
[0139] Based on the face bounding box annotation information, face bounding box prediction information, and the number of images in the test image set, the accuracy and recall of face bounding box detection corresponding to each threshold in the preset threshold range are calculated; based on the accuracy and recall of face bounding box detection, the second score value corresponding to each threshold is calculated; the candidate face threshold is determined based on the second score value; based on the human body binary classification threshold and the candidate face threshold, the face binary classification threshold of the face binary classification branch is determined.
[0140] Figure 7 This is a schematic diagram of the structure of a human face matching device provided in an embodiment of this disclosure. Figure 7 As shown, the human face matching device includes:
[0141] Image acquisition module 701 is configured to acquire images to be inspected.
[0142] The matching module 702 is configured to input the image to be inspected into the final human face matching model and output the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the face binary classification information belonging to the same person in the image to be inspected. The final human face matching model is trained by the training method of the human face matching model described above.
[0143] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.
[0144] Figure 8 This is a schematic diagram of the electronic device 8 provided in an embodiment of this disclosure. Figure 8 As shown, the electronic device 8 of this embodiment includes a processor 801, a memory 802, and a computer program 803 stored in the memory 802 and executable on the processor 801. When the processor 801 executes the computer program 803, it implements the steps in the various method embodiments described above. Alternatively, when the processor 801 executes the computer program 803, it implements the functions of each module / unit in the various device embodiments described above.
[0145] Electronic device 8 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 8 may include, but is not limited to, processor 801 and memory 802. Those skilled in the art will understand that... Figure 8 This is merely an example of electronic device 8 and does not constitute a limitation on electronic device 8. It may include more or fewer components than shown, or different components.
[0146] The processor 801 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0147] The memory 802 can be an internal storage unit of the electronic device 8, such as a hard disk or RAM of the electronic device 8. The memory 802 can also be an external storage device of the electronic device 8, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 8. The memory 802 can also include both internal and external storage units of the electronic device 8. The memory 802 is used to store computer programs and other programs and data required by the electronic device.
[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0149] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0150] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.
Claims
1. A method for training a human face matching model, characterized in that, include: Obtain a training image set, which includes multiple training images carrying first annotation information; Feature extraction and fusion are performed on the training image set to obtain a fused feature map; The fused feature map is input into the head network layer of the basic matching model for training until the preset iteration termination condition is reached to obtain the initial human face matching model. The head network layer includes a human binary classification branch and a multi-branch cell. The multi-branch cell includes a human bounding box regression branch, a face bounding box regression branch, and a face binary classification branch. Obtain a test image set, which includes multiple test images carrying second annotation information; Input the test image set into the initial human face matching model and output the prediction results; Based on the prediction results and the second annotation information, the human body binary classification threshold of the human body binary classification branch and the face binary classification threshold of the face binary classification branch are determined to obtain the final human body and face matching model. The fused feature map is input into the head network layer of the basic matching model for training until a preset iteration termination condition is reached, resulting in an initial human face matching model, including: The fused feature map is input into the human body binary classification branch to obtain the human body classification result; The fused feature map is input into the multi-branch cell to obtain human bounding box coordinate information, face bounding box coordinate information and face classification results, thus completing one round of training; When the number of training rounds reaches a preset threshold or the accuracy of the basic matching model reaches a preset accuracy threshold, training stops and an initial human face matching model is obtained. The fused feature map is input into the multi-branch cell to obtain human bounding box coordinate information, face bounding box coordinate information, and face classification results, including: The fused feature map is divided into multiple feature map blocks; Human detection is performed on each of the aforementioned feature maps to obtain human detection results; Based on the human body detection results, a first positive sample is selected from the plurality of feature blocks; The first positive sample is input into the multi-branch cell to obtain human body bounding box coordinate information, face bounding box coordinate information, and face classification results.
2. The method of claim 1, wherein, The first positive sample is input into the multi-branch cell to obtain human bounding box coordinate information, face bounding box coordinate information, and face classification results, including: Face detection is performed on the first positive sample to obtain the face detection results; Based on the face detection results, a second positive sample is selected from the first positive sample; The second positive sample is input into the multi-branch cell to obtain human body bounding box coordinate information, face bounding box coordinate information, and face classification results.
3. The method of claim 1, wherein, The second annotation information includes human bounding box annotation information, and the prediction result includes human bounding box prediction information; Based on the prediction results and the second annotation information, the human binary classification threshold of the human binary classification branch is determined, including: Based on the human bounding box annotation information, human bounding box prediction information, and the number of images in the test image set, the accuracy and recall of human bounding box detection corresponding to each threshold in the preset threshold range are calculated. Based on the accuracy and recall of the human body bounding box detection, calculate the first score value corresponding to each threshold; The human binary classification threshold of the human binary classification branch is determined based on the first score.
4. The method of claim 1, wherein, The second annotation information also includes face bounding box annotation information, and the prediction result includes face bounding box prediction information; Based on the prediction results and the second annotation information, the face binary classification threshold of the face binary classification branch is determined, including: Based on the face bounding box annotation information, face bounding box prediction information, and the number of images in the test image set, the accuracy and recall of face bounding box detection corresponding to each threshold in the preset threshold range are calculated. Based on the accuracy and recall of the face bounding box detection, a second score value corresponding to each threshold is calculated; The face binary classification threshold of the face binary classification branch is determined based on the second score.
5. The method of claim 3, wherein, The second annotation information also includes face bounding box annotation information, and the prediction result includes face bounding box prediction information; Based on the prediction results and the second annotation information, the face binary classification threshold of the face binary classification branch is determined, including: Based on the face bounding box annotation information, face bounding box prediction information, and the number of images in the test image set, the accuracy and recall of face bounding box detection corresponding to each threshold in the preset threshold range are calculated. Based on the accuracy and recall of the face bounding box detection, a second score value corresponding to each threshold is calculated; The candidate face threshold is determined based on the second score. Based on the human body binary classification threshold and the candidate face threshold, the face binary classification threshold of the face binary classification branch is determined.
6. A human face matching method, characterized by, include: Obtain the image to be inspected; The image to be inspected is input into the final human face matching model, which outputs the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the face binary classification information belonging to the same person in the image to be inspected. The final human face matching model is trained by the training method of the human face matching model as described in any one of claims 1 to 5.
7. A training device for a human face matching model, characterized in that, The apparatus is used to implement the method as described in any one of claims 1 to 5, the apparatus comprising: The first acquisition module is configured to acquire a training image set, which includes multiple training images carrying first annotation information. The fusion module is configured to extract and fuse features from the training image set to obtain a fused feature map. The training module is configured to input the fused feature map into the head network layer of the basic matching model for training until a preset iteration termination condition is reached to obtain an initial human face matching model. The head network layer includes a human binary classification branch and a multi-branch cell, and the multi-branch cell includes a human bounding box regression branch, a face bounding box regression branch and a face binary classification branch. The second acquisition module is configured to acquire a test image set, which includes multiple test images carrying second annotation information. The prediction module is configured to input the test image set into the initial human face matching model and output the prediction result; The determination module is configured to determine the human body binary classification threshold of the human body binary classification branch and the face binary classification threshold of the face binary classification branch based on the prediction result and the second annotation information, so as to obtain the final human body and face matching model.
8. A human face matching device, characterized in that, include: The image acquisition module is configured to acquire images to be inspected. The matching module is configured to input the image to be inspected into the final human face matching model, and output the coordinate information of the face bounding box, the coordinate information of the human body bounding box, and the face binary classification information belonging to the same person in the image to be inspected. The final human face matching model is trained by the training method of the human face matching model as described in any one of claims 1 to 5.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5, or the steps of the method as described in claim 6.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5, or the steps of the method as described in claim 6.