Specific face detection method and device driven by pedestrian retrieval

Abstract

The invention discloses a pedestrian retrieval-driven specific face detection method and device, and the method comprises the steps: collecting road monitoring video data, carrying out the random frame extraction, and marking pedestrians and faces, so as to construct a training set; constructing a target detection model based on the improved YOLOv8, and training and optimizing the model by using the training set; acquiring a pedestrian and face detection result by using the trained target detection model, and performing one-to-one matching on the face and the pedestrian by using a unique target matching method to ensure that each reserved pedestrian has unique matched face information; constructing and training a pedestrian re-identification model; and carrying out identity matching on the reserved pedestrians by utilizing the trained pedestrian re-identification model, accurately identifying target persons, and extracting and outputting facial images of the target persons. According to the invention, the target object in the non-front-face state can be tracked and detected in the continuous cross-camera monitoring video, and the face information of the target object can be accurately captured at the moment that the front face of the target object faces the camera.

Description

Technical Field

[0001] The present invention relates to the technical field of specific face detection in pedestrian detection and pedestrian re-identification, and in particular to a pedestrian retrieval-driven specific face detection method and device. Background Art

[0002] While traditional face detection algorithms can quickly locate faces in surveillance videos, they still have significant limitations when it comes to precisely locating specific target faces. Due to the prevalence of multiple pedestrians in video scenes, even after initial detection and screening, secondary localization is still required among a large number of candidate targets, a significant workload. Therefore, specific target face detection has become a key technical challenge urgently awaiting breakthroughs in the field of intelligent video analysis. Furthermore, in actual surveillance systems, when a pedestrian in the search image is viewed from a non-frontal perspective, such as from the back or side, accurately retrieving and extracting the target object's facial information remains a significant technical challenge.

[0003] In the field of video surveillance object detection, pedestrians and faces are typical cross-scale detection targets. It is estimated that the average pixel ratio of a face in a 1080p video frame is only 0.8%-1.2%, resulting in a significant disproportionate ratio (>1:100) compared to the background area. This small target detection challenge and the need for lightweight models constitute a core contradiction that needs to be resolved: traditional downsampling modules (such as strided convolution and pooling layers) cause the resolution of feature maps to decrease exponentially when compressing computational complexity (resolution decay rate = 1 / (2^n)), resulting in the loss of critical details. Therefore, it is urgent to develop a new feature preservation mechanism to achieve cross-layer progressive fusion of tiny facial features.

[0004] The present invention introduces person re-identification technology to timely locate the face of the target object in continuous video frames. In the person re-identification task in open scenes, due to the complex factors such as posture deformation, perspective difference, occlusion interference, etc. that the target object may have, how to effectively extract deformation features with strong discriminative power becomes a key bottleneck restricting the improvement of algorithm performance. Although the traditional method based on global features has advantages in computational efficiency, it is difficult to capture the subtle deformation features of the local structure of the human body, resulting in insufficient feature robustness in cross-camera scenarios. The current mainstream research mainly advances in two directions: (1) fine-grained analysis methods based on local feature alignment, through semantic segmentation-assisted block strategy or attention mechanism-guided feature localization, to achieve deformation modeling of human parts (such as head, torso, limbs); (2) deep learning architecture based on feature enhancement, using multi-branch network structure to fuse global-local features, or dynamically adjust the receptive field through deformable convolution to adapt to deformation features. However, these methods often face the problem of increased computational complexity. Therefore, how to efficiently extract human deformation features while keeping global features lightweight is a topic of great research value. Summary of the Invention

[0005] The purpose of the present invention is to address the deficiencies of the prior art and provide a method and device for detecting specific faces driven by pedestrian retrieval.

[0006] The objective of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a pedestrian retrieval-driven specific face detection method, comprising the following steps:

[0007] (1) Collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, mark all pedestrian targets and mark faces that meet the predetermined requirements to construct a training set;

[0008] (2) Build a target detection model based on the improved YOLOv8, and use the training set to train and optimize the target detection model. During the training process, the optimization goal is to minimize the loss function of the target detection model. By adjusting the model parameters, the target detection model is optimized, and finally a trained target detection model is obtained;

[0009] (3) Using the trained target detection model to obtain pedestrian and face detection results, the unique target matching method is used to perform one-to-one matching between faces and pedestrians to filter out pedestrians without faces, ensuring that each retained pedestrian has unique matching face information;

[0010] (4) Construct a person re-identification model and use an open dataset to train and optimize the model. During the training process, the parameters of the person re-identification model are adjusted with the goal of minimizing the loss function of the person re-identification model to optimize the model and finally obtain a trained person re-identification model.

[0011] (5) Use the trained person re-identification model to perform identity matching on the pedestrians retained in step (3), accurately identify the target person, and output their facial image.

[0012] Furthermore, the target detection model includes a backbone network, a neck network and a detection head network, wherein the backbone network includes an input layer, a multi-space to depth module, a C2f module and an SPPF module, the multi-space to depth module includes a separation layer, a convolution module, a splicing layer and an SPPCSPC module, the convolution module includes a two-dimensional convolution layer, a batch normalization layer and a SiLU layer, the SPPCSPC module includes a two-dimensional convolution layer, a two-dimensional maximum pooling layer and a splicing layer, the C2f module includes a convolution module, a separation layer, a DarknetBottleneck module and a splicing layer, the DarknetBottleneck module includes a convolution module, the SPPF module includes a convolution module, a two-dimensional maximum pooling layer and a splicing layer, the neck network includes an upsampling layer, a splicing layer, a C2f module and a multi-space to depth module, and the detection head network includes three output branches, respectively used for target detection at three scales, and each output branch is divided into two parts, respectively used to obtain target categories and target coordinates;

[0013] The video frame image is input into the target detection model. After passing through the backbone network, neck network and detection head network, the detected target category and its corresponding target coordinates are obtained.

[0014] Furthermore, the target detection model is trained and optimized using the training set. During the training process, minimizing the loss function of the target detection model is used as the optimization goal. By adjusting the model parameters, the target detection model is optimized, and finally a trained target detection model is obtained. Specifically, the following steps are performed:

[0015] The video frame samples in the training set are input into the target detection model to obtain the detected target category and its corresponding target coordinates; the cross entropy is calculated as the classification loss based on the detected target category and the corresponding true category in the training set, and the intersection-over-union loss and coordinate offset loss are calculated as the bounding box loss based on the detected target coordinates and the corresponding true bounding box in the training set. The sum of the classification loss and the bounding box loss is used as the loss function of the target detection model; with minimizing the loss function of the target detection model as the optimization goal, the parameters of the target detection model are adjusted until the preset training rounds are reached to obtain a trained target detection model.

[0016] Furthermore, the unique target matching method is specifically implemented by the following method:

[0017] For each pedestrian detection result, the pedestrian detection frame is horizontally divided into six equal parts, and the first sub-frame represents the position of the head; the intersection area between the face detection frame and the head sub-frame in the face detection result is calculated, and the face corresponding to the maximum value of multiple intersection areas that meet the condition that the intersection area is greater than the preset threshold is selected as the unique matching face of the pedestrian; and a one-to-one match is performed based on the pedestrian coordinates in the pedestrian detection result and the coordinates of each face in the face detection result, and pedestrians without faces are filtered out to ensure that each retained pedestrian has unique matching face information.

[0018] Furthermore, the pedestrian re-identification model includes an input layer, a dynamic deformable fusion module, a basic feature extraction network, a batch normalization layer BNNeck and an output layer; wherein, the dynamic deformable fusion module includes a first branch and a second branch, the first branch includes a deformable convolution, the second branch includes a separation layer and a dynamic snake-shaped convolution layer, the input image of the dynamic deformable fusion module obtains a global feature map through the deformable convolution of the first branch, the input image of the dynamic deformable fusion module is cropped into four image slices through the separation layer of the second branch, the four image slices are respectively passed through the dynamic snake-shaped convolution layer to obtain local feature maps, the global feature map and the four local feature maps are spliced ​​to obtain the final feature map output by the dynamic deformable fusion module; the basic feature extraction network selects a variety of deep convolutional neural network structures to achieve effective image representation and feature extraction.

[0019] Furthermore, the loss function of the pedestrian re-identification model includes triplet loss, center loss, identity recognition loss, and label smoothing loss.

[0020] A second aspect of an embodiment of the present invention provides an apparatus for a pedestrian retrieval-driven specific face detection method, comprising:

[0021] The surveillance video acquisition module is used to collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, all pedestrian targets in it are annotated, and faces that meet predetermined requirements are marked to construct a training set;

[0022] The model building and training module is used to build an object detection model based on the improved YOLOv8 and a pedestrian re-identification model. The training set is used to train and optimize the object detection model. During the training process, the optimization goal is to minimize the loss function of the object detection model. By adjusting the model parameters, the object detection model is optimized and the trained object detection model is finally obtained. The open dataset is used to train and optimize the pedestrian re-identification model. During the training process, the optimization goal is to minimize the loss function of the pedestrian re-identification model. The parameters of the pedestrian re-identification model are adjusted to optimize the pedestrian re-identification model and the trained pedestrian re-identification model is finally obtained.

[0023] Real-time object detection module, which uses the trained object detection model to obtain pedestrian and face detection results;

[0024] The unique target matching module is used to perform one-to-one matching between faces and pedestrians using the unique target matching method to filter out pedestrians that do not contain faces and ensure that each retained pedestrian has unique matching face information;

[0025] The target person detection module is used to use the trained pedestrian re-identification model to match the identity of the pedestrians retained in the unique target matching module, accurately identify the target person, and extract and output their facial image.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) When the pedestrian in the search image is in a non-frontal state such as back or side view, the present invention can track and detect the target object in continuous cross-camera surveillance video and accurately capture its facial information at the moment when it faces the camera. This is achieved by modifying the two-stage pedestrian retrieval method.

[0028] (2) The present invention can retrieve the pedestrian object in real time in continuous monitoring video based on the non-frontal state of the retrieval object, such as the back or side face, and capture and return its facial information when it is in the frontal state.

[0029] (3) In the field of video surveillance target detection, pedestrians and faces constitute typical cross-scale detection objects. In order to enhance the accuracy of face detection, this paper designs a new multi-space to depth (MTSD) module to replace the original downsampling module of YOLOv8n, so as to reduce the information loss caused by downsampling.

[0030] (4) After detecting faces and pedestrian targets, the present invention designs a unique target matching method (UTMA) to achieve one-to-one matching between detected pedestrians and faces, and filters out pedestrians that do not contain faces, thereby improving the inference speed of specific face detection.

[0031] (5) The present invention designs a dynamic deformable fusion (DDF) module to enhance the ability of the pedestrian re-identification BOT network to capture deformation features to cope with complex video scenes under different camera perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is an architectural flow chart of a pedestrian retrieval-driven specific face detection method of the present invention;

[0033] Figure 2 It is a network structure diagram of the YOLOv8-MSTD target detection model of the present invention;

[0034] Figure 3 It is a streamline diagram of the BOT network structure of the present invention embedded in the dynamic deformable fusion DDF module;

[0035] Figure 4 It is a network structure diagram of the dynamic deformable fusion DDF module of the present invention. DETAILED DESCRIPTION

[0036] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0037] The terms used in this invention are for the purpose of describing specific embodiments only and are not intended to limit the invention. The singular forms "a," "the," and "the" used in this invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0038] It should be understood that although the terms "first," "second," "third," etc. may be used in the present invention to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present invention. Depending on the context, the term "if" as used herein may be interpreted as "when," "when," or "in response to determining."

[0039] The present invention will be described in detail below with reference to the accompanying drawings. Unless there is any conflict, the features of the following embodiments and implementations may be combined with each other.

[0040] See also Figure 1 The pedestrian retrieval driven specific face detection method of the present invention specifically includes the following steps:

[0041] (1) Collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, mark all pedestrian targets and mark the faces that meet the predetermined requirements to construct a training set.

[0042] Among them, each sample in the training set includes a video frame and its corresponding pedestrian label and face label.

[0043] Specifically, surveillance video data is first collected from key urban locations, including commercial and residential areas, transportation hubs, and public spaces. Road surveillance video data is collected and randomly sampled, with 5,000 frames containing varying numbers of pedestrians. These frames require no preprocessing, as long as the imagery is sufficiently rich, with a sufficient number of scenes and characters. For each frame, standardized annotation tools and a rigorous quality control process are used to accurately label all pedestrians and carefully label faces that meet predefined requirements. Based on this, a training set is constructed for training the object detection model. Each frame in this training set includes two labels: a pedestrian label and a face label, with location boxes added for each of these two types of objects.

[0044] (2) Construct a target detection model based on the improved YOLOv8, and use the training set to train and optimize the target detection model. During the training process, the optimization goal is to minimize the loss function of the target detection model. By adjusting the model parameters, the target detection model is effectively optimized, and finally a trained target detection model is obtained.

[0045] In this embodiment, a target detection model is constructed based on YOLOv8. It is consistent with YOLOv8. The main part still uses CSPDarknet53 as the feature extraction network. In the entire network architecture, except for the C2f module and the detection head network, all standard convolution modules are innovatively replaced by multi-space to depth (MSTD) modules. In the field of video surveillance target detection, pedestrians and faces constitute typical cross-scale detection objects, that is, pedestrian detection and face detection are cross-scale detection problems. For this reason, an MSTD module is designed in the present invention to replace the original downsampling convolution part in YOLOv8, thereby minimizing information loss. The YOLOv8-MSTD network is defined as a target detection model, and its structure is as follows: Figure 2 shown.

[0046] See also Figure 2The target detection model includes a backbone network, a neck network and a detection head network, wherein the backbone network includes an input layer, an MSTD module, a C2f module and an SPPF module; the MSTD module includes a separation layer, a convolution module, a splicing layer and an SPPCSPC module; the convolution module includes a two-dimensional convolution layer, a batch normalization (BatchNorm2d) layer and a SiLU layer; the SPPCSPC module includes a two-dimensional convolution layer, a two-dimensional maximum pooling layer and a splicing layer; the C2f module includes a convolution module, a separation layer, a DarknetBottleneck module and a splicing layer; the DarknetBottleneck module includes a convolution module; the SPPF module includes a convolution module, a two-dimensional maximum pooling layer and a splicing layer; the neck network includes an upsampling layer, a splicing layer, a C2f module and an MSTD module; the detection head network includes three output branches, which are used for target detection of different scales (large, medium and small). Each output branch is divided into two parts, which are used for classification and regression respectively, namely, the target category branch and the target coordinate branch, which are used to obtain the target category and target coordinates respectively. The video frame image is input into the target detection model, and then Figure 2 After the backbone network, neck network, and detection head network in the object detection model shown, the detected object category and its corresponding target coordinates are obtained. For example, the pedestrian category is 0 and the face category is 1. If the detected object category is 0 and its target coordinates are (15, 23, 32, 78), the final output is (0, 15, 23, 32, 78). Therefore, the object detection model can detect pedestrians and faces.

[0047] Among them, the MSTD module consists of a space-to-depth structure and an original downsampling convolution module, and is finally output through the SPPCSPC module, which will be described in detail below.

[0048] In the MSTD module, the input intermediate feature map X is of size H×W×C in , H represents the height of the feature, W represents the width of the feature, C in The number of channels representing the original features. First, the four input copies are split into four Sub-feature map, f 0,0 , f 0,1 , f 1,0 , f 1,1 The calculation is as follows:

[0049] f 0,0 =X[0:H:2,0:W:2,:]

[0050] f 0,1 =X[0:H:2,1:W:2,:]

[0051] f 1,0=X[1:H:2,0:W:2,:]

[0052] f 1,1 =X[1:H:2,1:W:2,:]

[0053] This separation operation can reduce the spatial dimension of the feature and increase the channel dimension of the feature without losing the learnable information. The resulting 4 dimensions are In addition, the MSTD module also applies the original down-sampling convolution module to the intermediate feature map X of the original input to obtain a global down-sampling feature map of size Finally, the five sub-features are spliced ​​together to obtain The feature map Y of the same size is obtained. This feature map retains all the original information of the intermediate features and also retains the global downsampling information. In order to achieve the mapping fusion of local features and global features, the feature map Y is further output by the SPPCSPC module, and the obtained result is the input of the next module.

[0054] In this embodiment, a training set is used to train and optimize the target detection model. During the training process, minimizing the loss function of the target detection model is used as the optimization goal. By adjusting the model parameters, effective optimization of the target detection model is achieved, and finally a trained target detection model is obtained. Specifically, when using the training set to train the target detection model, the video frame samples in the training set are input into the target detection model to obtain the detected target category and its corresponding target coordinates; the cross entropy is calculated as the classification loss based on the detected target category and the corresponding true category in the training set, and the intersection-over-union loss and coordinate offset loss are calculated as the bounding box loss based on the detected target coordinates and the corresponding true bounding box in the training set, and the sum of the classification loss and the bounding box loss is used as the loss function of the target detection model; minimizing the loss function of the target detection model is used as the optimization goal, and the parameters of the target detection model are adjusted until a preset training round is reached to obtain a trained target detection model.

[0055] It should be understood that the calculation formulas for cross entropy, intersection-over-union loss, and coordinate offset loss are commonly used loss functions during YOLOv8 training and will not be repeated here.

[0056] (3) The trained target detection model is used to detect pedestrians and faces, and the pedestrian and face detection results are obtained. After detecting the faces and pedestrian targets, the unique target matching method (UTMA) is used to perform a one-to-one match between faces and pedestrians to filter out pedestrians that do not contain faces, ensuring that each retained pedestrian has unique matching valid face information.

[0057] Furthermore, the unique target matching method is implemented as follows: for each person detection result, the pedestrian detection box is horizontally divided into six equal parts, with the first sub-box representing the position of the head. The intersection area between the face detection box and the head sub-box in the face detection result is calculated, and the face corresponding to the maximum of the multiple intersection areas, where the intersection area exceeds a preset threshold, is selected as the unique matching face for the pedestrian. A one-to-one match is then performed between the pedestrian coordinates in the pedestrian detection result and the coordinates of each face in the face detection result, filtering out pedestrians without faces. This ensures that each retained pedestrian has unique matching face information. The unique target matching method reduces subsequent computational effort and achieves unique matching between pedestrians and faces. After the captured video frame is processed by the YOLOv8-MSTD object detection model, nearly all pedestrians and faces meeting the detection requirements are captured. However, to reduce the load on the subsequent person re-identification model, UTMA filters out pedestrians that meet the requirements and performs one-to-one matching between detected pedestrians and faces.

[0058] It should be understood that a face should be located on top of a pedestrian. That is, the top coordinates of the face should coincide with the top coordinates of the pedestrian. If any coordinates do not match, the face and pedestrian are not matched. Therefore, a one-to-one match can be achieved based on the pedestrian and face coordinates. UTMA is more directional and helpful for target population samples. After the first stage of network processing, pedestrians and faces are detected in the surveillance video. However, not all detected pedestrians need to be identified; only those with faces are sent to the second stage for identification. Pedestrians without faces lack recognition value, wasting computational resources and slowing down the model. To uniquely identify pedestrian faces, UTMA is designed and applied to target matching between faces and pedestrians. Taking into account the proportional relationship between head and height, the human detection box is horizontally divided into six equal parts. The first subbox represents the location of the head. The intersection area between the face detection box and the head subbox is then calculated. This value is called the IOF. The IOF value of a face must be greater than 80%. If multiple faces meet this condition, the face with the largest IOF is considered the only face.

[0059] (4) Construct a pedestrian re-identification model and use the open dataset Market1501 or CUHK01 to train and optimize the pedestrian re-identification model. During the training process, the optimization goal is to minimize the loss function of the pedestrian re-identification model, adjust the parameters of the pedestrian re-identification model, achieve effective optimization of the pedestrian re-identification model, and finally obtain the trained pedestrian re-identification model.

[0060] In this embodiment, in the second stage, the Bag of Tricks (BOT) is used to build a pedestrian re-identification model, taking into account the lightweight and performance of the model. In addition, in order to achieve a lightweight pedestrian re-identification model and fully utilize the advantages of the multi-branch structure, a lightweight multi-branch dynamic deformable fusion (DDF) module is designed, and the DDF module is embedded in the BOT as a pedestrian re-identification model. The DDF module can not only normalize the deformation features of various human postures, but also reduce the influence of limb swing deformation, thereby further enhancing the global features of BOT. Similar to BOT, the BOT-DDF pedestrian re-identification model uses ResNet50 as the basic feature extraction network and combines multiple optimization strategies to build a high-performance pedestrian re-identification benchmark model.

[0061] like Figure 3 As shown in , the pedestrian re-identification model includes an input layer, a DDF module, a basic feature extraction network, a batch normalization layer BNNeck and an output layer. Among them, the DDF module includes a first branch and a second branch. The first branch includes a deformable convolution, and the second branch includes a separation layer and a dynamic snake convolution layer. The input image of the DDF module obtains a global feature map through the deformable convolution of the first branch. The input image of the DDF module is cropped into four image slices through the separation layer of the second branch. The four image slices are respectively passed through the dynamic snake convolution layer to obtain local feature maps. The global feature map and the four local feature maps are spliced ​​to obtain the final feature map output by the DDF module, as shown in Figure 4 As shown in the figure, the basic feature extraction network uses the ResNet50 network. The batch normalization layer (BNNeck) is a batch normalization (BN) layer inserted between the backbone network and the classifier head of the model, usually before the output layer.

[0062] Among them, the network structure of the DDF module is as follows Figure 4As shown. In the DDF module, 3×64×128 images are used as input, and these images are processed separately by two sub-branches. In the first branch, deformable convolution is applied to the image, so as to focus more on the complex deformations that may exist in pedestrians in the surveillance video, and obtain a global feature map of size 64×32×64. In the second branch, dynamic snake convolution is introduced to improve the ability to extract complex information such as edges and details in the image. First, the input image is cropped into four image slices of size 3×32×64 through the separation operation of the separation layer. Subsequently, these four image slices are respectively subjected to the dynamic snake convolution layer to obtain local feature maps of size 16×32×64. Finally, the five feature maps obtained by the above two branches are spliced ​​to obtain a feature map of size 128×32×64, thereby further enhancing the global feature representation of the individual. The above DDF module can be formulated as:

[0063] F DDF (x) = η(cat(o 3×3 ([x]),s 3×3 ([x[…,0:H / 2,0:W / 2]]),s 3×3 ([x[…,0:H / 2,W / 2:W]]),

[0064] s 3×3 ([x[…,H / 2:H,0:W / 2]]),s 3×3 ([x[…,H / 2:H,W / 2:W]]),1))

[0065] Where, F DDF (x) represents the output of the DDF module; x is the input image; η is the Leaky Relu activation function; o 3×3 Indicates a deformable convolution operation with a convolution kernel of 3×3; S 3×3 represents a dynamic snake convolution operation with a convolution kernel of 3×3; cat represents a concatenation operation. The deformable convolution in the first branch is defined as:

[0066]

[0067] Where O(f0) represents the output of the deformable convolution in the first branch, Δf n Represents a learnable offset, which is calculated by an additional offset learning branch, and the sampling position is f0+f n Change to f0+f n +Δf n , so that the convolution kernel can be more adaptable to the deformation of the target. n +Δf n The coordinates may be non-integer, so the bilinear interpolation represented by the following formula is introduced to calculate the pixel value at the corresponding position:

[0068]

[0069] Where f represents the position of any sampling point, I(f) represents the pixel value at the sampling point f; q traverses all complete spatial positions in the feature map I; B represents the bilinear interpolation kernel. The dynamic snake convolution in the second branch can be defined in the x-axis direction as:

[0070]

[0071] In the formula, (x i ,y i ) represents the center of the convolution kernel, x i -c means the horizontal coordinate is offset from the center to the left by c units, i +c means the horizontal axis is shifted to the right by c units, and Δy means the offset that can be learned or dynamically predicted. Represents the sum of all vertical offsets from position i to position i+c. From the above formula, we can see that in dynamic snake convolution, when the convolution kernel "snakes" in the x direction, the y direction will also be adaptively fine-tuned according to local image features, so that the sampling point can better fit the target shape or edge.

[0072] Furthermore, Market1501 and CUHK01 are both widely used benchmark datasets in the field of person re-identification (ReID), in which the image identifiers of the same person are the same. When using the dataset Market1501 or CUHK01 to train a person re-identification model, pedestrian samples in the dataset Market1501 or CUHK01 are input into the person re-identification model, and the recognized face information is obtained after passing through the input layer, DDF module, basic feature extraction network, batch normalization layer BNNeck, and output layer in sequence. Based on the recognized face information and the corresponding labels in the dataset Market1501 or CUHK01, the triplet loss, center loss, identity recognition loss, and label smoothing loss are calculated. Then, the loss function of the person re-identification model is calculated based on the triplet loss, center loss, identity recognition loss, and label smoothing loss. During the training process, the parameters of the person re-identification model are adjusted with the goal of minimizing the loss function of the person re-identification model until the preset training rounds are reached to obtain a trained person re-identification model.

[0073] It should be understood that triplet loss, center loss, identity loss, and label smoothing loss are commonly used loss functions in person re-ID tasks, and they can work together. The triplet loss learns features by comparing the relationship between anchors, positive samples, and negative samples. Its goal is to make the distance between features of the same ID smaller than that of features of different IDs. The center loss maintains a class center (feature mean) for each identity. Its goal is to minimize the distance between sample features and their class center. During training, the center needs to be dynamically updated and is often combined with the cross-entropy loss (λ balancing weight). The person re-ID task is considered a classification problem, with each pedestrian ID treated as an independent class, to calculate the identity loss. The label smoothing loss can improve on traditional one-hot labeling and prevent overconfidence in the model. It adjusts the probability of the true label from 1.0 to a slightly smaller value (such as 0.9), and evenly distributes the remaining probability among other classes.

[0074] (5) Use the trained person re-identification model to perform identity matching on the pedestrians retained in step (3), accurately identify the target person, and output their facial image.

[0075] It is worth mentioning that an embodiment of the present invention also provides a pedestrian retrieval-driven specific face detection device, which is used to implement the pedestrian retrieval-driven specific face detection method in the above embodiment. The device includes a surveillance video acquisition module, a model building and training module, a real-time object detection module, a unique target matching module and a target person detection module. Among them, in the first stage, the surveillance video acquisition module and the real-time object detection module are used to collect on-site data and preliminarily identify key targets, providing a solid foundation for subsequent data processing and analysis. In the second stage, the unique target matching module and the target person detection module are used to filter the detected candidate targets to ensure that each retained pedestrian has unique matching valid face information, and the pedestrian re-identification model is used to achieve accurate retrieval of pedestrians, and the standardized facial data of the retrieved pedestrians is output.

[0076] In this embodiment, the surveillance video acquisition module is used to collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, all pedestrian targets in it are labeled, and faces that meet predetermined requirements are marked to construct a training set.

[0077] In this embodiment, the model construction and training module is used to construct a target detection model based on the improved YOLOv8 and a pedestrian re-identification model; the target detection model is trained and optimized using a training set. During the training process, the optimization goal is to minimize the loss function of the target detection model. By adjusting the model parameters, the target detection model is effectively optimized, and finally a trained target detection model is obtained; the open data set Market1501 or CUHK01 is used to train and optimize the pedestrian re-identification model. During the training process, the optimization goal is to minimize the loss function of the pedestrian re-identification model. The parameters of the pedestrian re-identification model are adjusted to achieve effective optimization of the pedestrian re-identification model, and finally a trained pedestrian re-identification model is obtained.

[0078] In this embodiment, the real-time object detection module is used to obtain pedestrian and face detection results using the trained target detection model.

[0079] In this embodiment, the unique target matching module is used to perform one-to-one matching between faces and pedestrians using a unique target matching method to filter out pedestrians that do not contain faces, ensuring that each retained pedestrian has unique matching facial information.

[0080] In this embodiment, the target person detection module is used to use the trained pedestrian re-identification model to perform identity matching on the pedestrians retained in the unique target matching module, accurately identify the target person, and extract and output his or her facial image.

[0081] This invention combines a two-stage improved model with a unique target matching algorithm to achieve a new feature for capturing the faces of specific pedestrians. In actual surveillance video scenarios, when searching for non-frontal faces, such as those from the back or side, the algorithm achieved a mean average performance (MAP) of 88.4% and a rank-1 score of 95.5%, effectively reducing the workload of employees in practical applications.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A pedestrian retrieval driven specific face detection method, characterized in that: The following steps are involved: (1) Collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, mark all pedestrian targets and mark faces that meet the predetermined requirements to construct a training set; (2) Build a target detection model based on the improved YOLOv8, and use the training set to train and optimize the target detection model. During the training process, the optimization goal is to minimize the loss function of the target detection model. By adjusting the model parameters, the target detection model is optimized, and finally a trained target detection model is obtained; (3) Using the trained target detection model to obtain pedestrian and face detection results, the unique target matching method is used to perform one-to-one matching between faces and pedestrians to filter out pedestrians without faces, ensuring that each retained pedestrian has unique matching face information; (4) Construct a person re-identification model and use an open dataset to train and optimize the model. During the training process, the parameters of the person re-identification model are adjusted with the goal of minimizing the loss function of the person re-identification model to optimize the model and finally obtain a trained person re-identification model. (5) Use the trained person re-identification model to perform identity matching on the pedestrians retained in step (3), accurately identify the target person, and output their facial image.

2. The pedestrian retrieval driven specific face detection method according to claim 1, characterized in that: The target detection model includes a backbone network, a neck network and a detection head network, wherein the backbone network includes an input layer, a multi-space to depth module, a C2f module and an SPPF module, the multi-space to depth module includes a separation layer, a convolution module, a splicing layer and an SPPCSPC module, the convolution module includes a two-dimensional convolution layer, a batch normalization layer and a SiLU layer, the SPPCSPC module includes a two-dimensional convolution layer, a two-dimensional maximum pooling layer and a splicing layer, the C2f module includes a convolution module, a separation layer, a DarknetBottleneck module and a splicing layer, the DarknetBottleneck module includes a convolution module, the SPPF module includes a convolution module, a two-dimensional maximum pooling layer and a splicing layer, the neck network includes an upsampling layer, a splicing layer, a C2f module and a multi-space to depth module, and the detection head network includes three output branches, which are used for target detection at three scales respectively, and each output branch is divided into two parts, which are used to obtain target categories and target coordinates respectively; The video frame image is input into the target detection model. After passing through the backbone network, neck network and detection head network, the detected target category and its corresponding target coordinates are obtained.

3. The pedestrian retrieval driven specific face detection method according to claim 1, characterized in that: The target detection model is trained and optimized using the training set. During the training process, the target detection model is optimized by minimizing the loss function of the target detection model. The model parameters are adjusted to achieve the target detection model optimization. Finally, the trained target detection model is obtained. Specifically, the following steps are performed: The video frame samples in the training set are input into the target detection model to obtain the detected target category and its corresponding target coordinates; the cross entropy is calculated as the classification loss based on the detected target category and the corresponding true category in the training set, and the intersection-over-union loss and coordinate offset loss are calculated as the bounding box loss based on the detected target coordinates and the corresponding true bounding box in the training set. The sum of the classification loss and the bounding box loss is used as the loss function of the target detection model; with minimizing the loss function of the target detection model as the optimization goal, the parameters of the target detection model are adjusted until the preset training rounds are reached to obtain a trained target detection model.

4. The pedestrian retrieval driven specific face detection method according to claim 1, characterized in that: The unique target matching method is specifically implemented by the following method: For each pedestrian detection result, the pedestrian detection frame is horizontally divided into six equal parts, and the first sub-frame represents the position of the head; the intersection area between the face detection frame and the head sub-frame in the face detection result is calculated, and the face corresponding to the maximum value of multiple intersection areas that meet the condition that the intersection area is greater than the preset threshold is selected as the unique matching face of the pedestrian; and a one-to-one match is performed based on the pedestrian coordinates in the pedestrian detection result and the coordinates of each face in the face detection result, and pedestrians without faces are filtered out to ensure that each retained pedestrian has unique matching face information.

5. The pedestrian retrieval driven specific face detection method according to claim 1, characterized in that: The pedestrian re-identification model includes an input layer, a dynamic deformable fusion module, a basic feature extraction network, a batch normalization layer BNNeck and an output layer; wherein, the dynamic deformable fusion module includes a first branch and a second branch, the first branch includes a deformable convolution, and the second branch includes a separation layer and a dynamic snake convolution layer. The input image of the dynamic deformable fusion module obtains a global feature map through the deformable convolution of the first branch, and the input image of the dynamic deformable fusion module is cropped into four image slices through the separation layer of the second branch. The four image slices are respectively passed through the dynamic snake convolution layer to obtain local feature maps. The global feature map and the four local feature maps are spliced ​​to obtain the final feature map output by the dynamic deformable fusion module; the basic feature extraction network selects a variety of deep convolutional neural network structures to achieve effective image representation and feature extraction.

6. The pedestrian retrieval driven specific face detection method according to claim 1, characterized in that: The loss function of the pedestrian re-identification model includes triplet loss, center loss, identity recognition loss, and label smoothing loss.

7. A device for implementing the pedestrian retrieval driven specific face detection method according to any one of claims 1 to 6, characterized in that: include: The surveillance video acquisition module is used to collect road surveillance video data and extract multiple video frames from it using a random sampling method. For each video frame, all pedestrian targets in it are annotated, and faces that meet predetermined requirements are marked to construct a training set; Model building and training module, used to build an object detection model based on improved YOLOv8 and a pedestrian re-identification model; The target detection model is trained and optimized using the training set. During the training process, the optimization goal is to minimize the loss function of the target detection model. By adjusting the model parameters, the target detection model is optimized and finally a trained target detection model is obtained. Use open datasets to train and optimize the person re-identification model. During the training process, minimize the loss function of the person re-identification model as the optimization goal, adjust the parameters of the person re-identification model, optimize the person re-identification model, and finally obtain the trained person re-identification model. Real-time object detection module, which uses the trained object detection model to obtain pedestrian and face detection results; The unique target matching module is used to perform one-to-one matching between faces and pedestrians using the unique target matching method to filter out pedestrians that do not contain faces and ensure that each retained pedestrian has unique matching face information; The target person detection module is used to use the trained pedestrian re-identification model to match the identity of the pedestrians retained in the unique target matching module, accurately identify the target person, and extract and output their facial image.

Images