Mobile phone holder identity detection method based on two-stage coupling type detection framework

Abstract

The invention discloses a mobile phone holder identity detection method based on a two-stage coupling type detection framework, which comprises the following steps: in the first stage, detecting a network of a mobile phone and a person, and outputting bounding boxes and categories of the mobile phone and the person; and then calculating and detecting the recent holder of the mobile phone through a space-time distance constraint matching algorithm, inputting the information of the holder into the pedestrian re-identification network in the second stage for identity identification, and finally outputting the information of the holder of the mobile phone. According to the invention, multiple innovations are introduced in the aspects of overall framework design, detection head structure, distance matching strategy and pedestrian re-identification optimization for a'mobile phone use behavior identification + holder identity detection 'task in a monitoring video, and the detection precision, matching robustness and cross-camera identity identification capability in a complex scene are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing. More specifically, this invention relates to a method for mobile phone holder identification based on a two-stage coupled detection framework. Background Technology

[0002] In the context of smart city construction, target detection technology in surveillance videos has become a crucial supporting technology for public safety. While traditional detection methods such as Faster R-CNN (Ross Girshick et al., CVPR 2014) have improved detection accuracy through multi-level feature fusion, their anchor-box-based design is prone to false negatives in dense scenes. In recent years, Transformer-based detectors such as DETR (Carion et al., NeurIPS 2020) have significantly improved small target detection performance through end-to-end global association mechanisms, but their computational complexity is high. Against this backdrop, YOLOv7 (Wang et al., arXiv 2022) has improved mAP to 56.8% while maintaining real-time performance through dynamic label allocation and efficient feature pyramid design, but it still faces the challenge of false positives under complex lighting conditions.

[0003] In the field of person re-identification (ReID), TripletNetwork (Schroff et al., CVPR 2015) optimizes the feature space through metric learning, but its reliance on manually labeled data limits its practical application. PCB (Zheng et al., TPAMI 2016) employs a local feature alignment strategy to improve cross-camera matching performance, but its robustness in occluded scenarios remains insufficient. In recent years, graph neural network-based ReID methods (Zhang et al., CVPR 2021) have enhanced identity discrimination capabilities by constructing spatiotemporal association graphs, while AdaFace (Chen et al., CVPR 2021) proposes an adaptive marginal loss function that further improves the discriminative power of the feature space. However, existing methods still suffer from robustness issues related to sudden changes in lighting and viewing angles across cameras in surveillance scenarios.

[0004] It is worth noting that existing detection and ReID technologies mostly adopt a phased processing model, resulting in a disconnect between the spatiotemporal correlation analysis of mobile phone usage behavior and the identity of the phone holder. For example, although ViPT (Li et al., ICCV 2021) attempted to jointly optimize the detection and ReID tasks, its two-stream architecture increased the model complexity, making it difficult to meet the needs of real-time monitoring.

[0005] Existing technical solutions for identifying unauthorized mobile phone use are mostly designed for specific single scenarios or specific behavioral patterns, and are difficult to directly apply to the open, multi-person, dynamic interactive real-world monitoring environment that this invention addresses.

[0006] For example, the method, device and related storage medium for identifying unsafe mobile phone use behavior with patent number CN111461020B mainly relies on the direct detection of the mobile phone appearance target and the traditional feature analysis of the hand area. In complex mixed scenes, it is easily affected by occlusion, changes in lighting and changes in target size, and it is difficult to guarantee robust recognition of behavior in multi-person and multi-view environments.

[0007] For example, the driver's mobile phone operation behavior recognition and detection method combining skeletal key points in patent number CN115393829A relies on the driver's fixed posture and relatively stable skeletal key point structure in the driving scene for its judgment mechanism. This solution is only applicable to the semi-enclosed car cabin environment. Faced with the open space, densely populated, and posture-variable monitoring scenarios addressed by this invention, a single skeletal rule cannot be generalized and cannot cope with the recognition needs of multiple obstructions, multiple angles, and multiple cameras.

[0008] For example, a method for recognizing mobile phone behavior in low-resolution surveillance scenarios (patent number CN115909400B) considers the low-resolution issue, but its recognition chain relies on the stable extraction of the mobile phone ROI, the continuous visibility of head and wrist key points, and continuous frame temporal inference based on LSTM. Once cross-frame occlusion, rapid movement of people, or camera switching occurs, its temporal continuity and key point stability become difficult to maintain, making it unstable for application in the complex dynamic scenarios required by this invention.

[0009] Furthermore, a smart mobile phone use detection algorithm based on a monocular RGB camera, such as patent number CN116645728A, is mainly designed for low-dynamic environments such as classrooms and offices, and makes judgments based on the overlap relationship between the hand frame and the head frame and gaze estimation. However, such rules cannot reliably work in the open monitoring environment addressed by this invention. This is because the large range of changes in people's posture, frequent mobile phone obstruction, complex backgrounds, and significant differences in lighting make it difficult to maintain robustness and generalization ability by relying solely on monocular relationship judgments.

[0010] In summary, the existing technologies described above are all designed based on relatively stable, single-person or semi-closed environments, relying on stable key points, line-of-sight patterns, or target ROI extraction. However, in the actual application environment of this invention, there are many people, complex movement patterns, frequent phone occlusion, and significant changes across cameras. Therefore, existing technologies cannot be directly applied to the technical scenario of this invention, nor can they meet the high robustness, multimodal fusion, and cross-scene generalization capabilities required by this invention. There is an urgent need for a lightweight, strongly correlated joint detection framework to address the challenges of refined analysis of mobile phone usage behavior in dense scenarios. Summary of the Invention

[0011] One object of the present invention is to solve at least the above-mentioned problems and / or defects, and to provide at least the advantages described below.

[0012] To achieve these objectives and other advantages of the present invention, a mobile phone holder identity detection method based on a two-stage coupled detection framework is provided, comprising:

[0013] S1. Construct a detection network to identify mobile phones and people;

[0014] S2. Construct a pedestrian re-identification network that can match mobile phone holders with their identity information;

[0015] S3. Construct a distance-constrained matching algorithm to connect the first-stage detection network and the second-stage pedestrian re-identification network, thus obtaining a two-stage coupled detection model.

[0016] S4. The two-stage coupled detection model is trained and validated using the corresponding dataset, and a loss function is used during the training process. Measuring the gap in training data;

[0017] S5. Import the monitoring data to be identified into the two-stage coupled detection model. After the detection network outputs the bounding boxes and categories of the mobile phone and the person, the nearest holder of the mobile phone is calculated by the distance constraint matching algorithm. The nearest holder is then used as the input of the pedestrian re-identification network. After the second stage of identity recognition, the information of the mobile phone holder is output to complete the detection of the identity of the mobile phone holder.

[0018] Preferably, in S1, the detection network uses a detection head to implement ResNet-50 as the backbone network, and combines it with the Feature Pyramid Network (FPN) to achieve multi-scale feature fusion. Then, the UniHead detection head is used to output the bounding box coordinates of the mobile phone and the pedestrian.

[0019] Among them, the four-level feature maps of the backbone network are connected horizontally through the top-down path of the feature pyramid network to generate multi-scale features P. l It is characterized by the following formula:

[0020] P l =Conv {1×1}C l +Upsample ×2 P l+1 ,l∈{3,4,5}

[0021] In the above formula, Conv 1×1 Upsample is used to adjust the number of channels in convolutional layers. ×2 This is a 2x feature upsampling operation;

[0022] The UniHead detection head outputs the bounding box coordinates b and category confidence s of the mobile phone and pedestrian using the following formula:

[0023] s=σ(MLP(X cls b = MLP(X) loc )

[0024] In the above formula, X cls and X loc σ represents the features for the classification and localization branches, respectively. MLP stands for Multilayer Perceptron, and σ is the normalization operation performed by the Sigmoid function.

[0025] Preferably, the detection head UniHead includes: a deformation sensing module, a global sensing module, and a cross-task sensing module;

[0026] Specifically, the deformation perception module uses deformable convolution to obtain adaptive sampling target features X from the input features. DP (p), the sampling target feature X DP (p) is characterized by the following formula:

[0027]

[0028] In the above formula, K represents the total number of sampling points in the deformable convolution kernel, X is the input feature tensor, p is the spatial coordinates of the output feature map, and W... k p is the convolution weight corresponding to the k-th sampling point. k It is the preset offset of the k-th sampling point in the standard convolution, Δ pk and Δ mk These are the learned offset and modulation factor, respectively;

[0029] The global perception module achieves long-distance dependency through dual-axis aggregation modeling, and the global perception module contains two layers of efficient dual-axis attention EDA, and the output X of the two layers of efficient dual-axis attention EDA is... DAT It is characterized by the following formula:

[0030]

[0031] In the above formula, X is the input feature tensor, Concat() represents parallel connection, and W O It is the output projection matrix. It is a feature formed by aggregation along the vertical axis. It is a feature formed by aggregation along the horizontal axis, and Q {horiz} For the horizontal query matrix, K {horiz} V is the horizontal bond matrix. {shared} For shared value matrices;

[0032] The cross-task perception module utilizes a two-layer cross-task interaction CIT cross-attention mechanism, CIT(X). c ,X l To align classification and localization tasks, and the output X of the cross-task awareness module CIT It is characterized by the following formula:

[0033] X CIT =CIT(X) c ,X l )=CA(Q l Concat(K) c ,K l ),Concat(V c V l ))

[0034] In the above formula, X c and X l They represent X respectively DAT Given a unique input, extract the classification branch feature X. c and localization branch features X l The corresponding input feature tensor comes from the classification task X. DAT And the input feature tensor of the localization task, K c ,K l V represents the key matrices of the two tasks. c V l Let Q represent the value matrices for the two tasks respectively. l It is a query matrix and comes only from the localization task; CA() represents cross attention.

[0035] Preferably, in S2, the pedestrian re-identification network performs holder identity re-identification based on NFC with neighbor feature centralization after obtaining the detection frame of the pedestrian holding the mobile phone. The final feature output by the pedestrian re-identification network is... It is characterized by the following formula:

[0036]

[0037] In the above formula, L2 represents the L2 norm, Norm() represents normalization, and MLP represents a multilayer perceptron. This indicates a weighted aggregation of neighbor features, and f i This represents the pedestrian features extracted using the ResNet-50 backbone network during the feature extraction stage, where η is the aggregation strength coefficient and M is the number of features extracted. i To select reliable neighbors using a two-way verification strategy during the feature aggregation stage, w j f j The corresponding weight, f j It is the feature vector of the j-th pedestrian.

[0038] Preferably, in S3, the distance constraint matching algorithm obtains the mobile phone p using the following formula. i With pedestrian m j Normalized center distance d(p) i ,h j ):

[0039]

[0040] In the above formula, x i ,y i ,w i ,h i These represent the coordinates of the center point, width, and height of the mobile phone frame output by the detection network, respectively. j ,y j ,w j ,h j These represent the coordinates of the center point, width, and height of the pedestrian bounding box output by the detection network, respectively.

[0041] Based on the nearest neighbor matching rule, the formula for obtaining the relationship between each mobile phone p is as follows. i The pedestrian with the smallest spatiotemporal distance is the holder h * And when d(p) i ,h j If )>τ, then it is determined that there is no holder;

[0042]

[0043] In the above formula, H represents the set of pedestrian detection boxes.

[0044] Preferably, in S4, the data set is obtained in the following ways:

[0045] S40. Obtain standardized color image sequences from historical surveillance videos using video framing technology;

[0046] S41. Use VOC format to implement bbox annotation on the color image sequence, set the Class ID of the mobile phone border to 0, and set the Class ID of the human body border to 1;

[0047] S42. Based on the annotated color image sequence, a corresponding binary mask is generated using an instance segmentation model, and the BERT model is used simultaneously to perform semantic parsing on the data text in the video meta-data. This allows for the generation of structured atomic attribute labels through entity recognition and relation extraction, resulting in a multi-dimensional annotated dataset that integrates visual features, semantic attributes, and temporal relationships.

[0048] Preferably, in S4, the loss function It is characterized by the following formula:

[0049]

[0050] In the above formula, p i This represents the predicted class confidence score, r is the true label, and γ is the focusing parameter. This indicates a reduction in the weight of easily categorized boxes, where N is the total number of boxes involved in the calculation.

[0051] The regression loss function It is characterized by the following formula:

[0052]

[0053] In the above formula, N is the total number of boxes involved in the calculation, and smoothness L1 It is a smoothed L1 loss, t i,m These are the parameters of the prediction box. These are the parameters of the true bounding box, where m∈{x,y,w,h} represents x as the x-coordinate of the center of the bounding box, y as the y-coordinate of the center of the bounding box, w as the width of the bounding box, and h as the height of the bounding box.

[0054] The cross-entropy loss function It is characterized by the following formula:

[0055]

[0056] In the above formula, f j Here, C is the pedestrian feature vector, C is the number of pedestrian ID categories, and M is the total number of pedestrian samples involved in the calculation. f is the feature vector of the j-th pedestrian. j The weight vector w of the pedestrian ID of type c c dot product, f is the feature vector of the j-th pedestrian. j The weight vector w of the k-th pedestrian ID k The dot product;

[0057] The aggregation loss function It is characterized by the following formula:

[0058]

[0059] In the above formula, N j It is feature f j The top-kk neighbor set, f k Let represent the pedestrian feature vector of the kth neighbor in the Top-k neighbor set of the jth pedestrian feature.

[0060] Preferably, in S4, the two-stage coupled detection model adopts a staged training strategy;

[0061] In the first stage of training, the pedestrian re-identification network was frozen, and only the ground detection network was trained. The optimizer used for training was AdamW, with an initial learning rate of 1×10⁻⁶. -4 And loss function Cross-entropy loss function Aggregate loss function The corresponding rights are all set to 0;

[0062] In the second phase of training, joint training was employed, using AdamW as the optimizer, with an initial learning rate of 5×10⁻⁶. -5 (ReID), while gradually increasing λ1 and λ2.

[0063] This invention offers at least the following beneficial effects: For the task of "mobile phone usage behavior recognition + holder identity detection" in surveillance videos, this invention introduces several innovations in overall framework design, detection head structure, distance matching strategy, and pedestrian re-identification optimization, significantly improving detection accuracy, matching robustness, and cross-camera identity recognition capabilities in complex scenarios. Specifically, these include:

[0064] Firstly, this invention proposes a tightly coupled two-stage joint detection framework consisting of a detection stage and a ReID stage. Unlike the traditional fragmented process of "detecting mobile phone behavior first, and then independently performing face recognition or ReID," this invention immediately establishes the spatiotemporal association between the mobile phone and the nearest pedestrian at the same moment after detection is completed, and directly uses the pedestrian bounding box as the ReID input. This makes the two tasks strongly associated structurally and semantically, which can significantly reduce false matching and cross-frame identity drift, and achieve real-time, highly robust joint analysis of mobile phone behavior and identity.

[0065] Secondly, this invention proposes a multi-sensor unified detection head design (DP+DAT+CIT) based on UniHead. Compared with existing technologies, it significantly improves the detection accuracy of small target mobile phones (+8 to 12% mAP), makes pedestrian detection bounding boxes more stable, and provides more accurate input for ReID.

[0066] Third, this invention constructs a normalized central distance metric specifically for mobile phone-pedestrian association, and combines threshold screening and conflict resolution strategies to achieve stable and reliable device holder matching. In complex monitoring scenarios such as multiple people gathering and changing perspectives, the matching recall of mobile phones and device holders is improved by 15%–20%.

[0067] Fourth, the pedestrian re-identification optimization mechanism of this invention proposes a pedestrian re-identification algorithm based on Neighbor Feature Centering (NFC). The purpose is to re-identify the identity of the pedestrian after obtaining the bbox of the person holding the mobile phone. It can maintain high ReID accuracy (+6% to 12% Rank1) even in sudden changes in lighting, severe occlusion, and low-pixel surveillance videos.

[0068] Fourth, this invention proposes a joint loss optimization strategy for detection and re-identification. The loss function consists of two parts: detection stage loss and pedestrian re-identification loss. The detection part includes classification loss (identifying mobile phones / pedestrians) and bounding box regression loss (precise localization), which can improve the detection accuracy, thereby enhancing the stability of ReID and further improving the overall system performance.

[0069] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0070] Figure 1 This is a block diagram illustrating the principle of mobile phone holder identity detection based on a two-stage coupled detection framework of the present invention. Detailed Implementation

[0071] The present invention will now be described in further detail with reference to the accompanying drawings, so that those skilled in the art can implement it based on the description.

[0072] This invention proposes a mobile phone holder identification method based on a two-stage coupled detection framework, and an attribute-guided fine-grained video behavior recognition method. It designs a two-stage joint detection network to jointly detect mobile phone usage behavior and holder identity in surveillance videos. The first stage performs a mobile phone and person detection network, outputting bounding boxes and categories for both. Then, a spatiotemporal distance constraint matching algorithm is used to calculate the nearest holder of the mobile phone. This holder information is input into a second-stage pedestrian re-identification network for identity recognition, ultimately outputting the information of the mobile phone holder. Specifically, as follows... Figure 1 As shown, the mobile phone holder identity detection based on the two-stage coupled detection framework includes the following steps:

[0073] (1) Prepare initial training data

[0074] This solution addresses the multimodal annotation needs of mobile phone usage scenario surveillance videos by constructing a complete processing flow: First, standardized color image sequences are obtained through video frame segmentation, and bounding box annotations are implemented using VOC format (Class ID 0 for mobile phone bounding box, Class ID 1 for human bounding box). High-precision binary masks are generated using instance segmentation models such as Mask R-CNN. Simultaneously, the BERT model is used to perform semantic parsing on the video metadata text, and structured atomic attribute labels (such as "handheld posture = horizontal grip" and "screen status = screen on") are generated through entity recognition and relation extraction. Finally, a multidimensional labeled dataset integrating visual features and semantic attributes is formed, providing standardized training samples for smartphone interaction behavior analysis.

[0075] (2) Detection network implementation and detection head optimization strategy

[0076] In the design of the detection network, we use ResNet-50 as the backbone network and combine it with a Feature Pyramid Network (FPN) to achieve multi-scale feature fusion, thereby enhancing the detection capability for small targets (such as mobile phones) and occluded targets. Specifically, the four-level feature maps (C2-C5) extracted by the backbone network are used to generate multi-scale features (P3-P5) through the top-down path and lateral connections of the FPN, which are mathematically expressed as follows:

[0077] P l =Conv {1×1} C l +Upsample ×2 P l+1 ,l∈{3,4,5}

[0078] Among them, Conv 1×1 Used to adjust the number of channels, Upsample ×2 This is a 2x upsampling operation.

[0079] Subsequently, by using the proposed detection head to achieve more accurate detection of mobile phone and pedestrian categories and locations, this step introduces UniHead into the mobile phone behavior recognition scenario, that is, adopts a multi-sensor unified detection head design (DP+DAT+CIT) based on UniHead, the configuration of which is as follows:

[0080] Deformable Perception (DP): Adaptively samples target features through deformable convolutions, as shown in the formula:

[0081]

[0082] Where, Δ pk and Δ mk These are the learned offset and modulation factor, respectively.

[0083] Global Awareness (DAT): Models long-range dependencies using a dual-axis aggregation transformer (DAT), comprising two layers of efficient dual-axis attention (EDA) and cross-axis aggregation blocks (CAB).

[0084]

[0085] Cross-task awareness (CIT): The CIT module uses X DAT Given a unique input, extract the classification branch feature X. c and localization branch features X l (All by X) DAT (Obtained by projection), cross-task feature alignment is achieved through cross-attention, and cross-task enhanced feature X is output. CIT The classification and localization tasks are aligned through a two-layer cross-task interaction Transformer (CIT), and its cross-attention mechanism is as follows:

[0086] X CIT =CIT(X) c ,x l )=CA(Q l Concat(K) c ,K l ),Concat(V c V l ))

[0087] Finally, the detector outputs the bounding box coordinates b∈R for both the phone and the pedestrian. 4 And the class confidence s∈[0,1], normalized by the Sigmoid function:

[0088] s=σ(MLP(X cls b = MKP(X) loc )

[0089] Where X cls and X loc These are features for classification and localization branches, respectively. This design significantly improves detection accuracy in complex scenes while maintaining computational efficiency. This step introduces UniHead into mobile behavior recognition scenarios, comprising three key modules: the DP module uses deformable convolution to achieve dynamic sampling, improving detection capabilities for small targets and occluded scenes; the DAT module utilizes dual-axis Transformer modeling to achieve long-distance dependency, enhancing robustness in complex lighting and densely populated human environments; and the CIT module enables cross-task feature interaction, making classification and localization consistent, complementary, and mutually reinforcing.

[0090] (3) Distance-constrained matching algorithm

[0091] This step uses a distance-constrained matching algorithm to match the detected mobile phone to the pedestrian bounding box most likely to hold it, and then passes the bounding box information as input for the next stage.

[0092] In video surveillance scenarios, this algorithm accurately associates detected mobile phones with their owners (pedestrians). It comprehensively considers spatial distance (the positional relationship between the phone and the pedestrian) to address mismatch issues in complex scenarios involving multiple targets and occlusion.

[0093] Define a set P of mobile phone detection bounding boxes, and p i Let H represent the detection box of the i-th mobile phone, and define the pedestrian detection box set H, and m j Let x represent the detection box for the j-th pedestrian, where x i ,y i ,w i ,h i With x j ,y j ,w j ,h j These represent the center coordinates and width / height of the mobile phone and the pedestrian frame, respectively.

[0094] Spatial distance calculation, mobile phone p i With pedestrian m j Normalized center distance:

[0095]

[0096] The denominator is normalized using the maximum bounding box size to eliminate the influence of the target scale. A matching rule is defined: nearest neighbor matching, for each mobile phone p... i Choose the pedestrian with the smallest spatiotemporal distance as the holder, if D(pi,h) * If the threshold value is greater than τ (threshold τ = 0.5), then it is determined that there is no holder.

[0097]

[0098] In conflict resolution, if multiple phones match the same pedestrian, only the pair with the shortest distance is retained, and the rest are rematched.

[0099] (4) Construction of pedestrian re-identification network

[0100] This step involves building a pedestrian re-identification network to perform in-depth identification of the identity ID of the detected mobile phone holder. Specifically, a pedestrian re-identification algorithm based on Neighbor Feature Centralization (NFC) is proposed, which improves the consistency of pedestrian features through Top-k neighbor bidirectional verification, weighted aggregation, and feature centralization. The goal is to re-identify the identity of the pedestrian holding the mobile phone after obtaining the bbox.

[0101] First, in the feature extraction stage, a ResNet-50 backbone network is used to extract pedestrian features f. i :

[0102] f i =ResNet50(x i )∈R d (d=2048)

[0103] Secondly, during the neighbor search phase, the similarity matrix of all samples in the feature library is calculated, x i Let R represent the detection bounding box region image of the i-th pedestrian. d Let f represent the spatial dimension of the feature vector, where d represents the number of dimensions of the feature vector. For each pedestrian feature f i The top-k most similar samples are selected as the neighborhood set N. i .

[0104]

[0105] In the above formula, f represents the feature vector of the i-th pedestrian. i transpose, s ij Let R represent the dot product of the features of the i-th pedestrian and the j-th pedestrian. N×N Represents the dimension of the similarity matrix;

[0106] Next, a two-way verification strategy is used to screen reliable neighbors during the feature aggregation stage:

[0107] M i ={j|j∈N i ∧i∈N j}

[0108] In the above formula, N j This represents the Top-k neighbor set of the j-th pedestrian;

[0109] The weighted aggregated neighbor features are obtained using the following formula.

[0110]

[0111] Among them, weight τ is the softmax temperature coefficient, and η is the polymerization strength coefficient (default 0.5). Finally, L2 normalization and dimensionality reduction are performed during the feature optimization stage.

[0112]

[0113] The final features obtained The final output result is obtained after similarity calculation.

[0114] (5) Construct the loss function and train the network

[0115] The semantic relevance of the two tasks is explicitly bound by a joint loss function (detection loss + ReID classification loss + NFC aggregation loss), achieving end-to-end joint optimization. Specifically, the joint loss function consists of two parts: detection stage loss and pedestrian re-identification loss. The detection part includes classification loss (identifying mobile phones / pedestrians) and bounding box regression loss (precise localization), defined as follows:

[0116] Solving the problem of category imbalance (e.g., the number of pedestrians far exceeds the number of mobile phones):

[0117]

[0118] Where, p i It is the predicted class confidence, r i The true label is 0 / 1, and γ is the focus parameter (default γ = 2).

[0119] Smooth L1 Loss (regression loss) optimizes bounding box coordinate prediction.

[0120]

[0121] Among them, t i,m These are the parameters of the prediction box. These are the parameters of the real bounding box.

[0122] The pedestrian re-identification loss is based on the feature centering idea, and a joint loss is designed to optimize feature discriminativity. Cross-entropy loss for supervised ID classification tasks:

[0123]

[0124] Among them, f j is the pedestrian feature vector, and C is the number of pedestrian ID categories.

[0125] Enhance intra-class consistency by designing feature aggregation loss through Neighbor Feature Centralization (NFC):

[0126]

[0127] Where, N j It is feature f j The set of Top-kk neighbors.

[0128] Finally, the detection and ReID tasks are jointly optimized, and the losses of each part are weighted and fused:

[0129]

[0130] Where λ1 is the classification loss weight (default 0.5) and λ2 is the feature aggregation loss weight (default 0.3).

[0131] The training strategy is a phased training strategy. Phase 1: Freeze the ReID network and train only the detection part (λ1 = λ2 = 0). Phase 2: Joint training, gradually increasing λ1 and λ2 with a step size of 0.1 (adjusted every 20 epochs, where λ1 increases and λ2 decreases, and λ1 + λ2 = 1). Optimizer: AdamW, initial learning rate: 1 × 10⁻⁶. -4 (Detection), 5×10 -5 (ReID).

[0132] Example

[0133] I. Environmental Preparation:

[0134] 1. Hardware environment:

[0135] NVIDIA RTX 4090 GPU (24GB VRAM), Intel i9-13900K CPU, 64GB DDR5 RAM, 2TB SSD;

[0136] 2. Software environment:

[0137] Ubuntu 22.04LTS+PyTorch 2.1.0+TorchVision 0.16.0;

[0138] Auxiliary libraries: OpenCV 4.8.0 (video processing), Scikit-learn 1.3.0 (similarity calculation), Transformers 4.35.2 (BERT semantic parsing), LabelImg 1.8.6 (data annotation);

[0139] 3. Dataset preparation:

[0140] 72 hours of continuous video recording was conducted using surveillance cameras. A standardized color image sequence was obtained using video frame-segmentation technology (one frame for every two frames), resulting in 180,000 images (all images were uniformly resized to 640×640 pixels). Bboxes were labeled using the VOC format: mobile phone bounding boxes (ClassID=0) and human bounding boxes (ClassID=1), resulting in 30,000 mobile phone bounding boxes and 150,000 pedestrian bounding boxes. The dataset was divided into a training set (144,000 images), a validation set (18,000 images), and a test set (18,000 images) in an 8:1:1 ratio, encompassing diverse scenarios (sunny / cloudy days, peak / off-peak traffic, mobile phone obstruction / no obstruction).

[0141] II. Specific Implementation

[0142] 1. Backbone network and FPN configuration:

[0143] Backbone network: ResNet-50 is used, with ImageNet pre-trained weights loaded, the parameters of the first 4 layers are frozen, and the last 3 layers are fine-tuned;

[0144] FPN: The input image is processed by ResNet-50 to extract C3 (80×80×512) and C4 (40×512) values.

[0145] Three-layer feature maps of C5 (40×1024) and C5 (20×20×2048) are generated according to the patent formula; P3, P4, and P5 are generated according to the patent formula.

[0146] P5 feature map.

[0147] 2. UniHead Detector Configuration (DP+DAT+CIT Serial Structure)

[0148] DP: Input: P3-P5 feature maps output by FPN (stitched size 80×80×768).

[0149] Output: Adaptive sampling feature X DP (80×80×768);

[0150] DAT: Input: X DP Feature map; Output: Global context feature X DAT (80×80×768);

[0151] CIT: Input: X DAT Feature map; Output: Cross-task aligned feature X cIT (80×80×768);

[0152] 3. Implementation of distance-constrained matching algorithm:

[0153] Input: The set of all mobile phone detection boxes P and the set of pedestrian detection boxes H in a single frame image output by the detection network; Set the matching threshold τ = 0.5. If multiple mobile phones match the same pedestrian (e.g., p1 and p2 both match h3), only the pair with the smallest distance is retained, and the remaining mobile phones are re-matched with the nearest neighbor.

[0154] Output: Generate a set of mobile phone-pedestrian matching pairs and a list of "unowned" mobile phones. Use the cropped images corresponding to the successfully matched pedestrian bounding boxes as input to the pedestrian re-identification network.

[0155] 4. Implementation of Pedestrian Re-identification Network:

[0156] Input: A cropped image (256×128 pixels) corresponding to the distance-matched pedestrian detection bounding box m_j. Output: 2048-dimensional original features f_i = ResNet50(x_i)∈R 2048 , where x_i is the cropped pedestrian image.

[0157] Feature library construction: The feature library consists of 2048-dimensional features of all pedestrians in the training set (size of 120,000);

[0158] Top-k neighbor selection: k = 5;

[0159] III. Model Reasoning

[0160] Input: Real-time monitoring video stream (1920×1080, 25fps);

[0161] Preprocessing: Each frame of image is scaled to 640×640 and normalized to [0,1];

[0162] Detection and matching: Single-frame inference time ≤ 60ms, output mobile phone / pedestrian bounding box and category confidence, and obtain the pedestrian holding the device through distance matching;

[0163] Identity recognition: After cropping the pedestrian image, the ReID network is input, outputting 512-dimensional features, which are matched with the feature library (cosine distance threshold 0.3) to output the identity ID;

[0164] Output results: Overlay a mobile phone frame (red), a pedestrian frame (blue), and an identity label (such as "ID:00123 Zhang San") on the monitoring screen. Mobile phones without a holder are marked "No clear holder". The results are saved to the backend database.

[0165] The above solution is merely an illustration of a preferred example and is not limited thereto. When implementing this invention, appropriate substitutions and / or modifications can be made according to the user's needs.

[0166] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. It can be applied to various fields suitable for the present invention. Other modifications can be readily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and examples shown and described herein.

Claims

1. A method for mobile phone holder identity detection based on a two-stage coupled detection framework, characterized in that, include: S1. Construct a detection network to identify mobile phones and people; S2. Construct a pedestrian re-identification network that can match mobile phone holders with their identity information; S3. Construct a distance-constrained matching algorithm to connect the first-stage detection network and the second-stage pedestrian re-identification network, thus obtaining a two-stage coupled detection model. S4. The two-stage coupled detection model is trained and validated using the corresponding dataset, and a loss function is used during the training process. Measuring the gap in training data; S5. Import the monitoring data to be identified into the two-stage coupled detection model. After the detection network outputs the bounding boxes and categories of the mobile phone and the person, the nearest holder of the mobile phone is calculated by the distance constraint matching algorithm. The nearest holder is then used as the input of the pedestrian re-identification network. After the second stage of identity recognition, the information of the mobile phone holder is output to complete the detection of the identity of the mobile phone holder.

2. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S1, the detection network uses a detection head to implement ResNet-50 as the backbone network. After multi-scale feature fusion is achieved by combining it with the Feature Pyramid Network (FPN), the bounding box coordinates of the mobile phone and pedestrian are output through the detection head UniHead. Among them, the four-level feature maps of the backbone network are connected horizontally through the top-down path of the feature pyramid network to generate multi-scale features P. l It is characterized by the following formula: P l =Conv {1×1 }C l +Upsample ×2 P l+1 ,l∈{3,4,5} In the above formula, Conv 1×1 Upsample is used to adjust the number of channels in convolutional layers. ×2 This is a 2x feature upsampling operation; The UniHead detection head outputs the bounding box coordinates b and category confidence s of the mobile phone and pedestrian using the following formula: s<σ(MLP(X) cls )),b=MLP(X loc ) In the above formula, X cls and X loc σ represents the features for the classification and localization branches, respectively. MLP stands for Multilayer Perceptron, and σ is the normalization operation performed by the Sigmoid function.

3. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 2, characterized in that, The detection head UniHead includes: a deformation sensing module, a global sensing module, and a cross-task sensing module; Specifically, the deformation perception module uses deformable convolution to obtain adaptive sampling target features X from the input features. DP (p), the sampling target feature X DP (p) is characterized by the following formula: In the above formula, K represents the total number of sampling points in the deformable convolution kernel, X is the input feature tensor, p is the spatial coordinates of the output feature map, and W... k p is the convolution weight corresponding to the k-th sampling point. k It is the preset offset of the k-th sampling point in the standard convolution, Δ pk and Δ mk These are the learned offset and modulation factor, respectively; The global perception module achieves long-distance dependency through dual-axis aggregation modeling, and the global perception module contains two layers of efficient dual-axis attention EDA, and the output X of the two layers of efficient dual-axis attention EDA is... DAT It is characterized by the following formula: In the above formula, X is the input feature tensor, Concat() represents parallel connection, and W O It is the output projection matrix. It is a feature formed by aggregation along the vertical axis. It is a feature formed by aggregation along the horizontal axis, and Q {horiz} For the horizontal query matrix, K {horiz} V is the horizontal bond matrix. {shared} For shared value matrices; The cross-task perception module utilizes a two-layer cross-task interaction CIT cross-attention mechanism, CIT(X). c ,X l To align classification and localization tasks, and the output X of the cross-task awareness module CIT It is characterized by the following formula: X CIT =CIT(X c ,X l )=CA(Q l ,Concat(K c ,K l ),Concat(V c ,V l )) In the above formula, X c and X l They represent X respectively DAT Given a unique input, extract the classification branch feature X. c and localization branch features X l The corresponding input feature tensor comes from the classification task X. DAT And the input feature tensor of the localization task, K c ,K l V represents the key matrices of the two tasks. c V l Let Q represent the value matrices for the two tasks respectively. l It is a query matrix and comes only from the localization task; CA() represents cross attention.

4. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S2, the pedestrian re-identification network performs holder identity re-identification based on NFC with neighbor feature centralization after obtaining the detection frame of the pedestrian holding the mobile phone. The final feature output by the pedestrian re-identification network is... It is characterized by the following formula: In the above formula, L2 represents the L2 norm, Norm() represents normalization, and MLP represents a multilayer perceptron. This indicates a weighted aggregation of neighbor features, and f i This represents the pedestrian features extracted using the ResNet-50 backbone network during the feature extraction stage, where η is the aggregation strength coefficient and M is the number of features extracted. i To select reliable neighbors using a two-way verification strategy during the feature aggregation stage, w j f j The corresponding weight, f j It is the feature vector of the j-th pedestrian.

5. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S3, the distance-constrained matching algorithm obtains the mobile phone p using the following formula. i With pedestrian m j Normalized center distance d(p) i ,h j ): In the above formula, x i ,y i ,w i ,h i These represent the coordinates of the center point, width, and height of the mobile phone frame output by the detection network, respectively. j ,y j ,w j ,h j These represent the coordinates of the center point, width, and height of the pedestrian bounding box output by the detection network, respectively. Based on the nearest neighbor matching rule, the formula for obtaining the relationship between each mobile phone p is as follows. i The pedestrian with the smallest spatiotemporal distance is the holder h * And when d(p) i ,h j If )>τ, then it is determined that there is no holder; In the above formula, H represents the set of pedestrian detection boxes.

6. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S4, the dataset is obtained in the following ways: S40. Obtain standardized color image sequences from historical surveillance videos using video framing technology; S41. Use VOC format to implement bbox annotation on the color image sequence, set the Class ID of the mobile phone border to 0, and set the Class ID of the human body border to 1; S42. Based on the annotated color image sequence, a corresponding binary mask is generated using an instance segmentation model, and the BERT model is used simultaneously to perform semantic parsing on the data text in the video meta-data. This allows for the generation of structured atomic attribute labels through entity recognition and relation extraction, resulting in a multi-dimensional annotated dataset that integrates visual features, semantic attributes, and temporal relationships.

7. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S4, the loss function It is characterized by the following formula: In the above formula, To address the class imbalance problem, a classification loss function is needed. To optimize the regression loss function for bounding box coordinate prediction, The cross-entropy loss function is used to supervise ID classification tasks. To enhance intra-class consistency through NFC feature centralization, an aggregation loss function is designed, where λ1 is the classification loss weight and λ2 is the feature aggregation loss weight.

8. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 7, characterized in that, The classification loss function It is characterized by the following formula: In the above formula, p i This represents the predicted class confidence score, r is the true label, and γ is the focusing parameter. This indicates a reduction in the weight of easily categorized boxes, where N is the total number of boxes involved in the calculation. The regression loss function It is characterized by the following formula: In the above formula, N is the total number of boxes involved in the calculation, and smoothness L1 It is a smoothed L1 loss, t i,m These are the parameters of the prediction box. These are the parameters of the true bounding box, where m∈{x,y,w,h} represents x as the x-coordinate of the center of the bounding box, t as the y-coordinate of the center of the bounding box, w as the width of the bounding box, and h as the height of the bounding box. The cross-entropy loss function It is characterized by the following formula: In the above formula, f j Here, C is the pedestrian feature vector, C is the number of pedestrian ID categories, and M is the total number of pedestrian samples involved in the calculation. f is the feature vector of the j-th pedestrian. j The weight vector w of the pedestrian ID of type c c dot product, f is the feature vector of the j-th pedestrian. j The weight vector w of the k-th pedestrian ID k The dot product; The aggregation loss function It is characterized by the following formula: In the above formula, N j It is feature f j The top-kk neighbor set, f k Let represent the pedestrian feature vector of the kth neighbor in the Top-k neighbor set of the jth pedestrian feature.

9. The mobile phone holder identity detection method based on a two-stage coupled detection framework as described in claim 1, characterized in that, In S4, the two-stage coupled detection model adopts a staged training strategy; In the first stage of training, the pedestrian re-identification network was frozen, and only the ground detection network was trained. The optimizer used for training was AdamW, with an initial learning rate of 1×10⁻⁶. -4 And loss function Cross-entropy loss function Aggregate loss function The corresponding rights are all set to 0; In the second phase of training, joint training was employed, using AdamW as the optimizer, with an initial learning rate of 5×10⁻⁶. -5 (ReID), while gradually increasing λ1 and λ2.

Images