Pedestrian re-identification method based on multi-sensor space-time joint constraint

Abstract

The invention belongs to the technical field of pedestrian re-identification, and particularly relates to a pedestrian re-identification method based on multi-sensor spatio-temporal joint constraint, which comprises the following steps: after data synchronization of a millimeter wave radar and a camera, subdividing areas of the view of the camera according to a preset threshold value; counting a first transfer event of pedestrians among the regions, calculating a refined inter-region transfer probability, and performing Gaussian kernel density estimation on time interval samples of each pair of sub-region transfer to obtain a time distribution model; and filtering candidates which do not conform to the space-time prior probability according to the region transition probability and the time distribution model, and multiplying the visual feature similarity of the candidate image and the query image by the space-time prior probability which is corrected through a Logistic smooth function to generate a final matching score. According to the method, fine-grained modeling of the motion path is considered, visual features and spatio-temporal information are fused, the accuracy and robustness of cross-camera matching are improved, the retrieval space is effectively reduced through the candidate pre-filtering strategy, and the real-time performance and efficiency of the system are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian re-identification, and in particular relates to a pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints. Background Art

[0002] Person re-identification aims to accurately match the same person across camera fields of view, and most re-ID methods approach this as a combination of visual learning and image search. At the visual feature level, mainstream methods rely on deep learning to extract appearance features such as clothing and body shape. However, these approaches inherently face the dual constraints of semantic ambiguity and environmental sensitivity. On the one hand, similar clothing or accessories can easily lead to mismatching between different individuals, while the same person can present distinct visual patterns due to changes in perspective and posture, exacerbating the problem of feature drift. On the other hand, lighting fluctuations and occlusion interference (such as vehicles and crowded crowds) in real-world surveillance scenes can significantly weaken the discriminability of appearance features. Although the introduction of auxiliary information such as human body parsing and posture alignment can partially alleviate these issues, their reliance on additional annotations and limited generalization capabilities for complex scenes make it difficult to completely address the inherent fragility of appearance representation.

[0003] Additionally, spatiotemporal constraint methods utilize camera layout topology and timestamp information to limit the matching range, which can mitigate the limitations of visual features to a certain extent. However, existing spatiotemporal modeling often remains at the camera level, ignoring the objective laws of pedestrian movement in real-world surveillance scenarios, which exhibit regional and fine-grained characteristics. This is particularly true in areas such as intersections, where a single global time window or fixed transition model struggles to capture the dynamic behavior of individuals across different fields of view.

[0004] In order to solve the above problems, the present invention proposes a pedestrian re-identification method based on multi-sensor spatiotemporal constraints. By combining radar data to subdivide the field of view of each camera, and statistically modeling the transition probability and transition delay between the four sub-regions, it can achieve an accurate description of the pedestrian's cross-regional movement pattern.

[0005] The information disclosed in this background technology section is only intended to enhance understanding of the overall background of the invention and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to a person skilled in the art. Summary of the Invention

[0006] The purpose of the present invention is to provide a pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints to solve the problem that traditional visual re-identification methods are easily affected by factors such as occlusion and lighting fluctuations in complex scenes, resulting in mismatching, and the defect that the existing spatiotemporal constraint modeling granularity is too coarse to accurately reflect the movement behavior of pedestrians.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints includes the following steps:

[0009] S1. Data collection and preprocessing, specifically:

[0010] Millimeter-wave radar data acquisition equipment is used to track pedestrians in real time to obtain the data matrix:

[0011] For the data output by the radar point cloud, objects with reflection intensity (RCS) below the threshold and marked as static targets are eliminated to reduce environmental clutter and static object interference;

[0012] S2. Regional transfer probability statistics and time distribution modeling, specifically:

[0013] After synchronizing millimeter-wave radar data with camera data in time, the field of view of each camera is divided into regions based on the specific relationship between camera positions and pedestrian movement trajectories according to preset thresholds;

[0014] Based on radar trajectories, the first pedestrian movement events between these areas are counted and refined regional transition probabilities are calculated. Regional transition probabilities essentially describe pedestrians' tendency to move along different spatial paths, statistically reflecting "which paths are more likely to be traversed by real targets." Regional transition probabilities, as spatial prior probabilities, can effectively enhance the ability to perceive the target's movement direction and provide important auxiliary evidence when visual features are not obvious.

[0015] The actual transfer time interval between each pair of regions is recorded, and the kernel density estimation method is used to fit its probability distribution to establish a time distribution model of path perception.

[0016] S3. Retrieve candidate image filtering:

[0017] Filter candidate images that do not meet the spatiotemporal prior probability based on the regional transition probability and time distribution model;

[0018] For candidate images that appear in the wrong time period, a lower score is assigned through the time distribution probability density function to automatically avoid false matches;

[0019] S4. Decision-level fusion and matching strategy:

[0020] The visual feature similarity between the candidate image filtered by S3 and the query image is multiplied by the spatiotemporal prior probability corrected by the Logistic smoothing function to generate the final matching score.

[0021] As a preferred method, ARS-408MMW radar is used as millimeter wave radar data acquisition equipment in S1, and the data matrix obtained is:

[0022] D radar ={(ID k ,t k ,x k ,y k ,v x,k ,v y,k )};

[0023] Among them, ID is the pedestrian identification, t is the timestamp, (x, y) is the pedestrian position coordinate, (v x, v y ) is the corresponding velocity component.

[0024] As a preferred method, in S2, for the urban road monitoring scenario with dual cameras, the field of view of each camera is subdivided into upper and lower sub-areas according to a preset threshold, forming a total of four areas. Among them, the sub-areas of camera A are respectively recorded as A1 and A2, and the sub-areas of camera B are respectively recorded as B1 and B2. For each pedestrian, its trajectory data can be represented as a set of sequences, and the area transition probability can be expressed as:

[0025]

[0026] Among them, t a For pedestrians entering A for the first time i The time of the region, t b For pedestrians entering B for the first time j time in the area; N is the total number of pedestrians, and N(A i →B j ) are all pedestrian trajectories from area A i to B j The total number of transfers, N(A i ) is to enter area A i the total number of;

[0027] Similarly, area B j to A i The transition probability is as follows:

[0028]

[0029] The above probability matrix can truly describe the preference and probability of the target moving between different sub-regions. At the same time, the time required for different paths to transfer varies significantly.

[0030] As a preferred method, a time model of path perception is established in S2, and a time interval ΔT=t is recorded for each area transfer event. b -t a , and use Gaussian kernel density estimation to perform smooth modeling on all ΔT samples, and the resulting time distribution model is:

[0031]

[0032] Among them, n ij is the total number of regional transfer events, h is the bandwidth parameter, and the optimal value is selected through cross-validation to ensure the accuracy of the time distribution model fitting.

[0033] Preferably, generating the final matching score in S4 includes the following steps:

[0034] First, in the re-identification and matching stage, the pre-trained visual model Restnet50 is used to extract query image features and candidate image features, and the cosine similarity is used to calculate the visual similarity S v :

[0035]

[0036] At the same time, according to the area to which the pedestrian picture belongs, the spatiotemporal prior probability is calculated from the regional transition probability and the time distribution probability density function:

[0037] P st =F(p spa ,p tem );

[0038] In order to improve the robustness to low sample areas, Logistic smoothing correction is used:

[0039]

[0040] Finally, the visual and spatiotemporal information are weightedly fused at the decision layer to obtain a comprehensive matching score:

[0041] S=S v ×P st .

[0042] The present invention not only makes full use of the motion and spatiotemporal information of millimeter-wave radar, but also adds fine-grained spatial regional constraints in the visual feature matching stage, achieving high-precision and efficient cross-camera matching of pedestrians.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints of the present invention utilizes sophisticated regional-level spatiotemporal modeling to effectively filter out unreasonable matches and reduce the false detection rate.

[0045] (2) The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints of the present invention is beneficial to overcoming the problem of unstable visual features in complex lighting and occlusion environments by fusing millimeter-wave radar spatiotemporal information with visual features.

[0046] (3) The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints of the present invention can fully reflect the movement path and dynamic state of pedestrians in actual scenes by dividing the camera field of view into regions.

[0047] (4) The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints of the present invention filters candidate images based on the regional transition probability and time distribution model, which can significantly compress the retrieval space and improve the matching efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of the pedestrian re-identification method of the present invention;

[0049] Figure 2 This is a flow chart of a multimodal fusion human weight recognition system constructed in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following is a clear and complete description of the technical solution of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts are within the scope of protection of the present invention.

[0051] In the description of the present invention, it should be noted that the orientations or positional relationships indicated by terms such as "center", "up", "down", "left", "right", "vertical", "horizontal", "inside" and "outside" are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the invention.

[0052] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and connections within two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention.

[0053] Refer to the attached Figure 1-2 Taking a standard person re-identification network based on ResNet-50 as an example, the proposed person re-identification method based on multi-sensor spatiotemporal constraints is integrated into the traditional visual matching process to construct a multimodal fusion person re-identification system. The system specifically includes the following four steps:

[0054] Step 1: Hardware deployment;

[0055] A data acquisition setup was built outdoors, consisting of an ARS408 millimeter-wave radar and two monocular cameras with a focal length of 2.8 mm, a resolution of (640 x 480), and a frame rate of 30 Hz. All sensors were pre-calibrated and mounted on a bracket at a height of 1.7 m.

[0056] Step 2: Use the classic ResNet-50 deep convolutional neural network as the backbone network for visual feature extraction;

[0057] The input image size is 256×128. After passing through standard convolutional layers, BatchNorm, residual structure, and global average pooling, the output is a global feature vector of dimension 2048. The network architecture includes standard residual blocks, batch normalization layers, and global average pooling layers. Finally, an L2 normalization layer is connected to constrain the feature space and ensure the stability of the calculated cosine similarity. During the training phase, labeled image data is used to jointly optimize the cross-entropy loss and triplet loss to obtain a visual feature representation with good discriminative ability.

[0058] Step 3: radar data processing and regional modeling;

[0059] First, the radar collects spatial trajectory data for each target at high frequency. The output includes the target ID, position, velocity, and timestamp for each frame. Simultaneously, each target undergoes trajectory aggregation and sliding average filtering to eliminate the effects of detection jitter and short-term occlusion.

[0060] After the trajectory data is stable, the camera’s fixed installation position and its corresponding radar coordinate range are combined to subdivide each camera’s field of view into two sub-areas, upper and lower (or left and right). For example, the C1 camera is divided into r1=C 1A ,r2=C 1B , C2 camera is divided into r3=C 2A ,r4=C 2B , thus forming four spatial sub-regions. By combining horizontal and vertical coordinates to define region boundaries, this introduces fine-grained spatial partitioning in cross-camera modeling. This region partitioning method enables automatic region modeling using radar data without relying on additional visual annotation or map information, and is highly portable and real-time.

[0061] Based on the spatial region division, the system can detect the target from a certain area r i Move to another area j The transfer behavior of the region is counted, a transfer is defined as a "first entry" event, and the regional transfer probability is calculated according to the following formula:

[0062]

[0063] Among them, N(r i →r j ) indicates that the target is from r i Transfer to r j The total number of events, N(r i ) indicates that all entries into r i The model fully captures the regional-level movement directionality and behavioral preferences, and constructs a 4×4 regional transition probability matrix.

[0064] Furthermore, for each region’s transfer event r i →r j The time interval ΔT=t j -t i Perform statistics to obtain a set of time difference samples And the Gaussian kernel density estimation method is used to fit the time interval probability density function:

[0065]

[0066] Among them, K(·) is the Gaussian kernel function, h is the bandwidth parameter, and n ij is the number of effective samples. Unlike traditional filtering methods that use a unified time window, this model implements dynamic temporal modeling of path direction perception. Leveraging established regional transition probabilities and time interval distributions, candidate images are filtered before visual comparison, retaining only those that meet reasonable transition paths and time windows. This significantly compresses the search space, improves overall system efficiency, and is suitable for large-scale deployment. It is particularly suitable for cross-field pedestrian tracking at urban intersections and in complex traffic scenarios.

[0067] Step 4: Vision and spatiotemporal joint decision fusion;

[0068] In order to transfer the above region probability and the time distribution probability density function f ij (ΔT) is fused into the visual re-identification model. In the inference stage, the present invention introduces a "visual-temporal joint scoring mechanism" and designs a joint scoring function to fuse the visual feature similarity with the regional transition probability and the time distribution probability density function.

[0069] First, the backbone network ResNet-50 is used to extract the deep feature vector v from the input image q 、v g , and calculate the visual similarity of the image pair:

[0070]

[0071] Then, combining the radar track areas of the query image and the candidate image (determined by the current frame track position), the system locates the area where they are located as (r i ,r j ), and obtain the corresponding regional transition probability and time distribution probability density function to calculate the spatiotemporal prior probability:

[0072]

[0073] To avoid the volatility problem caused by low statistical samples, the system uses the Logistic function to smooth the spatiotemporal prior probability:

[0074]

[0075] Among them, λ and γ are empirically determined adjustable parameters used to adjust the impact of spatiotemporal information on the matching score;

[0076] Finally, the fused matching score is composed of the product of visual similarity and spatiotemporal prior probability:

[0077] S final =S visual ·P st ;

[0078] This mechanism can be seamlessly integrated into any feature matching-based ReID system and does not rely on a specific backbone structure. It only requires expanding the post-processing scoring function to significantly improve the matching accuracy and system robustness across cameras.

[0079] The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints of the present invention can constrain and supplement pedestrian cross-camera matching through refined camera area division, inter-area transfer probability and time interval statistics. It not only takes into account the fine-grained modeling of motion paths, but also integrates visual features and spatiotemporal information. It not only significantly improves the accuracy and robustness of cross-camera matching, but also effectively reduces the retrieval space through the candidate pre-filtering strategy, ensuring the real-time and efficiency of the system.

[0080] The foregoing descriptions of specific exemplary embodiments of the present invention are for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the invention and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the invention and various options and modifications. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints, characterized by: The following steps are involved: S1. Data collection and preprocessing, specifically: Millimeter-wave radar data acquisition equipment is used to track pedestrians in real time to obtain the data matrix: For the data output by the radar point cloud, objects with reflection intensity below the threshold and marked as static targets are eliminated to reduce environmental clutter and static object interference; S2. Regional transfer probability statistics and time distribution modeling, specifically: After synchronizing millimeter-wave radar data with camera data in time, the field of view of each camera is divided into regions based on the specific relationship between camera positions and pedestrian movement trajectories according to preset thresholds; Based on the radar trajectory, the first pedestrian transfer events between these areas are counted and the refined area transfer probability is calculated; The actual transfer time interval between each pair of regions is recorded, and the kernel density estimation method is used to fit its probability distribution to establish a time distribution model for path perception. S3. Retrieve candidate image filtering: Filter candidate images that do not meet the spatiotemporal prior probability based on the regional transition probability and time distribution model; For candidate images that appear in the wrong time period, a lower score is assigned through the time distribution probability density function to automatically avoid false matches; S4. Decision-level fusion and matching strategy: The visual feature similarity between the candidate image filtered by S3 and the query image is multiplied by the spatiotemporal prior probability corrected by the Logistic smoothing function to generate the final matching score.

2. The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints according to claim 1 is characterized in that: In S1, ARS-408MMW radar is used as millimeter wave radar data acquisition equipment, and the data matrix obtained is: D radar ={(ID k ,t k ,x k ,y k ,v x,k ,v y,k )}; Among them, ID is the pedestrian identification, t is the timestamp, (x, y) is the pedestrian position coordinate, (v x, v y ) is the corresponding velocity component.

3. The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints according to claim 1 is characterized in that: In S2, for the urban road monitoring scenario with dual cameras, the field of view of each camera is subdivided into upper and lower sub-areas according to a preset threshold, forming a total of four areas. The sub-areas of camera A are denoted as A1 and A2, and the sub-areas of camera B are denoted as B1 and B2. For each pedestrian, their trajectory data can be represented as a set of sequences, and the area transition probability can be expressed as: Among them, t a For pedestrians entering A for the first time i The time of the region, t b For pedestrians entering B for the first time j time in the area; N is the total number of pedestrians, and N(A i →B j ) are all pedestrian trajectories from area A i to B j The total number of transfers, N(A i ) is to enter area A i the total number of; Similarly, area B j to A i The transition probability is as follows:

4. The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints according to claim 1 is characterized in that: In S2, a time model of path perception is established, and the time interval ΔT=t is recorded for each area transfer event. b -t a , and use Gaussian kernel density estimation to perform smooth modeling on all ΔT samples, and the resulting time distribution model is: Among them, n ij is the total number of regional transfer events, h is the bandwidth parameter, and the optimal value is selected through cross-validation to ensure the accuracy of the time distribution model fitting.

5. The pedestrian re-identification method based on multi-sensor spatiotemporal joint constraints according to claim 1 is characterized in that: Generating the final matching score in S4 includes the following steps: First, the pre-trained visual model Restnet50 is used to extract query image features and candidate image features, and the cosine similarity is used to calculate the visual similarity S v : At the same time, according to the area to which the pedestrian picture belongs, the spatiotemporal prior probability is calculated from the regional transition probability and the time distribution probability density function: P st =F(p spa ,p tem ); In order to improve the robustness to low sample areas, Logistic smoothing correction is used: Finally, the visual and spatiotemporal information are weightedly fused at the decision layer to obtain a comprehensive matching score: S=S v ×P st 。

Images