Video pedestrian feature extraction method based on pose estimation and elite pool logarithmic sampling

Abstract

This invention discloses a video pedestrian feature extraction method based on pose estimation and elite pool logarithmic sampling. The method utilizes a pose estimation network to detect human keypoints in several frames of images; determines valid keypoints and calculates the semantic integrity score of the images; divides the images in the video trajectory sequence into an elite pool and a backup pool, and sorts the images in descending order according to their semantic integrity scores; constructs a logarithmic index mapping function to generate a non-linear sampling index sequence based on a preset number of samples; extracts target images from the elite pool according to the sampling index sequence; if the number of images in the elite pool is insufficient, it is supplemented sequentially from the backup pool; and inputs the final sampled image sequence into a pedestrian re-identification model to extract and aggregate feature vectors. This invention significantly improves the temporal diversity and robustness of the sampled frames while ensuring high feature quality, effectively solving the problems of missed detections and feature collapse in complex monitoring scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and video surveillance technology, specifically to a method for extracting pedestrian features from videos based on pose estimation and elite pool log sampling. Background Technology

[0002] Computer vision technology is now widely used in fields such as intelligent security, smart cities, and human-computer interaction. Among these, person re-identification (Re-ID), a core technology in intelligent video surveillance systems, aims to retrieve and locate specific pedestrian targets in surveillance networks spanning multiple fields of view and cameras. Compared to single-frame image-based Re-ID, video-based Re-ID utilizes rich spatio-temporal cues and motion information, theoretically offering higher robustness and accuracy. In practical applications, the input data is typically a video trajectory segment (Tracklet) containing multiple frames of images generated by a tracking algorithm.

[0003] However, when processing video data, due to the large number of video frames, directly inputting all frames into a deep neural network for feature extraction would result in enormous computational overhead and storage pressure, making it difficult to meet real-time requirements. Therefore, "key frame sampling" has become an indispensable preprocessing step in video Re-ID tasks. Existing sampling techniques are mainly divided into two categories: unsupervised sampling and quality assessment-based sampling, but both have significant limitations in complex real-world scenarios.

[0004] First, existing unsupervised sampling strategies (such as uniform sampling and random sampling) lack the ability to perceive the semantic content of images. Traditional uniform sampling selects images only at fixed time intervals, while random sampling randomly selects several frames from the trajectory. This "blind" strategy assumes that every frame in the video has the same amount of information, which is obviously not true in actual surveillance scenarios. Surveillance videos often contain a large number of blurred frames caused by fast movement, incomplete frames obscured by obstacles, or noisy frames with cluttered backgrounds. If the sampling algorithm unfortunately selects these low-quality frames, the extracted feature vectors will deviate significantly from the true feature distribution of pedestrians, leading to a substantial decrease in the accuracy of subsequent feature matching.

[0005] Second, quality screening methods based on hard thresholds are not robust and are prone to data loss. To address the issue of low-quality frames, some existing technologies introduce image sharpness evaluation metrics (such as Laplacian gradient, Brenner gradient, etc.) as quality scores and set a fixed hard threshold to discard low-scoring frames. However, this pixel-level sharpness evaluation often fails to reflect "semantic integrity" (for example, an empty image with an extremely clear background but no human figure may be misjudged as a high-quality frame). More seriously, in low-light nighttime or long-distance monitoring scenarios, the quality of all frames along the entire trajectory may generally be below the preset threshold. In this case, a hard threshold-based strategy will directly discard the entire trajectory, leading to the complete loss of target information (i.e., system missed detection), which is unacceptable in security applications.

[0006] Third, a simple Top-K sorting sampling strategy can lead to "temporal collapse." Another common approach is to sort all frames by quality and select only the K highest-scoring frames (Top-K Sampling). While this ensures the quality of individual frames, it often results in highly clustered sampling results over time. For example, the image is clearest during the few seconds a pedestrian pauses; the Top-K strategy will select all similar frames within those few seconds, discarding gait changes and multi-angle information during the pedestrian's movement. This feature set, lacking temporal diversity, cannot fully represent the dynamic characteristics of pedestrians, reducing the model's generalization ability.

[0007] How to automatically remove low-quality frames from videos, such as those with motion blur, occlusion, and incomplete poses, from a semantic level without relying on manual annotation; how to avoid data loss and missed detections due to over-screening in low-quality video scenarios and achieve adaptive and robust feature extraction; and how to balance high-quality features with temporal distribution diversity under limited sampling constraints to avoid feature collapse over time. Therefore, finding a balance between "image quality," "data integrity," and "temporal diversity" is a critical problem that pedestrian re-identification in complex scenarios urgently needs to solve. Summary of the Invention

[0008] Purpose of the invention: To address the aforementioned deficiencies in existing video pedestrian re-identification technologies, this invention aims to provide a video pedestrian feature extraction method based on pose estimation and elite pool logarithmic sampling, thereby improving the temporal diversity and robustness of sampled frames and effectively solving the problems of missed detection and feature collapse in complex monitoring scenarios.

[0009] Technical solution: The present invention provides a video pedestrian feature extraction method based on pose estimation and elite pool log sampling, comprising the following steps:

[0010] (1) Obtain a video trajectory sequence including several frames of images, and use a pose estimation network to detect human key points in each frame of images;

[0011] (2) Based on the key point detection results, determine the effective key points and calculate the semantic integrity score of the image;

[0012] (3) Set an elite threshold and divide the images in the video trajectory sequence into an elite pool and a backup pool; sort the images in the elite pool and the backup pool in descending order according to their semantic integrity scores.

[0013] (4) Construct a logarithmic index mapping function to generate a non-linear sampling index sequence based on a preset number of samples, so that the sampling points are densely distributed at the beginning of the sorted image sequence and sparsely distributed at the end.

[0014] (5) Extract the target image from the elite pool according to the sampling index sequence; if the number of images in the elite pool is insufficient, fill them in order from the backup pool.

[0015] (6) Input the final sampled image sequence into the pedestrian re-identification model to extract and aggregate feature vectors.

[0016] Furthermore, the implementation process of step (1) is as follows:

[0017] The model uses a pre-trained deep learning pose estimation network to process each frame of the trajectory, detect and output a set of human skeleton key points; for each frame, the model outputs the coordinates of M key points and their corresponding confidence scores.

[0018] Furthermore, the determination of effective key points in step (2) specifically involves:

[0019] Determine valid key points based on key point detection results. Valid keypoints are those with a confidence level greater than a preset confidence threshold.

[0020]

[0021] Where M represents the total number of key points in the human skeleton. This represents the confidence level of the j-th key point. I(⋅) is the confidence threshold, and I(⋅) is the indicator function; the confidence threshold is 0.7.

[0022] Furthermore, the specific formula for the semantic completeness score in step (2) is as follows:

[0023]

[0024] Where M represents the total number of key points in the human skeleton. It is the number of valid key points; It is the minimum number of valid key points. Further, step (3) specifically means: if the semantic integrity score of a certain frame image is greater than or equal to the preset elite threshold λ, then it is assigned to the elite pool; otherwise, it is assigned to the reserve pool; the value range of the elite threshold λ is 0.75 to 0.85.

[0025] Furthermore, the specific mathematical expression of the logarithmic index mapping function described in step (4) is as follows:

[0026]

[0027] in, =0,1,...,K−1, are linearly increasing sample numbers; sample index It exhibits exponential growth.

[0028] Furthermore, the implementation process of step (5) is as follows:

[0029] Based on the system's preset target sampling quantity K, images are first extracted from the elite pool using logarithmic indexing. If the number of images in the elite pool is sufficient to meet the sampling requirements, only images are sampled from the elite pool to ensure the purity of the features. If the number of images in the elite pool is insufficient, the number of gaps is calculated, and a corresponding number of images are selected from the backup pool in descending order of semantic integrity score to fill in the gaps.

[0030] The present invention also proposes a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described above.

[0031] The present invention also proposes an electronic device, comprising a memory and a processor, wherein:

[0032] Memory is used to store computer programs that can run on a processor;

[0033] A processor, configured to, while running the computer program, perform the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described above.

[0034] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] 1. Significantly improves the semantic purity of features: This invention uses pose estimation confidence to replace the traditional fuzzy detection algorithm and defines image quality at the semantic level. This method can accurately remove inefficient frames that are difficult to identify by traditional algorithms, such as "only the lower body" and "severe occlusion", so that the extracted feature vectors are more focused on the torso and appearance details of pedestrians.

[0036] 2. Achieves high robustness in extreme scenarios: Traditional hard threshold screening is prone to missed detections such as "all data being deleted" in nighttime or long-distance monitoring. The "elite pool-backup pool" dual-layer mechanism of this invention achieves a perfect combination of "selecting the best from the best" and "backup guarantee"; in high-quality videos, the system automatically blocks poor frames; in low-quality videos, the system automatically downgrades to use suboptimal frames, ensuring availability in all-weather scenarios.

[0037] 3. Optimized the spatiotemporal distribution structure of features: This invention innovatively proposes a logarithmic sampling strategy, which breaks the temporal clustering effect of traditional Top-K sampling. By achieving "dense head and sparse tail" coverage in the feature space, it not only retains the most representative high-quality appearance features, but also introduces gait diversity in the movement process, which significantly improves the generalization ability of the Re-ID model to viewpoint and posture changes. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention;

[0039] Figure 2 This is a schematic diagram comparing the principle of the logarithmic index mapping sampling strategy proposed in this invention with that of the traditional linear sampling strategy in terms of feature space coverage. Detailed Implementation

[0040] The present invention will now be described in further detail with reference to the accompanying drawings:

[0041] like Figure 1 As shown, this invention proposes a video pedestrian feature extraction method based on pose estimation and elite pool log sampling. The specific implementation process is as follows:

[0042] S1: Fine-grained pose estimation and key point detection.

[0043] Obtain the video trajectory sequence (Tracklet), which consists of several consecutive frames of pedestrian images. A pre-trained deep learning pose estimation network (e.g., YOLO-Pose or AlphaPose) is used to process each frame of the trajectory, detecting and outputting a set of human skeletal keypoints. For each frame, the model outputs the coordinates and corresponding confidence scores of M=17 standard human keypoints (defined based on the COCO dataset, including nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles).

[0044] S2: Constructing a visibility quality model based on semantic completeness. Unlike traditional evaluation criteria based on pixel sharpness (such as Laplacian variance), this invention proposes a visibility quality scoring mechanism based on semantic completeness. Specifically, it first determines the effective keypoints. Valid keypoints are keypoints whose confidence level is greater than a preset confidence threshold (e.g., 0.7).

[0045]

[0046] Where M represents the total number of key points in the human skeleton (e.g., 17). This represents the confidence level of the j-th key point. Let I(⋅) be the confidence threshold and I(⋅) be the indicator function. This scoring mechanism can effectively distinguish between "invalid frames with clear backgrounds but no human figures" and "valid frames with complete human figures". If the number of valid keypoints detected in a frame is less than the minimum number of valid keypoints, the frame is directly determined to be invalid (Score=0); otherwise, its semantic integrity score is calculated according to the formula:

[0047]

[0048] in, This is the minimum number of valid keyframes (e.g., 5). This formula means it doesn't care whether the background is sharp, only whether the "person" is complete. For example, an image with a blurred background but a sharp human figure... The score will be very high; while a picture with a clear background but only half of the body is shown, its score will be very high. The score will be very low.

[0049] S3: Establish a two-layer screening mechanism: an "elite pool" and a "fallback pool." Set an elite threshold λ (preferably between 0.75 and 0.85). Based on the calculated quality score, divide all frames in the original trajectory sequence into two mutually exclusive sets: Elite Pool: Includes all high-confidence frames with a quality score greater than or equal to λ, representing the most identifiable core information in the video; Fallback Pool: Includes all frames with a quality score less than λ, serving as a supplementary data source in extreme cases. Sort the images in both pools from highest to lowest quality score. This step essentially establishes a "survival of the fittest" buffer, ensuring that subsequent sampling prioritizes high-quality data.

[0050] S4: Construct a non-linear logarithmic index mapping function, such as Figure 2 As shown.

[0051] To prioritize the preservation of high-quality frames while also considering temporal diversity, this invention abandons the traditional linear Top-K sampling and designs a logarithmic indexed mapping sampling strategy. This strategy generates sampling indices in the sorted frame sequence by constructing an exponentially growing index function. The characteristics of this mapping function are: densely distributed sampling points at the beginning of the sequence (high-quality region) to retain as many images as possible from the best perspective; and sparsely distributed sampling points at the end of the sequence (low-quality or redundant region) to capture pedestrian pose changes at different times with a larger interval span.

[0052] Assume the length of the elite pool after sorting by quality is The number of samples required is The present invention constructs the following index generation formula:

[0053] in, =0,1,..., -1 is the linearly increasing sampling index. Using this formula, the sampling index... This results in exponential growth. This leads to very dense sampling at the beginning of the sequence (i.e., the highest quality region) (e.g., indices 0, 1, 2, 3 may be selected consecutively), thus preserving the clearest appearance features; while at the end of the sequence (i.e., the slightly lower quality region), the sampling interval becomes larger (e.g., skipping 5-10 frames to select one), thus covering a longer time span at a lower cost and introducing information about the gait diversity of pedestrians.

[0054] S5: Perform adaptive sampling and dynamic completion.

[0055] Based on the system's preset target sampling quantity K, the system first attempts to extract images from the elite pool using logarithmic indexing. A dynamic completion mechanism is introduced during this process: if the number of images in the elite pool is sufficient to meet the sampling requirements, only images are sampled from the elite pool to ensure feature purity; if the number of images in the elite pool is insufficient (i.e., the overall video quality is poor), the number of missing images is calculated, and a corresponding number of images are selected from the backup pool in descending order of quality score to complete the image. This mechanism ensures that regardless of video quality, the system always outputs a fixed-dimensional feature tensor, avoiding program crashes or null value errors.

[0056] S6: Feature extraction and aggregation.

[0057] The final selected sequence of sampled frames is input into a pedestrian re-identification feature extraction network (such as ResNet-50 or ViT). The final selected K frames are then fed into a Re-ID feature extraction network (preferably using ResNet-50 as the backbone network and pre-trained on ImageNet). High-dimensional feature vectors (K 2048-dimensional feature vectors) are extracted from each frame. After L2 normalization, all feature vectors are aggregated (e.g., average pooling or weighted pooling) to generate the final feature representation of the video trajectory, which is used for subsequent pedestrian retrieval and comparison.

[0058] The present invention also proposes a storage medium storing a computer program, which, when executed by at least one processor, implements the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described above.

[0059] The present invention also proposes an electronic device, including a memory and a processor, wherein: the memory is used to store a computer program that can run on the processor; the processor is used to execute, when running the computer program, the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described above.

[0060] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for extracting pedestrian features from videos based on pose estimation and elite pool log sampling, characterized in that, Includes the following steps: (1) Obtain a video trajectory sequence including several frames of images, and use a pose estimation network to detect human key points in each frame of images; (2) Based on the key point detection results, determine the effective key points and calculate the semantic integrity score of the image; (3) Set an elite threshold and divide the images in the video trajectory sequence into an elite pool and a backup pool; sort the images in the elite pool and the backup pool in descending order according to their semantic integrity scores. (4) Construct a logarithmic index mapping function to generate a non-linear sampling index sequence based on a preset number of samples, so that the sampling points are densely distributed at the beginning of the sorted image sequence and sparsely distributed at the end. (5) Extract the target image from the elite pool according to the sampling index sequence; if the number of images in the elite pool is insufficient, fill them in order from the backup pool. (6) Input the final sampled image sequence into the pedestrian re-identification model to extract and aggregate feature vectors.

2. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, The implementation process of step (1) is as follows: The model uses a pre-trained deep learning pose estimation network to process each frame of the trajectory, detect and output a set of human skeleton key points; for each frame, the model outputs the coordinates of M key points and their corresponding confidence scores.

3. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, Step (2) of determining the effective key points specifically involves: Determine valid key points based on key point detection results. Valid keypoints are those with a confidence level greater than a preset confidence threshold. Where M represents the total number of key points in the human skeleton. This represents the confidence level of the j-th key point. is the confidence threshold, and I(⋅) is the indicator function.

4. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, The specific formula for the semantic completeness score in step (2) is as follows: Where M represents the total number of key points in the human skeleton. It is the number of valid key points; It is the minimum number of valid key points.

5. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, Step (3) specifically involves: if the semantic integrity score of a certain frame image is greater than or equal to the preset elite threshold λ, then it is assigned to the elite pool; otherwise, it is assigned to the reserve pool; the value range of the elite threshold λ is 0.75 to 0.

85.

6. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, The specific mathematical expression of the logarithmic index mapping function described in step (4) is as follows: in, =0,1,...,K−1, are linearly increasing sample numbers; sample index It exhibits exponential growth.

7. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 1, characterized in that, The implementation process of step (5) is as follows: Based on the system's preset target sampling quantity K, images are first extracted from the elite pool using logarithmic indexing. If the number of images in the elite pool is sufficient to meet the sampling requirements, only images are sampled from the elite pool to ensure the purity of the features. If the number of images in the elite pool is insufficient, the number of gaps is calculated, and a corresponding number of images are selected from the backup pool in descending order of semantic integrity score to fill in the gaps.

8. The video pedestrian feature extraction method based on pose estimation and elite pool log sampling according to claim 3, characterized in that, The confidence threshold is 0.

7.

9. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by at least one processor, implements the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described in any one of claims 1 to 8.

10. An electronic device, characterized in that, Includes memory and processor, wherein: Memory is used to store computer programs that can run on a processor; A processor, configured to, while running the computer program, perform the steps of the video pedestrian feature extraction method based on pose estimation and elite pool log sampling as described in any one of claims 1 to 8.

Images