Method, apparatus, equipment, and media for constructing passenger image datasets based on adaptive fusion frame extraction.

Abstract

This invention discloses a method, apparatus, device, and medium for constructing a passenger image dataset based on adaptive fusion frame extraction, specifically relating to the field of computer vision. It provides a method and apparatus for constructing a passenger image dataset based on adaptive fusion frame extraction. The process includes frame-by-frame decoding of bus surveillance video, followed by Gaussian filtering and CLAHE enhancement preprocessing; establishing a dynamic reference frame mechanism by calculating the structural difference, feature displacement, and edge intensity difference between the current frame and the reference frame; dynamically adjusting weights α, β, and γ to fuse the three features to obtain a score FS = α·SDI + β·FD + γ·ES; and initially screening keyframes according to a threshold T, dynamically reducing T until the required number of targets N is met. The apparatus includes video decoding, preprocessing, difference analysis, fusion scoring, and frame extraction decision modules. This invention can reduce redundancy, improve the accuracy of passenger detection models in occluded / small target scenes, and can be extended to intelligent transportation systems such as subways and stations.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method, apparatus, device, and medium for constructing a passenger detection image dataset based on adaptive fusion frame extraction. Background Technology

[0002] Constructing high-quality image datasets is a crucial step in improving the performance of object detection algorithms. Video keyframe selection, as an important preliminary step in image dataset construction, directly impacts the accuracy and generalization ability of subsequent detection models. Currently, various video frame selection methods have been proposed, which can be mainly categorized into the following three types:

[0003] The first category consists of sampling strategy-based methods. Representative techniques include uniform frame sampling, random frame sampling, continuous frame sampling, optical flow frame sampling, and frame difference sampling. These methods are simple to implement and have low computational overhead, especially uniform sampling, which is often used in benchmark experiments. However, due to the lack of consideration for changes in image content and scene, they are prone to high redundancy among selected frames, lacking diversity and representativeness, making it difficult to reflect dynamic changes in real scenes, and reducing the effectiveness of the constructed dataset.

[0004] The second category consists of clustering-based methods. These methods calculate the similarity of visual features such as color, texture, and motion between frames, then use clustering algorithms to divide the frame set and extract keyframes. While this method can improve the representativeness and diversity of keyframes to some extent, it also has significant drawbacks. Firstly, the clustering results depend on the number of clusters and initial parameter settings, resulting in poor robustness. Secondly, the computational complexity of clustering is high, making it difficult to apply to large-scale or real-time processing scenarios.

[0005] The third category consists of deep learning-based methods. These methods typically train specific neural network models to automatically select representative frames based on the semantic information of the image content. Some schemes use lightweight models to predict the importance of frames;

[0006] Frame selection is based on semantic scores, or attention mechanisms are used to model inter-frame relationships. While this approach has advantages in extracting frames with high semantic information, it suffers from high model training costs, slow processing speeds, and difficulty in effectively controlling redundancy. Furthermore, some methods introduce reinforcement learning mechanisms to optimize frame selection strategies and improve selection intelligence, but still face challenges such as unstable training and high resource consumption. Other methods attempt to improve the information coverage of selected frames through keypoint matching, but these methods are more suited to video summarization tasks and lack task-optimized designs for detection datasets. Still other methods focus on constructing datasets from web images, utilizing strategies such as sparse coding to enhance image diversity. However, since these methods are primarily based on static images, they lack adaptability to common problems in dynamic videos such as lighting changes, occlusion, and blurring, making them difficult to transfer to complex video environments such as real-world traffic monitoring.

[0007] In summary, existing video frame selection methods still suffer from problems such as insufficient representativeness, poor environmental adaptability, and excessive information redundancy when dealing with the complex and ever-changing surveillance videos of buses and subways. These methods are insufficient to meet the comprehensive requirements of efficiency, accuracy, and task relevance in the construction of target detection datasets.

[0008] Therefore, to address the above issues, a method, apparatus, device, and medium for constructing a passenger image dataset based on adaptive fusion frame extraction are proposed. Summary of the Invention

[0009] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method, apparatus, device, and medium for constructing a passenger image dataset based on adaptive fusion frame extraction, in order to solve the problems mentioned in the background art.

[0010] To achieve the above objectives, the present invention provides the following technical solution: a method for constructing a passenger image dataset based on adaptive fusion frame extraction, characterized by comprising the following steps:

[0011] S1. Video Decoding and Image Sequence Acquisition: Decode the bus surveillance video frame by frame to obtain a temporally continuous sequence of image frames, providing a data source for subsequent frame extraction;

[0012] S2. Image preprocessing: Perform two-dimensional Gaussian filtering on each frame of the image to remove background noise and compression distortion, and enhance the robustness of subsequent image structure and edge feature extraction. It is recommended to set the Gaussian kernel to 5×5, and the standard deviation σ can be set empirically.

[0013] S3. Reference Frame and Current Frame Setting: In the keyframe scoring process, the reference frame frefer represents the image frame most recently selected as a keyframe, and the current frame fcur represents the candidate frame participating in the scoring calculation. During initialization, the first frame in the sequence is set as the reference frame frefer, and the subsequent frames are successively used as fcur. The structural difference (SDI), feature displacement (FD), and edge intensity difference (ES) are calculated with frefer. Once a certain fcur meets the frame selection conditions and is selected as a keyframe, it replaces frefer as the new reference frame and participates in the subsequent scoring comparison.

[0014] S4. Structural Difference Calculation: The structural difference SDI between the current frame and the reference frame is calculated using structural similarity (SSIM), reflecting the degree of change in brightness, contrast, texture structure, etc. SDI = 1 - SSIM(frefer, fcur);

[0015] S5. Feature displacement calculation: The SIFT algorithm is used to extract the feature point set of the reference frame and the current frame. FLANN+RANSAC is used to match the feature points and calculate the average Euclidean distance of the matched points to obtain the feature displacement FD, which is used to evaluate the dynamic changes of passengers or background in the scene.

[0016] S6. Edge intensity calculation: After the image is enhanced by CLAHE histogram equalization, the Sobel operator is applied to extract the horizontal and vertical edge images, and the edge intensity difference ES between the current frame and the reference frame is calculated to reflect the detailed changes in the structural contours in the image.

[0017] S7. Fusion Score Calculation: The Structural Difference (SDI), Feature Displacement (FD), and Edge Intensity (ES) are linearly weighted and fused. The weights a, β, and Y are dynamically adjusted based on FD and ES to calculate the fusion score FS of the current keyframe. FS = a·SDI + β·FD + Y·ES, where the weight coefficients a, β, and Y are dynamically adjusted according to the changes in FD and ES to balance the influence of different feature dimensions on the scoring results.

[0018] S8. Threshold Judgment and Keyframe Selection: Set the number of target keyframes N, and select frames with FS≥T as keyframes based on the threshold T of the score FS. If the number of frames that meet the condition is less than N, then reduce T and re-select until the number of selected keyframes is not less than N. Finally, save the selected keyframes as images to construct the target detection image dataset.

[0019] An adaptive fusion video keyframe selection device, used to achieve the above, includes:

[0020] Video decoding module: used to acquire image frame sequences;

[0021] Image preprocessing module: used to perform noise reduction operations;

[0022] Difference Analysis Module: Used to calculate structural difference, characteristic displacement, and edge strength difference;

[0023] Fusion scoring module: used to calculate keyframe scores based on three types of feature information;

[0024] Frame extraction decision module: used to output keyframes based on scores and dynamic thresholds.

[0025] An electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, the computer program being used to perform the methods described above.

[0026] A computer-readable storage medium for storing computer program instructions for performing the methods described above.

[0027] The technical effects and advantages of this invention are as follows:

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. It can integrate multi-dimensional image change information and adaptively identify representative key frames in videos to avoid missing key information;

[0030] 2. Effectively reduces image redundancy and improves dataset construction efficiency;

[0031] 3. Improve the accuracy of the bus passenger detection model in small target recognition, occlusion handling, and complex lighting conditions; the method has good versatility and can be extended to video analysis tasks in intelligent transportation scenarios such as subways and stations. Attached Figure Description

[0032] Figure 1 This is an overall flowchart of the method described in this invention;

[0033] Figure 2 This is a schematic diagram illustrating the dynamic update relationship between the reference frame and the current frame;

[0034] Figure 3 A schematic diagram illustrating the calculation principles for structural dissimilarity, characteristic displacement, and edge strength dissimilarity.

[0035] Figure 4 This is a schematic diagram of the weighted fusion of the keyframe scoring function;

[0036] Figure 5 This is a block diagram of the module composition of the keyframe selection device of the present invention;

[0037] Figure 6 This is a diagram of a dynamic threshold adjustment mechanism based on the number of target keyframes. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Example 1

[0040] The first embodiment of this invention provides a method for constructing a passenger image dataset based on adaptive fusion frame extraction. This method can be executed by an adaptive fusion video keyframe selection device (hereinafter referred to as the video keyframe selection device), specifically by one or more processors within the video keyframe selection device. In this embodiment, the video keyframe selection device can be a computer or other terminal with data processing capabilities. For ease of understanding, a computer is used as an example in the following description. The method achieves the following steps:

[0041] S1 extracts the image sequence frame by frame from the bus surveillance video. Assume the video sequence is V, the total number of frames is N, and the i-th frame is fi, represented as V = {f1, f2, ..., fN}. Here, fi is the i-th frame image. The video decoding module is responsible for decoding the video and obtaining the image frame sequence.

[0042] The S2 denoising operation applies a two-dimensional Gaussian filter to each frame to remove background noise and compression distortion. It also enhances the image's structure and edge features; a contrast-limited adaptive histogram equalization (CLAHE) is applied to enhance the image, improving detail and contrast, which facilitates subsequent edge detection and feature extraction.

[0043] S3 reference frame and current frame settings: The reference frame refers to the most recently selected image frame as a keyframe, and the current frame refers to the candidate frame currently participating in the scoring calculation. During initialization, the first frame f1 in the sequence is set as the reference frame. Frame update: In the scoring calculation of subsequent frames, the current frame is compared with the reference frame. Once a frame meets the keyframe selection criteria, it is selected as the new reference frame and participates in the scoring comparison of subsequent frames.

[0044] S4. Calculate the structural difference (SDI). Use the structural similarity index to calculate the structural difference between the current frame and the reference frame, reflecting the changes in brightness, contrast, and texture between the two frames. The calculation formula is shown in (1): SDI(frefer,fcur)=1-SSIM(frefer,fcur) where SSIM represents the structural similarity index. The SDI value ranges from 0 to 1, where 0 indicates that the structures are the same and 1 indicates that they are completely dissimilar.

[0045] S5. The feature displacement is calculated using formula (2). The SIFT (Scale Invariant Feature Transform) algorithm is used to extract the feature point sets of the current frame and the reference frame. FLANN (Fast Nearest Neighbor Search) and RANSAC (Random Sample Consensus Algorithm) are used to match the feature points. The average Euclidean distance between the matching points is calculated to obtain the feature displacement and evaluate the dynamic changes in the scene.

[0046] Where, p i p i , represent the coordinates of the matched key points, and F represents the set of interior point matches filtered by RANSAC ||p i -p i , || represents the Euclidean distance between the matched keypoints.

[0047] S6. Edge intensity is calculated using formulas (3) and (4), and the Sobel operator is used to extract the edge images of the current frame and the reference frame. Histogram equalization enhancement is performed on the image by applying CLAHE, and then the edge intensity difference is calculated to reflect the changes in details and contours in the image.

[0048]

[0049] Where S represents the saliency mask region.

[0050] The S7 fusion score calculation uses weighted fusion structural difference (SDI), feature displacement (FD), and edge intensity difference (ES) to calculate the keyframe score (FS) of the current frame. The weighting coefficients are dynamically adjusted based on changes in FD and ES to balance the impact of different features on the score result.

[0051] The S8 thresholding and keyframe selection process sets an initial scoring threshold based on the target number of keyframes. Frames with scores greater than or equal to the calculated score are selected as keyframes. If the number of selected keyframes is less than the target number, the threshold is lowered and the selection is repeated until the number of selected keyframes reaches or exceeds the target number. The final selected keyframes are saved as image files to construct an image dataset suitable for object detection tasks.

[0052] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0053] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0054] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for constructing a passenger image dataset based on adaptive fusion frame extraction, characterized in that, Includes the following steps: S1. Video Decoding and Image Sequence Acquisition: Decode the bus surveillance video frame by frame to obtain a temporally continuous sequence of image frames, providing a data source for subsequent frame extraction; S2. Image preprocessing: Perform two-dimensional Gaussian filtering on each frame of the image to remove background noise and compression distortion, and enhance the robustness of subsequent image structure and edge feature extraction. It is recommended to set the Gaussian kernel to 5×5, and the standard deviation σ can be set empirically. S3. Reference Frame and Current Frame Setting: In the keyframe scoring process, the reference frame frefer represents the image frame most recently selected as a keyframe, and the current frame fcur represents the candidate frame participating in the scoring calculation. During initialization, the first frame in the sequence is set as the reference frame frefer, and the subsequent frames are successively used as fcur. The structural difference (SDI), feature displacement (FD), and edge intensity difference (ES) are calculated with frefer. Once a certain fcur meets the frame selection conditions and is selected as a keyframe, it replaces frefer as the new reference frame and participates in the subsequent scoring comparison. S4. Structural Difference Calculation: The structural difference SDI between the current frame and the reference frame is calculated using structural similarity (SSIM), reflecting the degree of change in brightness, contrast, texture structure, etc. SDI = 1 - SSIM(frefer, fcur); S5. Feature displacement calculation: The SIFT algorithm is used to extract the feature point set of the reference frame and the current frame. FLANN+RANSAC is used to match the feature points and calculate the average Euclidean distance of the matched points to obtain the feature displacement FD, which is used to evaluate the dynamic changes of passengers or background in the scene. S6. Edge intensity calculation: After the image is enhanced by CLAHE histogram equalization, the Sobel operator is applied to extract the horizontal and vertical edge images, and the edge intensity difference ES between the current frame and the reference frame is calculated to reflect the detailed changes in the structural contours in the image. S7. Fusion Score Calculation: The Structural Difference (SDI), Feature Displacement (FD), and Edge Intensity (ES) are linearly weighted and fused. The weights a, β, and Y are dynamically adjusted based on FD and ES to calculate the fusion score FS of the current keyframe. FS = a·SDI + β·FD + Y·ES, where the weight coefficients a, β, and Y are dynamically adjusted according to the changes in FD and ES to balance the influence of different feature dimensions on the scoring results. S8. Threshold Judgment and Keyframe Selection: Set the number of target keyframes N, and select frames with FS≥T as keyframes based on the threshold T of the score FS. If the number of frames that meet the condition is less than N, then reduce T and re-select until the number of selected keyframes is not less than N. Finally, save the selected keyframes as images to construct the target detection image dataset.

2. An adaptive fusion video keyframe selection device, used to implement the passenger image dataset construction method based on adaptive fusion frame extraction as described in claim 1, characterized in that, include: Video decoding module: used to acquire image frame sequences; Image preprocessing module: used to perform noise reduction operations; Difference Analysis Module: Used to calculate structural difference, characteristic displacement, and edge strength difference; Fusion scoring module: used to calculate keyframe scores based on three types of feature information; Frame extraction decision module: used to output keyframes based on scores and dynamic thresholds.

3. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, the computer program being used to execute the passenger image dataset construction method based on adaptive fusion frame extraction as described in claim 1.

4. A computer-readable storage medium, characterized in that, This is used to store computer program instructions for performing the method for constructing a passenger image dataset based on adaptive fusion frame extraction as described in claim 1.

Images